Hi there,
Installed a new 4 node 0.7.4 cluster on ec2. Brought up the first
node without issue with Ec2Snitch configured in the cassandra.yaml.
Brought up a second node, with the first node defined as the seed. No
visible issues. 3 & 4 however are giving me problems as shown in the
output below. Initially, I -did not- define tokens. When node 3 came
up, I had this error, so i went and manually moved the tokens and did
a nodetool move/repair/clean before getting on to node 4.
The tokens for the 4 nodes:
0
19095547144942516281182777765338228798
56713727820156410577229101238628035242
170141183460469231731687303715884105726
So now, when the 4th node comes online, with it's token set in the
cassandra.yaml (first one i did it for because of the errors I saw
with node 3) ... everything goes well at first, in joining the ring,
etc.....and then I see the following error in the system.log:
:~$ INFO [HintedHandoff:1] 2011-03-23 00:37:24,298
HintedHandOffManager.java (line 304) Started hinted handoff for
endpoint /10.0.0.2
INFO [HintedHandoff:1] 2011-03-23 00:37:24,298
HintedHandOffManager.java (line 360) Finished hinted handoff of 0 rows
to endpoint /10.0.0.2
INFO [GossipStage:2] 2011-03-23 00:37:55,381 StorageService.java
(line 702) Node /10.0.0.2 state jump to bootstrap
ERROR [GossipStage:2] 2011-03-23 00:37:55,381
DebuggableThreadPoolExecutor.java (line 103) Error in
ThreadPoolExecutor
java.lang.RuntimeException: Bootstrap Token collision between
/10.0.0.3 and /10.0.0.2 (token 19095547144942516281182777765338228798
at
org.apache.cassandra.locator.TokenMetadata.addBootstrapToken(TokenMetadata.java:143)
at
org.apache.cassandra.service.StorageService.handleStateBootstrap(StorageService.java:706)
at
org.apache.cassandra.service.StorageService.onChange(StorageService.java:648)
at org.apache.cassandra.gms.Gossiper.doNotifications(Gossiper.java:772)
at
org.apache.cassandra.gms.Gossiper.applyApplicationStateLocally(Gossiper.java:737)
at
org.apache.cassandra.gms.Gossiper.applyStateLocally(Gossiper.java:679)
at
org.apache.cassandra.gms.GossipDigestAck2VerbHandler.doVerb(GossipDigestAck2VerbHandler.java:60)
at
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:72)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
ERROR [GossipStage:2] 2011-03-23 00:37:55,382
AbstractCassandraDaemon.java (line 112) Fatal exception in thread
Thread[GossipStage:2,5,main]
java.lang.RuntimeException: Bootstrap Token collision between
/10.0.0.3 and /10.0.0.2 (token 19095547144942516281182777765338228798
at
org.apache.cassandra.locator.TokenMetadata.addBootstrapToken(TokenMetadata.java:143)
at
org.apache.cassandra.service.StorageService.handleStateBootstrap(StorageService.java:706)
at
org.apache.cassandra.service.StorageService.onChange(StorageService.java:648)
at org.apache.cassandra.gms.Gossiper.doNotifications(Gossiper.java:772)
at
org.apache.cassandra.gms.Gossiper.applyApplicationStateLocally(Gossiper.java:737)
at
org.apache.cassandra.gms.Gossiper.applyStateLocally(Gossiper.java:679)
at
org.apache.cassandra.gms.GossipDigestAck2VerbHandler.doVerb(GossipDigestAck2VerbHandler.java:60)
at
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:72)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
:~$ INFO [GossipStage:3] 2011-03-23 00:38:24,859 StorageService.java
(line 745) Nodes /10.0.0.2 and /10.0.0.3 have the same token
19095547144942516281182777765338228798. /10.0.0.2 is the new owner
WARN [GossipStage:3] 2011-03-23 00:38:24,859 TokenMetadata.java (line
115) Token 19095547144942516281182777765338228798 changing ownership
from /10.0.0.3 to /10.0.0.2
:~$ nodetool -h 10.0.0.1 -p 9090 ring
Address Status State Load Owns Token
170141183460469231731687303715884105726
10.0.0.1 Up Normal 99.31 KB 0.00% 0
10.0.0.2 Up Normal 122.67 KB 11.22%
19095547144942516281182777765338228798
10.0.0.4 Up Normal 103.75 KB 88.78%
170141183460469231731687303715884105726
:~$
Should I be a bit more hands off with the Ec2Snitch .... ? Now i have
3 nodes with 1 having a duplicate token ....
--
Sasha Dolgy
sasha.do...@gmail.com
