[ https://issues.apache.org/jira/browse/CASSANDRA-2415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13015715#comment-13015715 ]
Brandon Williams commented on CASSANDRA-2415: --------------------------------------------- Were they added simultaneously? > Ec2Snitch changing tokens > ------------------------- > > Key: CASSANDRA-2415 > URL: https://issues.apache.org/jira/browse/CASSANDRA-2415 > Project: Cassandra > Issue Type: Bug > Components: Core > Affects Versions: 0.7.4 > Environment: Amazon EC2 -- 4 nodes > Reporter: Sasha Dolgy > Assignee: Brandon Williams > Priority: Minor > Fix For: 0.7.5 > > > A new 4 node 0.7.4 cluster on Amazon EC2 > 1. Brought up the first node without issue with Ec2Snitch configured in the > cassandra.yaml. > 2. Brought up a second node, with the first node defined as the seed. No > visible issues. > 3. Brought up node 3 in the same manner. Receiving errors as shown in the > output below. > Initially, I -did not- define tokens for the nodes. > When node 3 was brought online, I had this problem and manually moved the > tokens and did a nodetool move/repair/clean before getting on to node 4. > The tokens for the 4 nodes: > 0 > 19095547144942516281182777765338228798 > 56713727820156410577229101238628035242 > 170141183460469231731687303715884105726 > When the 4th node comes online, with it's token set in the cassandra.yaml > (first one i did it for because of the errors I saw with node 3) ... > everything goes well at first in joining the ring, etc.....then I see the > following error in the system.log: > :~$ INFO [HintedHandoff:1] 2011-03-23 00:37:24,298 HintedHandOffManager.java > (line 304) Started hinted handoff for > endpoint /10.0.0.2 > INFO [HintedHandoff:1] 2011-03-23 00:37:24,298 HintedHandOffManager.java > (line 360) Finished hinted handoff of 0 rows > to endpoint /10.0.0.2 > INFO [GossipStage:2] 2011-03-23 00:37:55,381 StorageService.java (line 702) > Node /10.0.0.2 state jump to bootstrap > ERROR [GossipStage:2] 2011-03-23 00:37:55,381 > DebuggableThreadPoolExecutor.java (line 103) Error in > ThreadPoolExecutor java.lang.RuntimeException: Bootstrap Token collision > between /10.0.0.3 and /10.0.0.2 (token 19095547144942516281182777765338228798 > at > org.apache.cassandra.locator.TokenMetadata.addBootstrapToken(TokenMetadata.java:143) > at > org.apache.cassandra.service.StorageService.handleStateBootstrap(StorageService.java:706) > at > org.apache.cassandra.service.StorageService.onChange(StorageService.java:648) > at > org.apache.cassandra.gms.Gossiper.doNotifications(Gossiper.java:772) > at > org.apache.cassandra.gms.Gossiper.applyApplicationStateLocally(Gossiper.java:737) > at > org.apache.cassandra.gms.Gossiper.applyStateLocally(Gossiper.java:679) > at > org.apache.cassandra.gms.GossipDigestAck2VerbHandler.doVerb(GossipDigestAck2VerbHandler.java:60) > at > org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:72) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > ERROR [GossipStage:2] 2011-03-23 00:37:55,382 > AbstractCassandraDaemon.java (line 112) Fatal exception in thread > Thread[GossipStage:2,5,main] > java.lang.RuntimeException: Bootstrap Token collision between > /10.0.0.3 and /10.0.0.2 (token 19095547144942516281182777765338228798 > at > org.apache.cassandra.locator.TokenMetadata.addBootstrapToken(TokenMetadata.java:143) > at > org.apache.cassandra.service.StorageService.handleStateBootstrap(StorageService.java:706) > at > org.apache.cassandra.service.StorageService.onChange(StorageService.java:648) > at > org.apache.cassandra.gms.Gossiper.doNotifications(Gossiper.java:772) > at > org.apache.cassandra.gms.Gossiper.applyApplicationStateLocally(Gossiper.java:737) > at > org.apache.cassandra.gms.Gossiper.applyStateLocally(Gossiper.java:679) > at > org.apache.cassandra.gms.GossipDigestAck2VerbHandler.doVerb(GossipDigestAck2VerbHandler.java:60) > at > org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:72) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > :~$ INFO [GossipStage:3] 2011-03-23 00:38:24,859 StorageService.java > (line 745) Nodes /10.0.0.2 and /10.0.0.3 have the same token > 19095547144942516281182777765338228798. /10.0.0.2 is the new owner > WARN [GossipStage:3] 2011-03-23 00:38:24,859 TokenMetadata.java (line > 115) Token 19095547144942516281182777765338228798 changing ownership > from /10.0.0.3 to /10.0.0.2 > :~$ nodetool -h 10.0.0.1 -p 9090 ring > Address Status State Load Owns Token > 170141183460469231731687303715884105726 > 10.0.0.1 Up Normal 99.31 KB 0.00% 0 > 10.0.0.2 Up Normal 122.67 KB 11.22% > 19095547144942516281182777765338228798 > 10.0.0.4 Up Normal 103.75 KB 88.78% > 170141183460469231731687303715884105726 > :~$ > The EC2 nodes are all part of the same region (Singapore). > I have removed the Ec2Snitch configuration from all nodes and set it back to > the default, restarted all nodes, and moved tokens back to the original > values. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira