[
https://issues.apache.org/jira/browse/CASSANDRA-2415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13015633#comment-13015633
]
Brandon Williams commented on CASSANDRA-2415:
---------------------------------------------
What was the token set to in cassandra.yaml? The snitch can't change the
token, so this sounds like a normal bootstrap collision; either the token
specified conflicted, or the bootstrap rules for automatic selection weren't
adhered to (wait 2 mins between every bootstrap, etc.)
> Ec2Snitch changing tokens
> -------------------------
>
> Key: CASSANDRA-2415
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2415
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.7.4
> Environment: Amazon EC2 -- 4 nodes
> Reporter: Sasha Dolgy
> Assignee: Brandon Williams
> Priority: Minor
> Fix For: 0.7.5
>
>
> A new 4 node 0.7.4 cluster on Amazon EC2
> 1. Brought up the first node without issue with Ec2Snitch configured in the
> cassandra.yaml.
> 2. Brought up a second node, with the first node defined as the seed. No
> visible issues.
> 3. Brought up node 3 in the same manner. Receiving errors as shown in the
> output below.
> Initially, I -did not- define tokens for the nodes.
> When node 3 was brought online, I had this problem and manually moved the
> tokens and did a nodetool move/repair/clean before getting on to node 4.
> The tokens for the 4 nodes:
> 0
> 19095547144942516281182777765338228798
> 56713727820156410577229101238628035242
> 170141183460469231731687303715884105726
> When the 4th node comes online, with it's token set in the cassandra.yaml
> (first one i did it for because of the errors I saw with node 3) ...
> everything goes well at first in joining the ring, etc.....then I see the
> following error in the system.log:
> :~$ INFO [HintedHandoff:1] 2011-03-23 00:37:24,298 HintedHandOffManager.java
> (line 304) Started hinted handoff for
> endpoint /10.0.0.2
> INFO [HintedHandoff:1] 2011-03-23 00:37:24,298 HintedHandOffManager.java
> (line 360) Finished hinted handoff of 0 rows
> to endpoint /10.0.0.2
> INFO [GossipStage:2] 2011-03-23 00:37:55,381 StorageService.java (line 702)
> Node /10.0.0.2 state jump to bootstrap
> ERROR [GossipStage:2] 2011-03-23 00:37:55,381
> DebuggableThreadPoolExecutor.java (line 103) Error in
> ThreadPoolExecutor java.lang.RuntimeException: Bootstrap Token collision
> between /10.0.0.3 and /10.0.0.2 (token 19095547144942516281182777765338228798
> at
> org.apache.cassandra.locator.TokenMetadata.addBootstrapToken(TokenMetadata.java:143)
> at
> org.apache.cassandra.service.StorageService.handleStateBootstrap(StorageService.java:706)
> at
> org.apache.cassandra.service.StorageService.onChange(StorageService.java:648)
> at
> org.apache.cassandra.gms.Gossiper.doNotifications(Gossiper.java:772)
> at
> org.apache.cassandra.gms.Gossiper.applyApplicationStateLocally(Gossiper.java:737)
> at
> org.apache.cassandra.gms.Gossiper.applyStateLocally(Gossiper.java:679)
> at
> org.apache.cassandra.gms.GossipDigestAck2VerbHandler.doVerb(GossipDigestAck2VerbHandler.java:60)
> at
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:72)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> ERROR [GossipStage:2] 2011-03-23 00:37:55,382
> AbstractCassandraDaemon.java (line 112) Fatal exception in thread
> Thread[GossipStage:2,5,main]
> java.lang.RuntimeException: Bootstrap Token collision between
> /10.0.0.3 and /10.0.0.2 (token 19095547144942516281182777765338228798
> at
> org.apache.cassandra.locator.TokenMetadata.addBootstrapToken(TokenMetadata.java:143)
> at
> org.apache.cassandra.service.StorageService.handleStateBootstrap(StorageService.java:706)
> at
> org.apache.cassandra.service.StorageService.onChange(StorageService.java:648)
> at
> org.apache.cassandra.gms.Gossiper.doNotifications(Gossiper.java:772)
> at
> org.apache.cassandra.gms.Gossiper.applyApplicationStateLocally(Gossiper.java:737)
> at
> org.apache.cassandra.gms.Gossiper.applyStateLocally(Gossiper.java:679)
> at
> org.apache.cassandra.gms.GossipDigestAck2VerbHandler.doVerb(GossipDigestAck2VerbHandler.java:60)
> at
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:72)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> :~$ INFO [GossipStage:3] 2011-03-23 00:38:24,859 StorageService.java
> (line 745) Nodes /10.0.0.2 and /10.0.0.3 have the same token
> 19095547144942516281182777765338228798. /10.0.0.2 is the new owner
> WARN [GossipStage:3] 2011-03-23 00:38:24,859 TokenMetadata.java (line
> 115) Token 19095547144942516281182777765338228798 changing ownership
> from /10.0.0.3 to /10.0.0.2
> :~$ nodetool -h 10.0.0.1 -p 9090 ring
> Address Status State Load Owns Token
> 170141183460469231731687303715884105726
> 10.0.0.1 Up Normal 99.31 KB 0.00% 0
> 10.0.0.2 Up Normal 122.67 KB 11.22%
> 19095547144942516281182777765338228798
> 10.0.0.4 Up Normal 103.75 KB 88.78%
> 170141183460469231731687303715884105726
> :~$
> The EC2 nodes are all part of the same region (Singapore).
> I have removed the Ec2Snitch configuration from all nodes and set it back to
> the default, restarted all nodes, and moved tokens back to the original
> values.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
