[
https://issues.apache.org/jira/browse/CASSANDRA-2415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13015718#comment-13015718
]
Sasha Dolgy commented on CASSANDRA-2415:
----------------------------------------
no. they were added one at a time and only once the previous node had fully
joined the ring 100% and state in nodetool ring was "Normal".
problem was stopped by changing the endpoint_snitch to SimpleSnitch in the yaml
on each node, restarting each node in sequence, waiting for it to resume normal
operations (as per the system.log) before continuing with the next node. have
not seen the error since moving away from Ec2Snitch
> Ec2Snitch changing tokens
> -------------------------
>
> Key: CASSANDRA-2415
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2415
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.7.4
> Environment: Amazon EC2 -- 4 nodes
> Reporter: Sasha Dolgy
> Assignee: Brandon Williams
> Priority: Minor
> Fix For: 0.7.5
>
>
> A new 4 node 0.7.4 cluster on Amazon EC2
> 1. Brought up the first node without issue with Ec2Snitch configured in the
> cassandra.yaml.
> 2. Brought up a second node, with the first node defined as the seed. No
> visible issues.
> 3. Brought up node 3 in the same manner. Receiving errors as shown in the
> output below.
> Initially, I -did not- define tokens for the nodes.
> When node 3 was brought online, I had this problem and manually moved the
> tokens and did a nodetool move/repair/clean before getting on to node 4.
> The tokens for the 4 nodes:
> 0
> 19095547144942516281182777765338228798
> 56713727820156410577229101238628035242
> 170141183460469231731687303715884105726
> When the 4th node comes online, with it's token set in the cassandra.yaml
> (first one i did it for because of the errors I saw with node 3) ...
> everything goes well at first in joining the ring, etc.....then I see the
> following error in the system.log:
> :~$ INFO [HintedHandoff:1] 2011-03-23 00:37:24,298 HintedHandOffManager.java
> (line 304) Started hinted handoff for
> endpoint /10.0.0.2
> INFO [HintedHandoff:1] 2011-03-23 00:37:24,298 HintedHandOffManager.java
> (line 360) Finished hinted handoff of 0 rows
> to endpoint /10.0.0.2
> INFO [GossipStage:2] 2011-03-23 00:37:55,381 StorageService.java (line 702)
> Node /10.0.0.2 state jump to bootstrap
> ERROR [GossipStage:2] 2011-03-23 00:37:55,381
> DebuggableThreadPoolExecutor.java (line 103) Error in
> ThreadPoolExecutor java.lang.RuntimeException: Bootstrap Token collision
> between /10.0.0.3 and /10.0.0.2 (token 19095547144942516281182777765338228798
> at
> org.apache.cassandra.locator.TokenMetadata.addBootstrapToken(TokenMetadata.java:143)
> at
> org.apache.cassandra.service.StorageService.handleStateBootstrap(StorageService.java:706)
> at
> org.apache.cassandra.service.StorageService.onChange(StorageService.java:648)
> at
> org.apache.cassandra.gms.Gossiper.doNotifications(Gossiper.java:772)
> at
> org.apache.cassandra.gms.Gossiper.applyApplicationStateLocally(Gossiper.java:737)
> at
> org.apache.cassandra.gms.Gossiper.applyStateLocally(Gossiper.java:679)
> at
> org.apache.cassandra.gms.GossipDigestAck2VerbHandler.doVerb(GossipDigestAck2VerbHandler.java:60)
> at
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:72)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> ERROR [GossipStage:2] 2011-03-23 00:37:55,382
> AbstractCassandraDaemon.java (line 112) Fatal exception in thread
> Thread[GossipStage:2,5,main]
> java.lang.RuntimeException: Bootstrap Token collision between
> /10.0.0.3 and /10.0.0.2 (token 19095547144942516281182777765338228798
> at
> org.apache.cassandra.locator.TokenMetadata.addBootstrapToken(TokenMetadata.java:143)
> at
> org.apache.cassandra.service.StorageService.handleStateBootstrap(StorageService.java:706)
> at
> org.apache.cassandra.service.StorageService.onChange(StorageService.java:648)
> at
> org.apache.cassandra.gms.Gossiper.doNotifications(Gossiper.java:772)
> at
> org.apache.cassandra.gms.Gossiper.applyApplicationStateLocally(Gossiper.java:737)
> at
> org.apache.cassandra.gms.Gossiper.applyStateLocally(Gossiper.java:679)
> at
> org.apache.cassandra.gms.GossipDigestAck2VerbHandler.doVerb(GossipDigestAck2VerbHandler.java:60)
> at
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:72)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> :~$ INFO [GossipStage:3] 2011-03-23 00:38:24,859 StorageService.java
> (line 745) Nodes /10.0.0.2 and /10.0.0.3 have the same token
> 19095547144942516281182777765338228798. /10.0.0.2 is the new owner
> WARN [GossipStage:3] 2011-03-23 00:38:24,859 TokenMetadata.java (line
> 115) Token 19095547144942516281182777765338228798 changing ownership
> from /10.0.0.3 to /10.0.0.2
> :~$ nodetool -h 10.0.0.1 -p 9090 ring
> Address Status State Load Owns Token
> 170141183460469231731687303715884105726
> 10.0.0.1 Up Normal 99.31 KB 0.00% 0
> 10.0.0.2 Up Normal 122.67 KB 11.22%
> 19095547144942516281182777765338228798
> 10.0.0.4 Up Normal 103.75 KB 88.78%
> 170141183460469231731687303715884105726
> :~$
> The EC2 nodes are all part of the same region (Singapore).
> I have removed the Ec2Snitch configuration from all nodes and set it back to
> the default, restarted all nodes, and moved tokens back to the original
> values.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
