Ec2Snitch changing tokens
-------------------------
Key: CASSANDRA-2415
URL: https://issues.apache.org/jira/browse/CASSANDRA-2415
Project: Cassandra
Issue Type: Bug
Affects Versions: 0.7.4
Environment: Amazon EC2 -- 4 nodes
Reporter: Sasha Dolgy
A new 4 node 0.7.4 cluster on Amazon EC2
1. Brought up the first node without issue with Ec2Snitch configured in the
cassandra.yaml.
2. Brought up a second node, with the first node defined as the seed. No
visible issues.
3. Brought up node 3 in the same manner. Receiving errors as shown in the
output below.
Initially, I -did not- define tokens for the nodes.
When node 3 was brought online, I had this problem and manually moved the
tokens and did a nodetool move/repair/clean before getting on to node 4.
The tokens for the 4 nodes:
0
19095547144942516281182777765338228798
56713727820156410577229101238628035242
170141183460469231731687303715884105726
When the 4th node comes online, with it's token set in the cassandra.yaml
(first one i did it for because of the errors I saw with node 3) ... everything
goes well at first in joining the ring, etc.....then I see the following error
in the system.log:
:~$ INFO [HintedHandoff:1] 2011-03-23 00:37:24,298 HintedHandOffManager.java
(line 304) Started hinted handoff for
endpoint /10.0.0.2
INFO [HintedHandoff:1] 2011-03-23 00:37:24,298 HintedHandOffManager.java (line
360) Finished hinted handoff of 0 rows
to endpoint /10.0.0.2
INFO [GossipStage:2] 2011-03-23 00:37:55,381 StorageService.java (line 702)
Node /10.0.0.2 state jump to bootstrap
ERROR [GossipStage:2] 2011-03-23 00:37:55,381 DebuggableThreadPoolExecutor.java
(line 103) Error in
ThreadPoolExecutor java.lang.RuntimeException: Bootstrap Token collision
between /10.0.0.3 and /10.0.0.2 (token 19095547144942516281182777765338228798
at
org.apache.cassandra.locator.TokenMetadata.addBootstrapToken(TokenMetadata.java:143)
at
org.apache.cassandra.service.StorageService.handleStateBootstrap(StorageService.java:706)
at
org.apache.cassandra.service.StorageService.onChange(StorageService.java:648)
at org.apache.cassandra.gms.Gossiper.doNotifications(Gossiper.java:772)
at
org.apache.cassandra.gms.Gossiper.applyApplicationStateLocally(Gossiper.java:737)
at
org.apache.cassandra.gms.Gossiper.applyStateLocally(Gossiper.java:679)
at
org.apache.cassandra.gms.GossipDigestAck2VerbHandler.doVerb(GossipDigestAck2VerbHandler.java:60)
at
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:72)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
ERROR [GossipStage:2] 2011-03-23 00:37:55,382
AbstractCassandraDaemon.java (line 112) Fatal exception in thread
Thread[GossipStage:2,5,main]
java.lang.RuntimeException: Bootstrap Token collision between
/10.0.0.3 and /10.0.0.2 (token 19095547144942516281182777765338228798
at
org.apache.cassandra.locator.TokenMetadata.addBootstrapToken(TokenMetadata.java:143)
at
org.apache.cassandra.service.StorageService.handleStateBootstrap(StorageService.java:706)
at
org.apache.cassandra.service.StorageService.onChange(StorageService.java:648)
at org.apache.cassandra.gms.Gossiper.doNotifications(Gossiper.java:772)
at
org.apache.cassandra.gms.Gossiper.applyApplicationStateLocally(Gossiper.java:737)
at
org.apache.cassandra.gms.Gossiper.applyStateLocally(Gossiper.java:679)
at
org.apache.cassandra.gms.GossipDigestAck2VerbHandler.doVerb(GossipDigestAck2VerbHandler.java:60)
at
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:72)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
:~$ INFO [GossipStage:3] 2011-03-23 00:38:24,859 StorageService.java
(line 745) Nodes /10.0.0.2 and /10.0.0.3 have the same token
19095547144942516281182777765338228798. /10.0.0.2 is the new owner
WARN [GossipStage:3] 2011-03-23 00:38:24,859 TokenMetadata.java (line
115) Token 19095547144942516281182777765338228798 changing ownership
from /10.0.0.3 to /10.0.0.2
:~$ nodetool -h 10.0.0.1 -p 9090 ring
Address Status State Load Owns Token
170141183460469231731687303715884105726
10.0.0.1 Up Normal 99.31 KB 0.00% 0
10.0.0.2 Up Normal 122.67 KB 11.22%
19095547144942516281182777765338228798
10.0.0.4 Up Normal 103.75 KB 88.78%
170141183460469231731687303715884105726
:~$
The EC2 nodes are all part of the same region (Singapore).
I have removed the Ec2Snitch configuration from all nodes and set it back to
the default, restarted all nodes, and moved tokens back to the original values.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
