other nodes in the ring will throw exception when I use the same TokenID but
different IP address for a new node bootstrapping
------------------------------------------------------------------------------------------------------------------------------
Key: CASSANDRA-727
URL: https://issues.apache.org/jira/browse/CASSANDRA-727
Project: Cassandra
Issue Type: Bug
Components: Core
Affects Versions: 0.5
Reporter: david.pan
I find other nodes will throw exception when I use the same TokenID but
different IP address for a new node while bootstrapping. My operations are that
:
1) down nodeA which token is T, IP address is IPA;
2) add a new node B which token is also T but IP address is IPB;
3) After nodeA knows nodeB is bootstrapping, A throw exception:
DEBUG [GMFD:2] 2010-01-19 22:22:36,667 StorageService.java (line 439) Node
/10.81.37.52 state bootstrapping, token 136112946768375385385349842972707284580
ERROR [GMFD:2] 2010-01-19 22:22:36,667 DebuggableThreadPoolExecutor.java (line
157) Error in ThreadPoolExecutor
java.lang.RuntimeException: Bootstrap Token collision between /10.81.37.65 and
/10.81.37.52 (token 136112946768375385385349842972707284580
at
org.apache.cassandra.locator.TokenMetadata.addBootstrapToken(TokenMetadata.java:136)
at
org.apache.cassandra.service.StorageService.handleStateBootstrap(StorageService.java:456)
at
org.apache.cassandra.service.StorageService.onChange(StorageService.java:419)
at org.apache.cassandra.gms.Gossiper.doNotifications(Gossiper.java:692)
at
org.apache.cassandra.gms.Gossiper.applyApplicationStateLocally(Gossiper.java:657)
at
org.apache.cassandra.gms.Gossiper.applyStateLocally(Gossiper.java:611)
at
org.apache.cassandra.gms.GossipDigestAckVerbHandler.doVerb(Gossiper.java:979)
at
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:38)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
ERROR [GMFD:2] 2010-01-19 22:22:36,667 CassandraDaemon.java (line 71) Fatal
exception in thread Thread[GMFD:2,5,main]
java.lang.RuntimeException: Bootstrap Token collision between /10.81.37.65 and
/10.81.37.52 (token 136112946768375385385349842972707284580
at
org.apache.cassandra.locator.TokenMetadata.addBootstrapToken(TokenMetadata.java:136)
at
org.apache.cassandra.service.StorageService.handleStateBootstrap(StorageService.java:456)
at
org.apache.cassandra.service.StorageService.onChange(StorageService.java:419)
at org.apache.cassandra.gms.Gossiper.doNotifications(Gossiper.java:692)
at
org.apache.cassandra.gms.Gossiper.applyApplicationStateLocally(Gossiper.java:657)
at
org.apache.cassandra.gms.Gossiper.applyStateLocally(Gossiper.java:611)
at
org.apache.cassandra.gms.GossipDigestAckVerbHandler.doVerb(Gossiper.java:979)
at
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:38)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
I traced the codes and find the reason is that :
public void addBootstrapToken(Token token, InetAddress endpoint)
{
......
oldEndPoint = bootstrapTokens.get(token);
if (oldEndPoint != null && !oldEndPoint.equals(endpoint))
throw new RuntimeException("Bootstrap Token collision between "
+ oldEndPoint + " and " + endpoint + " (token " + token);
// the exception is here --| :-)
v
oldEndPoint = tokenToEndPointMap.get(token);
if (oldEndPoint != null && !oldEndPoint.equals(endpoint))
throw new RuntimeException("Bootstrap Token collision between "
+ oldEndPoint + " and " + endpoint + " (token " + token);
......
}
In my enviroment, A is still in the tokenToEndPointMap though it's down. As a
result, A will not know B is bootstrapping and will not route any request to B
until B changes to NORMAL.
I think the following changes may be one way to deal with this issue :
if (oldEndPoint != null && !oldEndPoint.equals(endpoint) &&
FailureDetector.instance().isAlive(oldEndPoint))
throw new RuntimeException("Bootstrap Token collision between "
+ oldEndPoint + " and " + endpoint + " (token " + token);
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
