[
https://issues.apache.org/jira/browse/CASSANDRA-11892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15353301#comment-15353301
]
Joel Knighton commented on CASSANDRA-11892:
-------------------------------------------
I suspect that it is another race. This is part of a set of tickets [~dikanggu]
reported at roughly the same time, and there were some cascading failures that
could have hit an interesting race here. For the record, these are still on my
radar if very slow to get to - my apologies.
> Can not replace a dead host
> ---------------------------
>
> Key: CASSANDRA-11892
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11892
> Project: Cassandra
> Issue Type: Bug
> Reporter: Dikang Gu
> Attachments: 0001-handle-hibernate-case.patch
>
>
> I got some errors when trying to replace a dead host.
> {code}
> 2016-05-25_20:59:37.61838 ERROR 20:59:37 [main]: Exception encountered during
> startup
> 2016-05-25_20:59:37.61839 java.lang.UnsupportedOperationException: Cannot
> replace token 100284002935427428580945058996711341062 which does not exist!
> 2016-05-25_20:59:37.61839 at
> org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:925)
> ~[apache-cassandra-2.1.14+git20160523.7442267.jar:2.1.14+git20160523.7442267]
> 2016-05-25_20:59:37.61839 at
> org.apache.cassandra.service.StorageService.initServer(StorageService.java:740)
> ~[apache-cassandra-2.1.14+git20160523.7442267.jar:2.1.14+git20160523.7442267]
> 2016-05-25_20:59:37.61839 at
> org.apache.cassandra.service.StorageService.initServer(StorageService.java:617)
> ~[apache-cassandra-2.1.14+git20160523.7442267.jar:2.1.14+git20160523.7442267]
> 2016-05-25_20:59:37.61840 at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:389)
> [apache-cassandra-2.1.14+git20160523.7442267.jar:2.1.14+git20160523.7442267]
> 2016-05-25_20:59:37.61840 at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:564)
> [apache-cassandra-2.1.14+git20160523.7442267.jar:2.1.14+git20160523.7442267]
> 2016-05-25_20:59:37.61841 at
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:653)
> [apache-cassandra-2.1.14+git20160523.7442267.jar:2.1.14+git20160523.7442267]
> 2016-05-25_20:59:37.61910 Exception encountered during startup: Cannot
> replace token 100284002935427428580945058996711341062 which does not exist!
> {code}
> the status of the node is DN:
> {code}
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> -- Address Load Tokens Owns Host ID
> Rack
> DN 2401:db00:2050:4196:face:0:13:0 809.83 GB 256 ? null
> ash5-04-pp
> {code}
> I add some logging and find something like this:
> {code}
> 2016-05-25_20:58:33.44305 INFO 20:58:33 [main]: Gathering node replacement
> information for /2401:db00:2050:4196:face:0:13:0
> 2016-05-25_20:58:34.36966 INFO 20:58:34 [GossipStage:1]: InetAddress
> /2401:db00:2050:4196:face:0:13:0 is now DOWN
> 2016-05-25_20:58:41.12167 INFO 20:58:41 [GossipStage:1]: InetAddress
> /2401:db00:2050:4196:face:0:13:0 is now DOWN
> 2016-05-25_20:58:41.12248 INFO 20:58:41 [GossipStage:1]: Node
> /2401:db00:2050:4196:face:0:13:0 state STATUS
> 2016-05-25_20:58:41.12250 INFO 20:58:41 [GossipStage:1]: Node
> /2401:db00:2050:4196:face:0:13:0 movename hibernate
> 2016-05-25_20:58:41.12252 INFO 20:58:41 [GossipStage:1]: Node
> /2401:db00:2050:4196:face:0:13:0 state LOAD
> {code}
> I find in the StorageService.onChange, we do not handle the "hibernate"
> VersionValue, does it cause the problem?
> Is it safe to apply the patch to fix it?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
