[
https://issues.apache.org/jira/browse/CASSANDRA-8318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14212751#comment-14212751
]
Jens Rantil edited comment on CASSANDRA-8318 at 11/14/14 8:04 PM:
------------------------------------------------------------------
bq. What actual token does .51 have?
It has a token of -2.
{noformat}
$ nodetool ring
Note: Ownership information does not include topology; for complete
information, specify a keyspace
Datacenter: Analytics
==========
Address Rack Status State Load Owns Token
6148914691236510000
X.X.X.52 rack1 Up Normal 18.34 GB 0.02%
-9223372036854770000
X.X.X.50 rack1 Up Normal 18.36 GB 0.24%
-6148914691236510000
X.X.X.55 rack1 Up Normal 18.51 GB 0.19%
-3074457345618250000
X.X.X.51 rack1 Down Normal 195.67 KB 0.02% -2
X.X.X.54 rack1 Up Normal 19.09 GB 0.04%
3074457345618258600
X.X.X.53 rack1 Up Normal 18.5 GB 0.07%
6148914691236510000
Datacenter: Cassandra
==========
Address Rack Status State Load Owns Token
9219239585832170071
...
Warning: "nodetool ring" is used to output all the tokens of a node.
To view status related info of a node use "nodetool status" instead.
{noformat}
was (Author: ztyx):
> What actual token does .51 have?
It has a token of -2.
{noformat}
$ nodetool ring
Note: Ownership information does not include topology; for complete
information, specify a keyspace
Datacenter: Analytics
==========
Address Rack Status State Load Owns Token
6148914691236510000
X.X.X.52 rack1 Up Normal 18.34 GB 0.02%
-9223372036854770000
X.X.X.50 rack1 Up Normal 18.36 GB 0.24%
-6148914691236510000
X.X.X.55 rack1 Up Normal 18.51 GB 0.19%
-3074457345618250000
X.X.X.51 rack1 Down Normal 195.67 KB 0.02% -2
X.X.X.54 rack1 Up Normal 19.09 GB 0.04%
3074457345618258600
X.X.X.53 rack1 Up Normal 18.5 GB 0.07%
6148914691236510000
Datacenter: Cassandra
==========
Address Rack Status State Load Owns Token
9219239585832170071
...
Warning: "nodetool ring" is used to output all the tokens of a node.
To view status related info of a node use "nodetool status" instead.
{noformat}
> Unable to replace a node
> ------------------------
>
> Key: CASSANDRA-8318
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8318
> Project: Cassandra
> Issue Type: Bug
> Environment: 2.0.8.39 (Datastax DSE 4.5.3)
> Reporter: Jens Rantil
> Attachments: X.X.X.56.log
>
>
> Had a hardware failure of a node. I followed the Datastax documentation[1] on
> how to replace the node X.X.X.51 using a brand new node with the same IP.
> Since it didn't come up after waiting for ~5 minutes or so, I decided to
> replace X.X.X.51 with a brand new unused IP X.X.X.56 instead. It now seems
> like my gossip is some weird state. When I start the replacement node I see
> line like
> {noformat}
> INFO [GossipStage:1] 2014-11-14 14:57:03,025 Gossiper.java (line 901)
> InetAddress /X.X.X.51 is now DOWN
> INFO [GossipStage:1] 2014-11-14 14:57:03,042 Gossiper.java (line 901)
> InetAddress /X.X.X.56 is now DOWN
> {noformat}
> . The latter is somewhat surprising since that is the IP of the actual
> replacement node. It doesn't surprise me it can't talk to itself if it hasn't
> started!
> Eventually the replacement node shuts down with
> {noformat}
> ERROR [main] 2014-11-14 14:58:06,031 CassandraDaemon.java (line 513)
> Exception encountered during startup
> java.lang.UnsupportedOperationException: Cannot replace token -2 which does
> not exist!
> at
> org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:782)
> at
> org.apache.cassandra.service.StorageService.initServer(StorageService.java:614)
> at
> org.apache.cassandra.service.StorageService.initServer(StorageService.java:503)
> at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:378)
> at com.datastax.bdp.server.DseDaemon.setup(DseDaemon.java:374)
> at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:496)
> at com.datastax.bdp.server.DseDaemon.main(DseDaemon.java:615)
> INFO [Thread-2] 2014-11-14 14:58:06,035 DseDaemon.java (line 461) DSE
> shutting down...
> INFO [StorageServiceShutdownHook] 2014-11-14 14:58:06,037 Gossiper.java
> (line 1307) Announcing shutdown
> INFO [Thread-2] 2014-11-14 14:58:06,046 PluginManager.java (line 355) All
> plugins are stopped.
> INFO [Thread-2] 2014-11-14 14:58:06,047 CassandraDaemon.java (line 463)
> Cassandra shutting down...
> ERROR [Thread-2] 2014-11-14 14:58:06,047 CassandraDaemon.java (line 199)
> Exception in thread Thread[Thread-2,5,main]
> java.lang.NullPointerException
> at
> org.apache.cassandra.service.CassandraDaemon.stop(CassandraDaemon.java:464)
> at com.datastax.bdp.server.DseDaemon.stop(DseDaemon.java:464)
> at com.datastax.bdp.server.DseDaemon$1.run(DseDaemon.java:364){noformat}
> All nodes are showing
> {noformat}
> root@machine-2:~# nodetool status company
> Datacenter: Analytics
> =====================
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> -- Address Load Tokens Owns (effective) Host ID
> Rack
> UN X.X.X.50 18.35 GB 1 16.7%
> 25efdbcd-14d3-4e9c-803a-3db5795d4efa rack1
> DN X.X.X.51 195.67 KB 1 16.7%
> d97cf86f-bfaf-4488-b716-26d71635a8fc rack1
> UN X.X.X.52 18.7 GB 1 16.7%
> caa32f68-5a6b-4d87-80bd-baa66a9b61ce rack1
> UN X.X.X.53 18.56 GB 1 16.7%
> e219321e-a6d5-48c4-9bad-d2e25429b1d2 rack1
> UN X.X.X.54 19.69 GB 1 16.7%
> 3cd36895-ee47-41c1-a5f5-41cb0f8526a6 rack1
> UN X.X.X.55 18.88 GB 1 16.7%
> 7d3f73c4-724e-45a6-bcf9-d3072dfc157f rack1
> Datacenter: Cassandra
> =====================
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> -- Address Load Tokens Owns (effective) Host ID
> Rack
> UN X.X.X.33 128.95 GB 256 100.0%
> 871968c9-1d6b-4f06-ba90-8b3a8d92dcf0 rack1
> UN X.X.X.32 115.3 GB 256 100.0%
> d7cacd89-8613-4de5-8a5e-a2c53c41ea45 rack1
> UN X.X.X.31 130.45 GB 256 100.0%
> 48cb0782-6c9a-4805-9330-38e192b6b680 rack1
> {noformat}
> , but when X.X.X.56 is starting is shows
> {noformat}
> root@machine-1:/var/lib/cassandra# nodetool status
> Note: Ownership information does not include topology; for complete
> information, specify a keyspace
> Datacenter: Analytics
> =====================
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> -- Address Load Tokens Owns Host ID
> Rack
> UN X.X.X.50 18.41 GB 1 0.2% 25efdbcd-14d3-4e9c-803a-3db5795d4efa
> rack1
> UN X.X.X.52 19.07 GB 1 0.0% caa32f68-5a6b-4d87-80bd-baa66a9b61ce
> rack1
> UN X.X.X.53 18.65 GB 1 0.1% e219321e-a6d5-48c4-9bad-d2e25429b1d2
> rack1
> UN X.X.X.54 19.69 GB 1 0.0% 3cd36895-ee47-41c1-a5f5-41cb0f8526a6
> rack1
> UN X.X.X.55 18.97 GB 1 0.2% 7d3f73c4-724e-45a6-bcf9-d3072dfc157f
> rack1
> Datacenter: Cassandra
> =====================
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> -- Address Load Tokens Owns Host ID
> Rack
> UN X.X.X.33 129.72 GB 256 21.7% 871968c9-1d6b-4f06-ba90-8b3a8d92dcf0
> rack1
> UN X.X.X.32 116 GB 256 12.4% d7cacd89-8613-4de5-8a5e-a2c53c41ea45
> rack1
> UN X.X.X.31 130.62 GB 256 65.3% 48cb0782-6c9a-4805-9330-38e192b6b680
> rack1
> {noformat}
> The above cluster state does not seem to replicate to the rest of the cluster
> (hasn't so far).
> Any input on how I can restore world order is appreciated.
> [1]
> http://www.datastax.com/documentation/cassandra/2.0/cassandra/operations/ops_replace_node_t.html
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
