One of the nodes in a cassandra cluster has died.
I'm using cassandra 2.0.7 throughout.
When I do a nodetool status this is what I see (real addresses have been
replaced with fake 10 nets)
[root@beta-new:/opt] #nodetool status
Datacenter: datacenter1
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns Host ID
Rack
UN 10.10.1.94 171.02 KB 256 49.4%
fd2f76ae-8dcf-4e93-a37f-bf1e9088696e rack1
DN 10.10.1.98 ? 256 50.6%
f2a48fc7-a362-43f5-9061-4bb3739fdeaf rack1
I tried to get the token ID for the down node by doing a nodetool ring
command, grepping for the IP and doing a head -1 to get the initial one.
[root@beta-new:/opt] #nodetool ring | grep 10.10.1.98 | head -1
10.10.1.98 rack1 Down Normal ? 50.59%
-9042969066862165996
I then started following this documentation on how to replace the node:
[
http://www.datastax.com/documentation/cassandra/2.0/cassandra/operations/ops_replace_node_t.html?scroll=task_ds_aks_15q_gk][1
]
So I installed cassandra on a new node but did not start it.
Set the following options:
cluster_name: 'Test Cluster'
seed_provider:
- seeds: "10.10.1.94"
listen_address: 10.10.1.94
endpoint_snitch: SimpleSnitch
And set the initial token of the new install as the token -1 of the node
I'm trying to replace in cssandra.yaml:
initial_token: -9042969066862165995
And after making sure there was no data yet in:
/var/lib/cassandra
I started up the database:
[root@web2:/etc/alternatives/cassandrahome] #./bin/cassandra -f
-Dcassandra.replace_address=10.10.1.98
The documentation I link to above says to use the replace_address directive
on the command line rather than cassandra-env.sh if you have a tarball
install (which we do) as opposed to a package install.
After I start it up, cassandra fails with the following message:
Exception encountered during startup: Cannot replace_address /
10.10.10.98 because it doesn't exist in gossip
So I'm wondering at this point if I've missed any steps or if there is
anything else I can try to replace this dead cassandra node?
