You could try... - delete / move the system data directory - set the initial_token for each node to what they were before - restart and recreate the schema - run repair and then clean
It would have been a good idea to drain the nodes, this would checkpoint the logs and clear them. If you do not know the initial tokens, I would start a new empty node as suggested and do the the same. Hope that helps. Aaron On 27/04/2011, at 7:07 PM, maneela a <manee...@yahoo.com> wrote: > Hi, > > I had a 2 node cassandra cluster with replication factor 2 and > OrderPreservingPartitioner but we did not provide InitialToken in the > configuration files. One of the node was affected in the recent AWS EBS > outage and had been partitioned from cluster. However, I continued to allowed > all write operations to other survived node because I thought AWS could > recovered EBS issues with in 24 hours so Survived node might take care of > propagating 2nd replica from its hinted column family to the bad node when it > recovered from EBS issues. Unfortunately AWS had taken longer than we > expected almost 4 days. So instead of recovering 2nd node by playing hinted > CF from node1, I did the following sequence of events in order to get 2nd > node back to cluster > > 1) shut down cassandra service on good node > 2) removed all hinted CF files > 3) Taken EBS snapshot > 4) Launched new EBS volumes from above snapshot and mounted them on 2nd node > 5) Also copied commitlogs from node1 to node2 > in other words, I cloned node1 and mounted on node2, my assumption is cluster > with 2 nodes with replication 2 should likely be mirrored images > > 6) brought up service on both nodes > 7) I am not seeing both IP address as part of ring when I ran nodetool command > > root@domU-12-31-39-0F-CA-61:/mnt/logs/cassandra# nodetool -h localhost ring > Address Status Load Range > Ring > 10.193.201.139Up 434.77 GB RVtMj8gWiKG0baPy > |<--| > > root@ip-10-196-107-47:/data/cassandra/data/system# nodetool -h localhost ring > Address Status Load Range > Ring > 10.193.201.139Up 434.77 GB RVtMj8gWiKG0baPy > |<--| > > > I guess this behavior is happening because both nodes are having same data > including Locationinfo CF and commit logs as well. Can someone direct me what > should be done here to get both IPs as part of ring? > > Thanks > niru >
