Hi, We are testing the process of adding a node to a cluster using a simple procedure, and seeing data loss.
System: Ubuntu 12.04 on AWS Version: Cassandra + dsc 1.2.10 Here is what we did: Created 2 new m1.large instances Installed Java Installed Cassandra 1.2.10 (the version we are using in our production system) In server1: Adjusted cassandra.yaml comment out the initial_token uncomment num_tokens: 256 changed "seeds" to the address of this server changed listen_address to the address of this server changed rpc_address to 0.0.0.0 changed practitioner to org.apache.cassandra.dht.RandomPartitioner made sure cassandra is not running sudo rm -rf /var/lib/cassandra/* started cassandra connected via cqlsh Created a new keyspace with replication factor 1 Created a new table Populated the table with 4000 row of simple data using cql copy command cqlsh> select count(*) - returns 4000 nodetool status shows a single server at this point (using vnodes) In server2: made sure cassandra is not running sudo rm -rf /var/lib/cassandra/* Adjusted cassandra.yaml as above except that for seeds put both server addresses started cassandra waited a couple of min What we found: nodetool status on either server shows two servers, each with appox 50% (but not exactly) cqlsh>select count(*) - return 1870 (on either server) This process was repeated 3 times. each time the number was a bit different, but ~2000 Notes Replication factor is 1. No nodetool cleanup was run We have successfully added nodes in the past, but not since we moved to using vnodes THIS WAS A TEST. CLEAN MACHINES, SIMPLE DATA - What are we doing wrong? Thanks, Tamar Rosen Senior Data Architect Correlor.com
