On Sat, Jul 20, 2013 at 7:30 AM, E S <tr1skl...@yahoo.com> wrote: > I am trying to understand the best procedure for adding new nodes. The > one that I see most often online seems to have a hole where there is a low > probability of permanently losing data. I want to understand what I am > missing in my understanding. >
You aren't missing anything. Congratulations, you have deduced the following bug : https://issues.apache.org/jira/browse/CASSANDRA-2434 "Range movements violate consistency" I don't think this problem would normally happen without vnodes, because > when doubling you would alternate the new nodes with the old nodes in the > ring, so while quorum might not work until the final repair, "all" would, > and a repair would solve the problem. With vnodes though, some of the > ranges will follow the pattern above (range ownership moving from A,B,C to > D,E,F). > I believe this is correct, though adding nodes "between" other nodes is just convention, there is nothing actually keeping you from being exposed to 2434 if you do not do this. > Am I missing something here? If I'm right, I think the only way to avoid > this is adding less then a quorum of new nodes (in this case 1) before > doing a repair. That would be painful since repairs take a while. > If you read/write at QUORUM, this is true. If you read/write at ONE, my reading of 2434 is that any range movement is capable of losing data because your data may be only on the node which has lost the range at the end of the movement, which is not necessarily the host streaming its keys to the bootstrapping node. Even running a repair before the movement only reduces the chance of loss, it does not eliminate it. =Rob
