On Tue, Oct 27, 2009 at 10:02 PM, Jonathan Ellis <[email protected]> wrote: > On Tue, Oct 27, 2009 at 10:10 PM, Edmond Lau <[email protected]> wrote: >> I'd like to improve my mental model of how Cassandra bootstrapping >> works. My understanding is that bootstrapping is just an extra step >> during a node's startup where the node copies data from neighboring >> nodes that, according to its token, it should own; afterwards, the >> node behaves like any other node. >> >> If that's correct, I have a few follow-up questions: >> >> - At what point does the new node get inserted into the hash ring so >> that reads/writes for keys get directed to it? > > It starts accepting reads when bootstrap is complete, but during > bootstrap it will also be forwarded writes so it stays up-to-date.
Got it - good to know. > >> - What are the semantics of bootstrapping a node that's been in the >> cluster before and already has some data that's possibly outdated? >> Should this work? This might be useful if a node's been out of >> commission for sufficiently long period of time. > > Cluster membership is intended to be relatively stable. When removing > nodes is supported (see > https://issues.apache.org/jira/browse/CASSANDRA-435), it won't be > worth special-casing the case of "new node, that has some data locally > that is arbitrarily out of date." So you will want to wipe its data > directories and re-bootstrap it. Sounds reasonable. Until CASSANDRA-435 is complete, there's no way currently to take down a node and have it be removed from the list of nodes that's responsible for the data in its token range, correct? All other nodes will just assume that it's temporarily unavailable? > >> - If we pick a poor set of initial tokens, would it be sensible to >> modify the tokens on existing nodes and then restart them with >> bootstrapping in order to rebalance? > > Not until https://issues.apache.org/jira/browse/CASSANDRA-193 is done > so you can fix the damage you've done to your replicas. Assume that we had the ability to permanently remove a node. Would modifying the token on an existing node and restarting it with bootstrapping somehow be incorrect, or merely not performant b/c we'll be performing lazy repair on most reads until the node is up to date? > >> I've also noticed that I can get my cassandra cluster into a weird >> state via bootstrapping, where it stops accepting reads/writes. I'm >> on Cassandra 0.4.1. A simple repro case is to start all 3 nodes of a >> 3 node cluster (replication factor of 2) using bootstrapping. Getting >> a key that I've inserted then leads to an IndexOutOfBoundsException. >> Another IndexOutOfBoundsException was thrown later while flushing. > > Starting all nodes in bootstrap mode is not a supported operation. > Don't do that. (Bootstrap is much more automatic in trunk, and fixes > this specific problem as well as others.) The unsupported operation is that all nodes in the cluster are simultaneously in bootstrap mode, and not that all nodes in the cluster were bootstrapped at some point in time, right? A reasonable scenario that could cause the second situation would be if I wanted to migrate my cluster to a completely new set of machines. I would then bootstrap all the new nodes in the new cluster, and then decommission my old nodes one by one (assuming https://issues.apache.org/jira/browse/CASSANDRA-435 was done). After the migration, all my nodes would've been bootstrapped. Thanks for the info, Edmond > > -Jonathan >
