Re: Bringing a dead node back up after fixing hardware issues
Thanks Brandon. Thanks, Eran Chinthaka Withana On Thu, Jul 26, 2012 at 12:46 PM, Brandon Williams dri...@gmail.com wrote: On Wed, Jul 25, 2012 at 6:16 PM, Eran Chinthaka Withana eran.chinth...@gmail.com wrote: Alright, lets assume I want to go on this route. I have RF=2 in the data center and I believe I need at least RF=3 to set the replication to LOCAL_QUORUM and hide the node failures. But if I increase the RF to 3 now then won't it trigger more read misses until repair completes? Given this is a production cluster which can not afford downtime, how can we do this? Switch to LQ and increase the RF to 3, then repair to actually have the RF bumped up. As long as nothing fails during the first step (which should take perhaps minutes) you'll be ok. -Brandon
Re: Bringing a dead node back up after fixing hardware issues
On Wed, Jul 25, 2012 at 6:16 PM, Eran Chinthaka Withana eran.chinth...@gmail.com wrote: Alright, lets assume I want to go on this route. I have RF=2 in the data center and I believe I need at least RF=3 to set the replication to LOCAL_QUORUM and hide the node failures. But if I increase the RF to 3 now then won't it trigger more read misses until repair completes? Given this is a production cluster which can not afford downtime, how can we do this? Switch to LQ and increase the RF to 3, then repair to actually have the RF bumped up. As long as nothing fails during the first step (which should take perhaps minutes) you'll be ok. -Brandon
Re: Bringing a dead node back up after fixing hardware issues
Hi Brandon, Increasing CL is tricky for us for now, as our RF on that datacenter is 2 and CL is set to ONE. If we make the CL to be LOCAL_QUORUM, then, if a node goes down we will have trouble. I will try to increase the RF to 3 in that data center and set the CL to LOCAL_QUORUM if nothing works out. Increasing the RF and and using LOCAL_QUORUM is the right thing in this case. By choosing CL.ONE, you are agreeing that read misses are acceptable. If they are not, then adjusting your RF/CL is the only path. Alright, lets assume I want to go on this route. I have RF=2 in the data center and I believe I need at least RF=3 to set the replication to LOCAL_QUORUM and hide the node failures. But if I increase the RF to 3 now then won't it trigger more read misses until repair completes? Given this is a production cluster which can not afford downtime, how can we do this? Thanks, Eran
Re: Bringing a dead node back up after fixing hardware issues
On Mon, Jul 23, 2012 at 10:24 PM, Eran Chinthaka Withana eran.chinth...@gmail.com wrote: Thanks Brandon for the answer (and I didn't know driftx = Brandon Williams. Thanks for your awesome support in Cassandra IRC) Thanks :) Increasing CL is tricky for us for now, as our RF on that datacenter is 2 and CL is set to ONE. If we make the CL to be LOCAL_QUORUM, then, if a node goes down we will have trouble. I will try to increase the RF to 3 in that data center and set the CL to LOCAL_QUORUM if nothing works out. Increasing the RF and and using LOCAL_QUORUM is the right thing in this case. By choosing CL.ONE, you are agreeing that read misses are acceptable. If they are not, then adjusting your RF/CL is the only path. About decommissioning, if the node goes down. There is no way of knowing running that command on that node, right? IIUC, decommissioning should be run on a node that needs to be decommissioned. Well, decom and removetoken are both ways of removing a node. The former is for a live node, and the latter is for a dead node. Since your node was actually alive you could have decommissioned it. Coming back to the original question, without touching the CL, can we bring back a dead node (after fixing it) and somehow tell Cassandra that the node is backup and do not send read requests until it gets all the data? No, as I said, you are accepting this behavior by choosing CL.ONE. -Brandon
Bringing a dead node back up after fixing hardware issues
Hi, In my cluster, one of the nodes went down (due to a hardware failure). We managed to get it fixed in couple of days. But it seems its harder to bring this same node back into cluster without creating read misses. Here is what I did. Method 1: I copied the data from all the nodes in that data center, into the repaired node, and brought it back up. But because of the rate of updates happening, the read misses started going up. Method 2: I issued a removetoken command for that node's token and let the cluster stream the data into relevant nodes. At the end of this process, the dead node was not showing up in the ring output. Then I brought the node back up. I was expecting, Cassandra to first stream data into the new node (which happens to be the dead node which was in the cluster earlier) and once its done then make it serve reads. But, in the server log, I can see as soon the node comes up, it started serving reads, creating a large number of read misses. So the question is, what is the best way to bring back a dead node (once its hardware issues are fixed) without impacting read misses? Thanks, Eran Chinthaka Withana
Re: Bringing a dead node back up after fixing hardware issues
On Mon, Jul 23, 2012 at 6:26 PM, Eran Chinthaka Withana eran.chinth...@gmail.com wrote: Method 1: I copied the data from all the nodes in that data center, into the repaired node, and brought it back up. But because of the rate of updates happening, the read misses started going up. That's not really a good method when you scale up and the amount of data in the cluster won't fit on a single machine. Method 2: I issued a removetoken command for that node's token and let the cluster stream the data into relevant nodes. At the end of this process, the dead node was not showing up in the ring output. Then I brought the node back up. I was expecting, Cassandra to first stream data into the new node (which happens to be the dead node which was in the cluster earlier) and once its done then make it serve reads. But, in the server log, I can see as soon the node comes up, it started serving reads, creating a large number of read misses. Removetoken is for dead nodes, so the node has no way of locally knowing it shouldn't be a cluster member any longer when it starts up. Instead if you had decommissioned, it would have saved a flag to indicate it should bootstrap at the next startup. So the question is, what is the best way to bring back a dead node (once its hardware issues are fixed) without impacting read misses? Increase your consistency level. Run a repair on the node once it's back up, unless the repair time took longer than gc_grace, in which case you need to removetoken it, delete all the data, and bootstrap it back in if you don't want anything deleted to resurrect. -Brandon
Re: Bringing a dead node back up after fixing hardware issues
Thanks Brandon for the answer (and I didn't know driftx = Brandon Williams. Thanks for your awesome support in Cassandra IRC) Increasing CL is tricky for us for now, as our RF on that datacenter is 2 and CL is set to ONE. If we make the CL to be LOCAL_QUORUM, then, if a node goes down we will have trouble. I will try to increase the RF to 3 in that data center and set the CL to LOCAL_QUORUM if nothing works out. About decommissioning, if the node goes down. There is no way of knowing running that command on that node, right? IIUC, decommissioning should be run on a node that needs to be decommissioned. Coming back to the original question, without touching the CL, can we bring back a dead node (after fixing it) and somehow tell Cassandra that the node is backup and do not send read requests until it gets all the data? Thanks, Eran Chinthaka Withana On Mon, Jul 23, 2012 at 6:48 PM, Brandon Williams dri...@gmail.com wrote: On Mon, Jul 23, 2012 at 6:26 PM, Eran Chinthaka Withana eran.chinth...@gmail.com wrote: Method 1: I copied the data from all the nodes in that data center, into the repaired node, and brought it back up. But because of the rate of updates happening, the read misses started going up. That's not really a good method when you scale up and the amount of data in the cluster won't fit on a single machine. Method 2: I issued a removetoken command for that node's token and let the cluster stream the data into relevant nodes. At the end of this process, the dead node was not showing up in the ring output. Then I brought the node back up. I was expecting, Cassandra to first stream data into the new node (which happens to be the dead node which was in the cluster earlier) and once its done then make it serve reads. But, in the server log, I can see as soon the node comes up, it started serving reads, creating a large number of read misses. Removetoken is for dead nodes, so the node has no way of locally knowing it shouldn't be a cluster member any longer when it starts up. Instead if you had decommissioned, it would have saved a flag to indicate it should bootstrap at the next startup. So the question is, what is the best way to bring back a dead node (once its hardware issues are fixed) without impacting read misses? Increase your consistency level. Run a repair on the node once it's back up, unless the repair time took longer than gc_grace, in which case you need to removetoken it, delete all the data, and bootstrap it back in if you don't want anything deleted to resurrect. -Brandon
