Robert,
This is how I interpret the implications of the 3 steps you suggested.Please confirm my interpretation as this is really important. Suppose, gc grace seconds=10days, max hinted handoff period=3 hrs, 3 nodes are there A,B & C,RF =3 and my client is reading at CL ONE. C remains down for 5 hours and misses many updates including those which happened after max hinted handoff period of 3 hrs. Now I bring back node C with auto_bootstrap false and run repair. If client queries at CL ONE and fetches a row which got updated after max hinted handoff period, there is a very high possibility of client returning stale data from node C . But as soon as node C has joined the ring, it will start participating in WRITEs. But if we follow the procedure you suggested, node C will come back, run repair but wont participate in reads till we join it to the cluster. During repair, if client queries at CL ONE and fetches a row which got updated after max hinted handoff period expired and was missed by node C, it will still get latest data from A and B. So, the integrity of data is not lost similar to the case when we auto_bootsrap with true. Additionally we save the unique data of node C. While repair is going on, node C will get all the Writes. @Anishek Hinted handoffs are not related to gc grace seconds. Thanks Anuj Sent from Yahoo Mail on Android From:"Anishek Agarwal" <anis...@gmail.com> Date:Wed, 18 Nov, 2015 at 1:49 pm Subject:Re: handling down node cassandra 2.0.15 @Rob interesting something i will try next time, for step 3 you mentioned -- I just remove the -Dcassandra.join_ring=false option and restart the cassandra service? @Anuj, gc_grace_seconds dictates how long hinted handoff are stored right. These might be good where we explicitly delete values from the table. we just have ttl and DTCS should delete data older than 1 month. In this case do i need to wipe the node and then start copy of key space again ? or can i run a repair once it joins the right with auto_bootstrap=false. On Wed, Nov 18, 2015 at 1:20 AM, Robert Coli <rc...@eventbrite.com> wrote: On Tue, Nov 17, 2015 at 4:33 AM, Anuj Wadehra <anujw_2...@yahoo.co.in> wrote: Only if gc_grace_seconds havent passed since the failure. If your machine is down for more than gc_grace_seconds you need to delete the data directory and go with auto bootstrap = true . Since CASSANDRA-6961 you can : 1) bring up the node with join_ring=false 2) repair it 3) join it to the cluster https://issues.apache.org/jira/browse/CASSANDRA-6961 This prevents you from decreasing your unique replica count, which is usually a good thing! =Rob
