Probably, I needed to be clearer in my inquiry …. I’m investigating a situation where our diagnostic data is telling us that C* has lost some of the application data. I mean, getsstables for the data returns zero on all nodes in all racks.
The last pickle article below & Jeff Jirsa had described a situation where bootstrapping a node to extend the cluster can loose data if this new node bootstraps from a stale SECONDARY replica (node that was offline > hinted had-off window). This was fixed in cassandra-2434. http://thelastpickle.com/blog/2017/05/23/auto-bootstrapping-part1.html The article & the Jira above describe bootstrapping when extending a cluster. I understand replacing a dead node does not involve range movement, but will the above Jira fix prevent the bootstrap process when a replacing a dead node from using secondary replica? Thanks ---------------- Thank you From: Fred Habash Sent: Wednesday, May 1, 2019 6:50 AM To: user@cassandra.apache.org Subject: Re: Bootstrapping to Replace a Dead Node vs. Adding a New Node:Consistency Guarantees Thank you. Range movement is one reason this is enforced when adding a new node. But, what about forcing a consistent bootstrap i.e. bootstrapping from primary owner of the range and not a secondary replica. How’s consistent bootstrap enforced when replacing a dead node. ————- Thank you. On Apr 30, 2019, at 7:40 PM, Alok Dwivedi <alok.dwiv...@instaclustr.com> wrote: When a new node joins the ring, it needs to own new token ranges. This should be unique to the new node and we don’t want to end up in a situation where two nodes joining simultaneously can own same range (and ideally evenly distributed). Cassandra has this 2 minute wait rule for gossip state to propagate before a node is added. But this on its does not guarantees that token ranges can’t overlap. See this ticket for more details https://issues.apache.org/jira/browse/CASSANDRA-7069 To overcome this issue, the approach was to only allow one node joining at a time. When you replace a dead node the new token range selection does not applies as the replacing node just owns the token ranges of the dead node. I think that’s why the restriction of only replacing one node at a time does not applies in this case. Thanks Alok Dwivedi Senior Consultant https://www.instaclustr.com/platform/ From: Fd Habash <fmhab...@gmail.com> Reply-To: "user@cassandra.apache.org" <user@cassandra.apache.org> Date: Wednesday, 1 May 2019 at 06:18 To: "user@cassandra.apache.org" <user@cassandra.apache.org> Subject: Bootstrapping to Replace a Dead Node vs. Adding a New Node: Consistency Guarantees Reviewing the documentation & based on my testing, using C* 2.2.8, I was not able to extend the cluster by adding multiple nodes simultaneously. I got an error message … Other bootstrapping/leaving/moving nodes detected, cannot bootstrap while cassandra.consistent.rangemovement is true I understand this is to force a node to bootstrap from the former owner of the range when adding a node as part of extending the cluster. However, I was able to bootstrap multiple nodes to replace dead nodes. C* did not complain about it. Is consistent range movement & the guarantee it offers to bootstrap from primary range owner not applicable when bootstrapping to replace dead nodes? ---------------- Thank you
