[
https://issues.apache.org/jira/browse/CASSANDRA-2496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13042254#comment-13042254
]
Brandon Williams commented on CASSANDRA-2496:
---------------------------------------------
Some explanation of what changed and why it was necessary:
Consider nodes A through D. D is partitioned, and C is dead and needs to be
removed. A removetoken will be issued to A for this.
In the current way we do things, A will modify it's own state by appending
information to its status indicating that it will be removing C. B will see
this, re-replicate as needed, then report to A that is is done. The problem
however, is that since A modified its own state, A is also free to wipe that
state out, either by restarting, or simple remove another token, because
there's only space for one. If A reboots and then D's partition heals, D will
never know C was removed. Worse, it will still have state for C that neither A
nor B do, and so it will repopulate the ring with C again.
This patch changes this by instead having A sleep for RING_DELAY to make sure
the generation for C is stable, and then it modifies C's state to indicate it
is being removed, just as if C itself had done this. It also appends some extra
state to indicate that A will be the removal coordinator. The others nodes see
this, re-replicate and report back to A, which then modifies C's state once
more to indicate it is completely removed. At this point, it doesn't matter if
A dies completely and D's partition heals, since the state is stored in C's
gossip information. If A reboots, it will be able to get the correct state
information from B, or any other node.
If A fails while the other nodes are re-replicating, a new removetoken can be
started elsewhere, or in the case of other replicas being down preventing
removetoken from completing, a removetoken force will remove the node and then
repair can be run to restore the replica count.
> Gossip should handle 'dead' states
> ----------------------------------
>
> Key: CASSANDRA-2496
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2496
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Reporter: Brandon Williams
> Assignee: Brandon Williams
> Attachments: 0001-Rework-token-removal-process.txt,
> 0002-add-2115-back.txt
>
>
> For background, see CASSANDRA-2371
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
