Brandon Williams created CASSANDRA-8260:
-------------------------------------------
Summary: Replacing a node can leave the old node in system.peers
on the replacement
Key: CASSANDRA-8260
URL: https://issues.apache.org/jira/browse/CASSANDRA-8260
Project: Cassandra
Issue Type: Bug
Components: Core
Reporter: Brandon Williams
Assignee: Brandon Williams
Fix For: 2.0.12
Here's what happens:
Nodes: X, Y, Z. Z replaces Y which is dead.
0. Replacement finishes
1. Z removes Y, quarantines and evicts (that is, removes the state)
2. X sees the replacement, quarantines, but keeps state
3. 60s elapses
4. quarantine on Z expires
5. X sends syn to Z, repopulates Y endpoint state and persists to system.peers,
but Z sees the conflict and does not update tMD for Y.
6. FatClient timer on Z starts counting.
7. quarantine on X expires, fat client has been idle, evicts and re-quarantines
8. 30s elapses
9. Fat client timeout occurs on Z, evicts and re-quarantines
10. 30s elapses
11. quarantine on X expires, so it never gets repopulated with Y since Z
already removed it
It's important to note here that there is a small but relevant gap between
steps 1 and 2, which then correlates to steps 4 and 5, and step 5 is where the
problem occurs. This also explains why it looks related to RING_DELAY, since
the quarantine is RING_DELAY * 2, but Y never quarantines and the fat client
timeout is RING_DELAY, effectively making the discrepancy near equal to
RING_DELAY in the end.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
