Szymon Miezal created CASSANDRA-18824:
-----------------------------------------
Summary: Cleanup behaviour during node decommission caused missing
replica
Key: CASSANDRA-18824
URL: https://issues.apache.org/jira/browse/CASSANDRA-18824
Project: Cassandra
Issue Type: Bug
Components: Local/SSTable
Reporter: Szymon Miezal
Node decommission triggers data transfer to other nodes. While this transfer is
in progress,
receiving nodes temporarily hold token ranges in a pending state. However, the
cleanup process currently doesn't consider these pending ranges when
calculating token ownership.
As a consequence, data that is already stored in sstables gets inadvertently
cleaned up.
STR:
* Create two node cluster
* Create keyspace with RF=1
* Insert sample data (assert data is available when querying both nodes)
* Start decommission process of node 1
* Start running cleanup in a loop on node 2 until decommission on node 1
finishes
* Verify of all rows are in the cluster - it will fail as the previous step
removed some of the rows
It seems that the cleanup process does not take into account the pending
ranges, it uses only the local ranges -
https://github.com/apache/cassandra/blob/caad2f24f95b494d05c6b5d86a8d25fbee58d7c2/src/java/org/apache/cassandra/db/compaction/CompactionManager.java#L466.
There are two solutions to the problem.
One would be to change the cleanup process in a way that it start taking
pending ranges into account. Even thought it might sound tempting at first it
will require involving changes and a lot of testing effort.
Alternatively we could interrupt/prevent the cleanup process from running when
any pending range on a node is detected. That sounds like a reasonable
alternative to the problem and something that is relatively easy to implement.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
