[
https://issues.apache.org/jira/browse/CASSANDRA-16418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17675737#comment-17675737
]
Paulo Motta commented on CASSANDRA-16418:
-----------------------------------------
In order to check the tests were reliably reproducing the issue on Lindsey's
[branch|https://github.com/apache/cassandra/pull/2061] I commented out the
following excerpt:
{noformat}
InetAddressAndPort localAddress =
FBUtilities.getBroadcastAddressAndPort();
Integer pendingRangesCount =
tokenMetadata.getPendingRanges(keyspaceName, localAddress).size();
if (pendingRangesCount > 0)
{
throw new RuntimeException("Node is involved in cluster membership
changes. Not safe to run cleanup.");
}
{noformat}
And expected both
[testCleanupFailsDuringOngoingDecommission|https://github.com/apache/cassandra/pull/2061/files#diff-68d2cd75caa0e4091c7206717116594bdcb0aab38f72f6d6afa44eac60466e13R41]
and
[testCleanupFailsDuringOngoingBootstrap|https://github.com/apache/cassandra/pull/2061/files#diff-68d2cd75caa0e4091c7206717116594bdcb0aab38f72f6d6afa44eac60466e13R85]
to fail.
Even though the tests failed most of the time, sometimes the tests passed so
data was not being wrongly cleaned up as expected.
The reason for this is that these tests require that the cleanup is executed
between the sstables are transferred by streaming and the ring membership
operation is finished. There is a small chance cleanup is not executed within
this window so the issue will not reproduce, especially if we run this test on
faster hardware.
I took a slightly different testing approach on [this
commit|https://github.com/pauloricardomg/cassandra/blob/702f77d247893a51461823268ad6a20cd6c1a021/test/distributed/org/apache/cassandra/distributed/test/ring/CleanupFailureTest.java#L40]
that inserts data while a node is bootstrapping or decommissioning and checks
the data is present after a cleanup is run. This was able to reliably reproduce
the issue when the excerpt above is commented out.
The updated test is more deterministic because we don't depend on streaming nor
timing. Furthermore this makes the test faster since we don't need so many rows
to reproduce the issue, which is needed with the streaming approach.
A nice benefit of this approach is that since we only run cleanup a single time
while the node is bootstrapping/decommissioning, we're able to [verify that the
cleanup fails with the expected error
message|https://github.com/pauloricardomg/cassandra/blob/702f77d247893a51461823268ad6a20cd6c1a021/test/distributed/org/apache/cassandra/distributed/test/ring/CleanupFailureTest.java#L105].
> Unsafe to run nodetool cleanup during bootstrap or decommission
> ---------------------------------------------------------------
>
> Key: CASSANDRA-16418
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16418
> Project: Cassandra
> Issue Type: Bug
> Components: Consistency/Bootstrap and Decommission
> Reporter: James Baker
> Assignee: Lindsey Zurovchak
> Priority: Normal
> Fix For: 4.0.x
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> What we expected: Running a cleanup is a safe operation; the result of
> running a query after a cleanup should be the same as the result of running a
> query before a cleanup.
> What actually happened: We ran a cleanup during a decommission. All the
> streamed data was silently deleted, the bootstrap did not fail, the cluster's
> data after the decommission was very different to the state before.
> Why: Cleanups do not take into account pending ranges and so the cleanup
> thought that all the data that had just been streamed was redundant and so
> deleted it. We think that this is symmetric with bootstraps, though have not
> verified.
> Not sure if this is technically a bug but it was very surprising (and
> seemingly undocumented) behaviour.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
