Stefan Miklosovic created CASSANDRA-21476:
---------------------------------------------
Summary: Possible race condition in
CassandraRoleManager#invalidRoleDisconnectTask
Key: CASSANDRA-21476
URL: https://issues.apache.org/jira/browse/CASSANDRA-21476
Project: Apache Cassandra
Issue Type: Bug
Reporter: Stefan Miklosovic
I found this when I was running multiplexer for CASSANDRA-21262 and I saw
{{disconnectsAttemptedOnPeriodWithJitter}} failing occasionally.
The bellow is the dump from Claude:
This is a genuine production race condition in the self-rescheduling task, not
just a flaky-test-timing issue. Here's the mechanism.
How the task works
The disconnect task reschedules itself (CassandraRoleManager.java:857-871):
{code}
protected void invalidRoleDisconnectTask(LongSupplier delayMillis,
ScheduledExecutorService executor) {
disconnectInvalidRoles(); // ← increments the
counter in the test
long nextDelayMillis = delayMillis.getAsLong();
this.invalidRoleDisconnectTask = executor.schedule( // ← reschedules
ITSELF, overwrites the field
() -> invalidRoleDisconnectTask(delayMillis, executor),
nextDelayMillis, MILLISECONDS);
}
{code}
And stopping it (via setInvalidClientDisconnectPeriodMillis(0) →
scheduleDisconnectInvalidRoleTask(), lines 873-895):
{code}
if (this.invalidRoleDisconnectTask != null)
this.invalidRoleDisconnectTask.cancel(true); // cancel whatever future
is in the field
long period = getInvalidClientDisconnectPeriodMillis();
if (period <= 0) { logger.info("...disabled"); return; } // period 0 → stop
here
{code}
The race (lost cancel → orphaned chain)
Two threads touch this.invalidRoleDisconnectTask with no shared lock: the
test thread (calling setPeriod(0)) and the scheduler thread (running a task
that's mid-flight). The
breaking interleaving:
1. Scheduler thread is inside invalidRoleDisconnectTask, has already executed
disconnectInvalidRoles() but not yet reached line 870. The field still holds
the currently running
future F1.
2. Test thread runs scheduleDisconnectInvalidRoleTask(): reads F1, calls
F1.cancel(true) — but F1 is already running, so cancel does nothing useful —
then sees period == 0 and
returns.
3. Scheduler thread now executes line 870: this.invalidRoleDisconnectTask =
executor.schedule(...F2...) — installs a fresh future F2 and overwrites the
field.
4. F2 was never cancelled (the test thread already finished and is gone). It
fires, runs disconnectInvalidRoles(), reschedules F3, and so on — the chain
runs forever.
Two things make it worse:
- The period is captured by value in the closure. delayMillis = () -> period
+ ThreadLocalRandom... (line 889) closes over the local period. So an escaped
chain keeps using its
original 100 ms period regardless of the later change to 0 — only
cancellation can stop it, and cancellation is exactly what was lost.
- cancel(true) can't unwind a self-reschedule. Cancelling future Fn doesn't
stop an already-running Fn from scheduling Fn+1
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
