[
https://issues.apache.org/jira/browse/CASSANDRA-21476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18091646#comment-18091646
]
Abe Ratnofsky commented on CASSANDRA-21476:
-------------------------------------------
Looks like a valid race to me, thanks for catching! Are you able to author a
patch and I can +1?
> Possible race condition in CassandraRoleManager#invalidRoleDisconnectTask
> -------------------------------------------------------------------------
>
> Key: CASSANDRA-21476
> URL: https://issues.apache.org/jira/browse/CASSANDRA-21476
> Project: Apache Cassandra
> Issue Type: Bug
> Reporter: Stefan Miklosovic
> Priority: Normal
>
> I found this when I was running multiplexer for CASSANDRA-21262 and I saw
> {{disconnectsAttemptedOnPeriodWithJitter}} failing occasionally.
> The bellow is the dump from Claude:
> This is a genuine production race condition in the self-rescheduling task,
> not just a flaky-test-timing issue. Here's the mechanism.
> How the task works
> The disconnect task reschedules itself (CassandraRoleManager.java:857-871):
> {code}
> protected void invalidRoleDisconnectTask(LongSupplier delayMillis,
> ScheduledExecutorService executor) {
> disconnectInvalidRoles(); // ← increments the
> counter in the test
> long nextDelayMillis = delayMillis.getAsLong();
> this.invalidRoleDisconnectTask = executor.schedule( // ← reschedules
> ITSELF, overwrites the field
> () -> invalidRoleDisconnectTask(delayMillis, executor),
> nextDelayMillis, MILLISECONDS);
> }
> {code}
> And stopping it (via setInvalidClientDisconnectPeriodMillis(0) →
> scheduleDisconnectInvalidRoleTask(), lines 873-895):
> {code}
> if (this.invalidRoleDisconnectTask != null)
> this.invalidRoleDisconnectTask.cancel(true); // cancel whatever
> future is in the field
> long period = getInvalidClientDisconnectPeriodMillis();
> if (period <= 0) { logger.info("...disabled"); return; } // period 0 →
> stop here
> {code}
> The race (lost cancel → orphaned chain)
> Two threads touch this.invalidRoleDisconnectTask with no shared lock: the
> test thread (calling setPeriod(0)) and the scheduler thread (running a task
> that's mid-flight). The
> breaking interleaving:
> 1. Scheduler thread is inside invalidRoleDisconnectTask, has already
> executed disconnectInvalidRoles() but not yet reached line 870. The field
> still holds the currently running
> future F1.
> 2. Test thread runs scheduleDisconnectInvalidRoleTask(): reads F1, calls
> F1.cancel(true) — but F1 is already running, so cancel does nothing useful —
> then sees period == 0 and
> returns.
> 3. Scheduler thread now executes line 870: this.invalidRoleDisconnectTask =
> executor.schedule(...F2...) — installs a fresh future F2 and overwrites the
> field.
> 4. F2 was never cancelled (the test thread already finished and is gone).
> It fires, runs disconnectInvalidRoles(), reschedules F3, and so on — the
> chain runs forever.
> Two things make it worse:
> - The period is captured by value in the closure. delayMillis = () ->
> period + ThreadLocalRandom... (line 889) closes over the local period. So an
> escaped chain keeps using its
> original 100 ms period regardless of the later change to 0 — only
> cancellation can stop it, and cancellation is exactly what was lost.
> - cancel(true) can't unwind a self-reschedule. Cancelling future Fn doesn't
> stop an already-running Fn from scheduling Fn+1
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
