[
https://issues.apache.org/jira/browse/CASSANDRA-21819?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Artur Malchanau updated CASSANDRA-21819:
----------------------------------------
Description:
h3. Summary
When an auto-repair task exceeds {{repair_session_timeout{}}}, AutoRepair
"cancels" it with {{Future.cancel(true)}} and retries the same ranges.
That cancel is a no-op: {{RepairCoordinator.run()}} only wires an asynchronous
future chain and returns right away, so the executor future is already done and
{{cancel()}} returns {{false}}.
The repair keeps running, or keeps hanging, in the background:
* the incremental repair (IR) session stays {{REPAIRING}} on every
participant, and its sstables stay {{pendingRepair}};
* every retry, and every later incremental repair of the same ranges, fails in
the prepare phase with {{...encountered intersecting sstables belonging to
another incremental repair session (<old id>)}};
* the {{RepairSession}} stays registered on the coordinator.
Nothing ever cleans this up:
* LocalSessions never auto-fails the session, because the participants keep
answering each other's status requests with {{REPAIRING}}, and since
CASSANDRA-20312 each such answer refreshes {{lastUpdate}};
* the coordinator has no timeout of its own on merkle-tree responses.
In our production cluster the session stayed hung for 7 days, until an operator
ran {{nodetool repair_admin cancel}}. Incremental repair of the affected ranges
was blocked for that whole time.
h3. Root cause
[AutoRepair.java (5.0.9), lines
344-398|https://github.com/apache/cassandra/blob/cassandra-5.0.9/src/java/org/apache/cassandra/repair/autorepair/AutoRepair.java#L344-L398]:
{code:java}
f = repairRunnableExecutors.get(repairType).submit(task); // task =
RepairCoordinator
listener.await(config.getRepairSessionTimeout(repairType)); // times out ->
success = false
...
boolean cancellationStatus = f.cancel(true); // always false:
run() already returned
// sleep repair_retry_backoff, then submit a NEW RepairCoordinator for the same
ranges
{code}
[RepairCoordinator.run()/runMayThrow()|https://github.com/apache/cassandra/blob/cassandra-5.0.9/src/java/org/apache/cassandra/repair/RepairCoordinator.java#L273-L309]
starts prepare -> repair -> finalize as a future chain and returns. Nothing in
AutoRepair ever fails the parent repair, its {{{}RepairSession}}s, or the IR
{{CoordinatorSession{}}}. As a result, no {{FailSession}} is sent and the
participants keep the sstables in {{{}pendingRepair{}}}.
The same logic is present on the current {{cassandra-5.0}}, {{cassandra-6.0}}
and {{trunk}} branches.
Trunk only adds an {{isShutDown}} branch, which uses the same
{{f.cancel(true)}}.
h3. How the session got stuck in production (trigger; not the bug itself)
5.0.9, three DCs, incremental auto-repair, {{repair.retries}} not configured
(default: disabled).
* 10:16:48 UTC: the coordinator (node C) starts IR session
{{8681eb40-cd38-11f1-b81f-4153cf3313e6}} with 9 participants. Prepare succeeds,
and all 54 VALIDATION_REQs (6 tables x 9 replicas) are acknowledged.
* 10:17:00: the LARGE_MESSAGES connections from two remote-DC replicas to node
C fail to connect ({{connection timed out after 2000 ms}}). While a connection
is failing to connect, {{OutboundConnection}} rejects any message that doesn't
fit the per-connection send queue, so three 8 MiB merkle trees are dropped:
{noformat}
WARN [AntiEntropyStage:1] 2026-10-21T10:17:00,569 NoSpamLogger.java:107 -
/R1:7000->/C:7000-LARGE_MESSAGES-[no-channel] overloaded; dropping 8.019MiB
message (queue: 4.488MiB local, 500.008KiB endpoint, 500.008KiB global)
WARN [AntiEntropyStage:1] 2026-10-21T10:17:00,298 NoSpamLogger.java:107 -
/R2:7000->/C:7000-LARGE_MESSAGES-[no-channel] overloaded; dropping 8.315MiB
message (queue: 12.507MiB local, 8.507MiB endpoint, 8.507MiB global)
{noformat}
The connections came back about 3s later. With retries disabled, the drop is
never retried and the coordinator is never told; this matches CASSANDRA-20560,
which was closed as expected without retries. The
{{Connection.LargeMessageDroppedTasksDueToOverload}} metrics confirm 2+1 drops
at exactly that time.
* 13:16:48: AutoRepair's 3h timeout fires; the cancel is a no-op, and the
retry immediately collides:
{noformat}
WARN [AutoRepair-Repair-incremental:1] 2026-10-21T13:16:48,372
AutoRepair.java:370 - Repair failed for range
-2779633433731331207--2763034864604948441 for cell tables [...] with
cancellationStatus: false retrying after 30 seconds...
ERROR [AntiEntropyStage:1] 2026-10-21T13:17:59,740 AutoRepair.java:581 - Repair
failure for repair INCREMENTAL: [...] Repair command #89 failed with error
Incremental repair session cf929050-cd51-11f1-b81f-4153cf3313e6 has failed
{noformat}
* For the next 7 days, every incremental auto-repair touching those ranges
failed with:
{noformat}
WARN [Repair-46a1cca0-...:2] 2026-10-28T08:55:08,385
PendingAntiCompaction.java:210 - Prepare phase for incremental repair session
46a1cca0-d2ad-11f1-b81f-4153cf3313e6 has failed because it encountered
intersecting sstables belonging to another incremental repair session
(8681eb40-cd38-11f1-b81f-4153cf3313e6). ...
{noformat}
** {{repair_admin list}} on all 9 participants showed {{8681eb40...}} as
{{{}REPAIRING{}}}, with "last activity" always below one hour; the status-check
keep-alive kept it alive.
** {{Table.BytesPendingRepair}} on the coordinator stayed at about 364 MB the
whole time.
** A second session in the same cluster
({{{}349f6f90-cd39-11f1-9a8e-0962703fd491{}}}, a different coordinator, a
dropped merkle tree in the same minute) had been hung in the same way.
h3. Steps to reproduce
The attached in-JVM dtest {{AutoRepairSessionTimeoutTest}} reproduces this
deterministically in about 2 minutes:
# 3 nodes, RF=3, one table with 1000 rows, flushed.
# Incremental auto-repair runs on node1 only, with {{{}repair_session_timeout:
60s{}}}, {{repair_max_retries: 1}} and {{{}repair_retry_backoff: 1s{}}}. These
two are set in the incremental override, because the global values are ignored
on 5.0.9 (CASSANDRA-21748).
# The LocalSessions timers are shortened (status check 5s, auto-fail 15s,
cleanup 5s) to show that they don't recover the session either.
# A message filter drops exactly one {{VALIDATION_RSP}} (node2 -> node1) for
the test table. {{repair.retries}} is left at its default (disabled).
{code:java}
git checkout cassandra-5.0.9
git apply AutoRepairSessionTimeoutTest-repro.patch
export JAVA_HOME=<JDK 17>
ant jar
ant test-jvm-dtest-some
-Dtest.name=org.apache.cassandra.distributed.test.repair.AutoRepairSessionTimeoutTest
{code}
Observed on 5.0.9 (all intermediate checks pass; the final check, which asserts
the expected behaviour, fails):
{noformat}
[P1] T0: dropped VALIDATION_RSP 2->1 for incremental repair session
d76f47ca-d2f9-11f1-89c1-17ed1be92f21
[P2] session d76f47ca-... still REPAIRING on all nodes at T0+35s (> 2x
the 15s LocalSessions auto-fail timeout)
[P3] node1: WARN ... AutoRepair.java:370 - Repair failed for range
-9223372036854775808--3074457345618258603 for distributed_test_keyspace tables
[tbl] with cancellationStatus: false retrying after 1 seconds...
[P4] WARN ... PendingAntiCompaction.java:210 - Prepare phase for incremental
repair session fbcc35b0-... has failed because it encountered intersecting
sstables belonging to another incremental repair session
(d76f47ca-d2f9-11f1-89c1-17ed1be92f21) ...
[P5] node1: ERROR ... AutoRepair.java:394 - Repair failed for range ... after 2
retries, total assignments: 2,processed assignments: 1, cancellationStatus:
false
[FINAL] node1 ActiveRepairService sessionCount=1
junit.framework.AssertionFailedError: AutoRepair gave up after
repair_session_timeout, but incremental repair session
d76f47ca-d2f9-11f1-89c1-17ed1be92f21 is still REPAIRING on node1; its
pendingRepair sstables block every later incremental repair of these ranges
expected:<[FAILED]> but was:<[REPAIRING]>
{noformat}
Notes on the test:
* It uses the in-JVM message path, without the NETWORK/GOSSIP features.
* The matcher finds the table by scanning the raw message bytes for the
RepairJobDesc keyspace/table strings, because a {{ValidationResponse}} can't be
deserialized in the test classloader.
* It has only been run on 5.0.9 so far.
h3. Expected behaviour
When AutoRepair gives up on a repair after {{{}repair_session_timeout{}}}, that
repair should actually be aborted:
* the IR session is failed on all participants ({{{}FailSession{}}}), so its
{{pendingRepair}} sstables are released back to unrepaired;
* the coordinator's {{{}RepairSession{}}}/{{{}ParentRepairSession{}}} are
removed;
* the retry starts from a clean state.
h3. Suggested fix
# On timeout, abort the specific repair instead of calling
{{{}f.cancel(true){}}}. Use the parent session id
({{{}RepairCoordinator.state.id{}}}) to:
** force-shutdown that parent's \{{RepairSession}}s, so the coordinator chain
completes exceptionally;
** fail the IR {{{}CoordinatorSession{}}}, which sends {{FailSession}} to the
participants;
** remove the {{{}ParentRepairSession{}}}.
This may need a small {{{}RepairCoordinator{}}}/{{{}ActiveRepairService{}}} API
for aborting a single repair, like {{terminateSessions()}} but scoped to one
repair.
# Log the real outcome of the abort; the current {{cancellationStatus}} is
always {{false}} and misleading.
# Optionally: don't start a retry for the same ranges while the previous IR
session is still non-terminal.
# Optionally: when incremental auto-repair is enabled while {{repair.retries}}
(merkle_tree_response) is disabled, warn at startup or document it prominently.
A single dropped merkle tree otherwise hangs the repair indefinitely
(CASSANDRA-20560).
was:
h3. Summary
When an auto-repair task exceeds {{{}repair_session_timeout{}}}, AutoRepair
"cancels" it with {{Future.cancel(true)}} and retries the same ranges. That
cancel is a no-op: {{RepairCoordinator.run()}} only wires an asynchronous
future chain and returns right away, so the executor future is already done and
{{cancel()}} returns {{{}false{}}}. The repair keeps running, or keeps hanging,
in the background:
* the incremental repair (IR) session stays {{REPAIRING}} on every
participant, and its sstables stay {{{}pendingRepair{}}};
* every retry, and every later incremental repair of the same ranges, fails in
the prepare phase with {{{}...encountered intersecting sstables belonging to
another incremental repair session (<old id>){}}};
* the {{RepairSession}} stays registered on the coordinator.
Nothing ever cleans this up:
* LocalSessions never auto-fails the session, because the participants keep
answering each other's status requests with {{{}REPAIRING{}}}, and since
CASSANDRA-20312 each such answer refreshes {{{}lastUpdate{}}};
* the coordinator has no timeout of its own on merkle-tree responses.
In our production cluster the session stayed hung for 7 days, until an operator
ran {{{}nodetool repair_admin cancel{}}}. Incremental repair of the affected
ranges was blocked for that whole time.
h3. Root cause
[AutoRepair.java (5.0.9), lines
344-398|https://github.com/apache/cassandra/blob/cassandra-5.0.9/src/java/org/apache/cassandra/repair/autorepair/AutoRepair.java#L344-L398]:
{code:java}
f = repairRunnableExecutors.get(repairType).submit(task); // task =
RepairCoordinator
listener.await(config.getRepairSessionTimeout(repairType)); // times out ->
success = false
...
boolean cancellationStatus = f.cancel(true); // always false:
run() already returned
// sleep repair_retry_backoff, then submit a NEW RepairCoordinator for the same
ranges
{code}
[RepairCoordinator.run()/runMayThrow()|https://github.com/apache/cassandra/blob/cassandra-5.0.9/src/java/org/apache/cassandra/repair/RepairCoordinator.java#L273-L309]
starts prepare -> repair -> finalize as a future chain and returns. Nothing in
AutoRepair ever fails the parent repair, its {{{}RepairSession}}s, or the IR
{{CoordinatorSession{}}}. As a result, no {{FailSession}} is sent and the
participants keep the sstables in {{{}pendingRepair{}}}.
The same logic is present on the current {{{}cassandra-5.0{}}},
{{cassandra-6.0}} and {{trunk}} branches. Trunk only adds an {{isShutDown}}
branch, which uses the same {{{}f.cancel(true){}}}.
h3. How the session got stuck in production (trigger; not the bug itself)
5.0.9, three DCs, incremental auto-repair, {{repair.retries}} not configured
(default: disabled).
* 10:16:48 UTC: the coordinator (node C) starts IR session
{{8681eb40-cd38-11f1-b81f-4153cf3313e6}} with 9 participants. Prepare succeeds,
and all 54 VALIDATION_REQs (6 tables x 9 replicas) are acknowledged.
* 10:17:00: the LARGE_MESSAGES connections from two remote-DC replicas to node
C fail to connect ({{{}connection timed out after 2000 ms{}}}). While a
connection is failing to connect, {{OutboundConnection}} rejects any message
that doesn't fit the per-connection send queue, so three 8 MiB merkle trees are
dropped:
{noformat}
WARN [AntiEntropyStage:1] 2026-10-21T10:17:00,569 NoSpamLogger.java:107 -
/R1:7000->/C:7000-LARGE_MESSAGES-[no-channel] overloaded; dropping 8.019MiB
message (queue: 4.488MiB local, 500.008KiB endpoint, 500.008KiB global)
WARN [AntiEntropyStage:1] 2026-10-21T10:17:00,298 NoSpamLogger.java:107 -
/R2:7000->/C:7000-LARGE_MESSAGES-[no-channel] overloaded; dropping 8.315MiB
message (queue: 12.507MiB local, 8.507MiB endpoint, 8.507MiB global)
{noformat}
The connections came back about 3s later. With retries disabled, the drop is
never retried and the coordinator is never told; this matches CASSANDRA-20560,
which was closed as expected without retries. The
{{Connection.LargeMessageDroppedTasksDueToOverload}} metrics confirm 2+1 drops
at exactly that time.
* 13:16:48: AutoRepair's 3h timeout fires; the cancel is a no-op, and the
retry immediately collides:
{noformat}
WARN [AutoRepair-Repair-incremental:1] 2026-10-21T13:16:48,372
AutoRepair.java:370 - Repair failed for range
-2779633433731331207--2763034864604948441 for cell tables [...] with
cancellationStatus: false retrying after 30 seconds...
ERROR [AntiEntropyStage:1] 2026-10-21T13:17:59,740 AutoRepair.java:581 - Repair
failure for repair INCREMENTAL: [...] Repair command #89 failed with error
Incremental repair session cf929050-cd51-11f1-b81f-4153cf3313e6 has failed
{noformat}
* For the next 7 days, every incremental auto-repair touching those ranges
failed with:
{noformat}
WARN [Repair-46a1cca0-...:2] 2026-10-28T08:55:08,385
PendingAntiCompaction.java:210 - Prepare phase for incremental repair session
46a1cca0-d2ad-11f1-b81f-4153cf3313e6 has failed because it encountered
intersecting sstables belonging to another incremental repair session
(8681eb40-cd38-11f1-b81f-4153cf3313e6). ...
{noformat}
** {{repair_admin list}} on all 9 participants showed {{8681eb40...}} as
{{{}REPAIRING{}}}, with "last activity" always below one hour; the status-check
keep-alive kept it alive.
** {{Table.BytesPendingRepair}} on the coordinator stayed at about 364 MB the
whole time.
** A second session in the same cluster
({{{}349f6f90-cd39-11f1-9a8e-0962703fd491{}}}, a different coordinator, a
dropped merkle tree in the same minute) had been hung in the same way.
h3. Steps to reproduce
The attached in-JVM dtest {{AutoRepairSessionTimeoutTest}} reproduces this
deterministically in about 2 minutes:
# 3 nodes, RF=3, one table with 1000 rows, flushed.
# Incremental auto-repair runs on node1 only, with {{{}repair_session_timeout:
60s{}}}, {{repair_max_retries: 1}} and {{{}repair_retry_backoff: 1s{}}}. These
two are set in the incremental override, because the global values are ignored
on 5.0.9 (CASSANDRA-21748).
# The LocalSessions timers are shortened (status check 5s, auto-fail 15s,
cleanup 5s) to show that they don't recover the session either.
# A message filter drops exactly one {{VALIDATION_RSP}} (node2 -> node1) for
the test table. {{repair.retries}} is left at its default (disabled).
{code:java}
git checkout cassandra-5.0.9
git apply AutoRepairSessionTimeoutTest-repro.patch
export JAVA_HOME=<JDK 17>
ant jar
ant test-jvm-dtest-some
-Dtest.name=org.apache.cassandra.distributed.test.repair.AutoRepairSessionTimeoutTest
{code}
Observed on 5.0.9 (all intermediate checks pass; the final check, which asserts
the expected behaviour, fails):
{noformat}
[P1] T0: dropped VALIDATION_RSP 2->1 for incremental repair session
d76f47ca-d2f9-11f1-89c1-17ed1be92f21
[P2] session d76f47ca-... still REPAIRING on all nodes at T0+35s (> 2x
the 15s LocalSessions auto-fail timeout)
[P3] node1: WARN ... AutoRepair.java:370 - Repair failed for range
-9223372036854775808--3074457345618258603 for distributed_test_keyspace tables
[tbl] with cancellationStatus: false retrying after 1 seconds...
[P4] WARN ... PendingAntiCompaction.java:210 - Prepare phase for incremental
repair session fbcc35b0-... has failed because it encountered intersecting
sstables belonging to another incremental repair session
(d76f47ca-d2f9-11f1-89c1-17ed1be92f21) ...
[P5] node1: ERROR ... AutoRepair.java:394 - Repair failed for range ... after 2
retries, total assignments: 2,processed assignments: 1, cancellationStatus:
false
[FINAL] node1 ActiveRepairService sessionCount=1
junit.framework.AssertionFailedError: AutoRepair gave up after
repair_session_timeout, but incremental repair session
d76f47ca-d2f9-11f1-89c1-17ed1be92f21 is still REPAIRING on node1; its
pendingRepair sstables block every later incremental repair of these ranges
expected:<[FAILED]> but was:<[REPAIRING]>
{noformat}
Notes on the test:
* It uses the in-JVM message path, without the NETWORK/GOSSIP features.
* The matcher finds the table by scanning the raw message bytes for the
RepairJobDesc keyspace/table strings, because a {{ValidationResponse}} can't be
deserialized in the test classloader.
* It has only been run on 5.0.9 so far.
h3. Expected behaviour
When AutoRepair gives up on a repair after {{{}repair_session_timeout{}}}, that
repair should actually be aborted:
* the IR session is failed on all participants ({{{}FailSession{}}}), so its
{{pendingRepair}} sstables are released back to unrepaired;
* the coordinator's {{{}RepairSession{}}}/{{{}ParentRepairSession{}}} are
removed;
* the retry starts from a clean state.
h3. Suggested fix
# On timeout, abort the specific repair instead of calling
{{{}f.cancel(true){}}}. Use the parent session id
({{{}RepairCoordinator.state.id{}}}) to:
** force-shutdown that parent's \{{RepairSession}}s, so the coordinator chain
completes exceptionally;
** fail the IR {{{}CoordinatorSession{}}}, which sends {{FailSession}} to the
participants;
** remove the {{{}ParentRepairSession{}}}.
This may need a small {{{}RepairCoordinator{}}}/{{{}ActiveRepairService{}}} API
for aborting a single repair, like {{terminateSessions()}} but scoped to one
repair.
# Log the real outcome of the abort; the current {{cancellationStatus}} is
always {{false}} and misleading.
# Optionally: don't start a retry for the same ranges while the previous IR
session is still non-terminal.
# Optionally: when incremental auto-repair is enabled while {{repair.retries}}
(merkle_tree_response) is disabled, warn at startup or document it prominently.
A single dropped merkle tree otherwise hangs the repair indefinitely
(CASSANDRA-20560).
> AutoRepair repair_session_timeout does not abort the timed-out repair,
> leaking a hung incremental repair session that blocks all later incremental
> repairs of its ranges
> ------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: CASSANDRA-21819
> URL: https://issues.apache.org/jira/browse/CASSANDRA-21819
> Project: Apache Cassandra
> Issue Type: Bug
> Components: Consistency/Repair
> Reporter: Artur Malchanau
> Priority: Normal
> Attachments: AutoRepairSessionTimeoutTest-repro.patch
>
>
> h3. Summary
> When an auto-repair task exceeds {{repair_session_timeout{}}}, AutoRepair
> "cancels" it with {{Future.cancel(true)}} and retries the same ranges.
> That cancel is a no-op: {{RepairCoordinator.run()}} only wires an
> asynchronous future chain and returns right away, so the executor future is
> already done and {{cancel()}} returns {{false}}.
> The repair keeps running, or keeps hanging, in the background:
> * the incremental repair (IR) session stays {{REPAIRING}} on every
> participant, and its sstables stay {{pendingRepair}};
> * every retry, and every later incremental repair of the same ranges, fails
> in the prepare phase with {{...encountered intersecting sstables belonging to
> another incremental repair session (<old id>)}};
> * the {{RepairSession}} stays registered on the coordinator.
> Nothing ever cleans this up:
> * LocalSessions never auto-fails the session, because the participants keep
> answering each other's status requests with {{REPAIRING}}, and since
> CASSANDRA-20312 each such answer refreshes {{lastUpdate}};
> * the coordinator has no timeout of its own on merkle-tree responses.
> In our production cluster the session stayed hung for 7 days, until an
> operator ran {{nodetool repair_admin cancel}}. Incremental repair of the
> affected ranges was blocked for that whole time.
> h3. Root cause
> [AutoRepair.java (5.0.9), lines
> 344-398|https://github.com/apache/cassandra/blob/cassandra-5.0.9/src/java/org/apache/cassandra/repair/autorepair/AutoRepair.java#L344-L398]:
> {code:java}
> f = repairRunnableExecutors.get(repairType).submit(task); // task =
> RepairCoordinator
> listener.await(config.getRepairSessionTimeout(repairType)); // times out ->
> success = false
> ...
> boolean cancellationStatus = f.cancel(true); // always false:
> run() already returned
> // sleep repair_retry_backoff, then submit a NEW RepairCoordinator for the
> same ranges
> {code}
> [RepairCoordinator.run()/runMayThrow()|https://github.com/apache/cassandra/blob/cassandra-5.0.9/src/java/org/apache/cassandra/repair/RepairCoordinator.java#L273-L309]
> starts prepare -> repair -> finalize as a future chain and returns. Nothing
> in AutoRepair ever fails the parent repair, its {{{}RepairSession}}s, or the
> IR {{CoordinatorSession{}}}. As a result, no {{FailSession}} is sent and the
> participants keep the sstables in {{{}pendingRepair{}}}.
> The same logic is present on the current {{cassandra-5.0}}, {{cassandra-6.0}}
> and {{trunk}} branches.
> Trunk only adds an {{isShutDown}} branch, which uses the same
> {{f.cancel(true)}}.
> h3. How the session got stuck in production (trigger; not the bug itself)
> 5.0.9, three DCs, incremental auto-repair, {{repair.retries}} not configured
> (default: disabled).
> * 10:16:48 UTC: the coordinator (node C) starts IR session
> {{8681eb40-cd38-11f1-b81f-4153cf3313e6}} with 9 participants. Prepare
> succeeds, and all 54 VALIDATION_REQs (6 tables x 9 replicas) are acknowledged.
> * 10:17:00: the LARGE_MESSAGES connections from two remote-DC replicas to
> node C fail to connect ({{connection timed out after 2000 ms}}). While a
> connection is failing to connect, {{OutboundConnection}} rejects any message
> that doesn't fit the per-connection send queue, so three 8 MiB merkle trees
> are dropped:
> {noformat}
> WARN [AntiEntropyStage:1] 2026-10-21T10:17:00,569 NoSpamLogger.java:107 -
> /R1:7000->/C:7000-LARGE_MESSAGES-[no-channel] overloaded; dropping 8.019MiB
> message (queue: 4.488MiB local, 500.008KiB endpoint, 500.008KiB global)
> WARN [AntiEntropyStage:1] 2026-10-21T10:17:00,298 NoSpamLogger.java:107 -
> /R2:7000->/C:7000-LARGE_MESSAGES-[no-channel] overloaded; dropping 8.315MiB
> message (queue: 12.507MiB local, 8.507MiB endpoint, 8.507MiB global)
> {noformat}
> The connections came back about 3s later. With retries disabled, the drop is
> never retried and the coordinator is never told; this matches
> CASSANDRA-20560, which was closed as expected without retries. The
> {{Connection.LargeMessageDroppedTasksDueToOverload}} metrics confirm 2+1
> drops at exactly that time.
> * 13:16:48: AutoRepair's 3h timeout fires; the cancel is a no-op, and the
> retry immediately collides:
> {noformat}
> WARN [AutoRepair-Repair-incremental:1] 2026-10-21T13:16:48,372
> AutoRepair.java:370 - Repair failed for range
> -2779633433731331207--2763034864604948441 for cell tables [...] with
> cancellationStatus: false retrying after 30 seconds...
> ERROR [AntiEntropyStage:1] 2026-10-21T13:17:59,740 AutoRepair.java:581 -
> Repair failure for repair INCREMENTAL: [...] Repair command #89 failed with
> error Incremental repair session cf929050-cd51-11f1-b81f-4153cf3313e6 has
> failed
> {noformat}
> * For the next 7 days, every incremental auto-repair touching those ranges
> failed with:
> {noformat}
> WARN [Repair-46a1cca0-...:2] 2026-10-28T08:55:08,385
> PendingAntiCompaction.java:210 - Prepare phase for incremental repair session
> 46a1cca0-d2ad-11f1-b81f-4153cf3313e6 has failed because it encountered
> intersecting sstables belonging to another incremental repair session
> (8681eb40-cd38-11f1-b81f-4153cf3313e6). ...
> {noformat}
> ** {{repair_admin list}} on all 9 participants showed {{8681eb40...}} as
> {{{}REPAIRING{}}}, with "last activity" always below one hour; the
> status-check keep-alive kept it alive.
> ** {{Table.BytesPendingRepair}} on the coordinator stayed at about 364 MB
> the whole time.
> ** A second session in the same cluster
> ({{{}349f6f90-cd39-11f1-9a8e-0962703fd491{}}}, a different coordinator, a
> dropped merkle tree in the same minute) had been hung in the same way.
> h3. Steps to reproduce
> The attached in-JVM dtest {{AutoRepairSessionTimeoutTest}} reproduces this
> deterministically in about 2 minutes:
> # 3 nodes, RF=3, one table with 1000 rows, flushed.
> # Incremental auto-repair runs on node1 only, with
> {{{}repair_session_timeout: 60s{}}}, {{repair_max_retries: 1}} and
> {{{}repair_retry_backoff: 1s{}}}. These two are set in the incremental
> override, because the global values are ignored on 5.0.9 (CASSANDRA-21748).
> # The LocalSessions timers are shortened (status check 5s, auto-fail 15s,
> cleanup 5s) to show that they don't recover the session either.
> # A message filter drops exactly one {{VALIDATION_RSP}} (node2 -> node1) for
> the test table. {{repair.retries}} is left at its default (disabled).
> {code:java}
> git checkout cassandra-5.0.9
> git apply AutoRepairSessionTimeoutTest-repro.patch
> export JAVA_HOME=<JDK 17>
> ant jar
> ant test-jvm-dtest-some
> -Dtest.name=org.apache.cassandra.distributed.test.repair.AutoRepairSessionTimeoutTest
> {code}
> Observed on 5.0.9 (all intermediate checks pass; the final check, which
> asserts the expected behaviour, fails):
> {noformat}
> [P1] T0: dropped VALIDATION_RSP 2->1 for incremental repair session
> d76f47ca-d2f9-11f1-89c1-17ed1be92f21
> [P2] session d76f47ca-... still REPAIRING on all nodes at T0+35s (> 2x
> the 15s LocalSessions auto-fail timeout)
> [P3] node1: WARN ... AutoRepair.java:370 - Repair failed for range
> -9223372036854775808--3074457345618258603 for distributed_test_keyspace
> tables [tbl] with cancellationStatus: false retrying after 1 seconds...
> [P4] WARN ... PendingAntiCompaction.java:210 - Prepare phase for incremental
> repair session fbcc35b0-... has failed because it encountered intersecting
> sstables belonging to another incremental repair session
> (d76f47ca-d2f9-11f1-89c1-17ed1be92f21) ...
> [P5] node1: ERROR ... AutoRepair.java:394 - Repair failed for range ... after
> 2 retries, total assignments: 2,processed assignments: 1, cancellationStatus:
> false
> [FINAL] node1 ActiveRepairService sessionCount=1
> junit.framework.AssertionFailedError: AutoRepair gave up after
> repair_session_timeout, but incremental repair session
> d76f47ca-d2f9-11f1-89c1-17ed1be92f21 is still REPAIRING on node1; its
> pendingRepair sstables block every later incremental repair of these ranges
> expected:<[FAILED]> but was:<[REPAIRING]>
> {noformat}
> Notes on the test:
> * It uses the in-JVM message path, without the NETWORK/GOSSIP features.
> * The matcher finds the table by scanning the raw message bytes for the
> RepairJobDesc keyspace/table strings, because a {{ValidationResponse}} can't
> be deserialized in the test classloader.
> * It has only been run on 5.0.9 so far.
> h3. Expected behaviour
> When AutoRepair gives up on a repair after {{{}repair_session_timeout{}}},
> that repair should actually be aborted:
> * the IR session is failed on all participants ({{{}FailSession{}}}), so its
> {{pendingRepair}} sstables are released back to unrepaired;
> * the coordinator's {{{}RepairSession{}}}/{{{}ParentRepairSession{}}} are
> removed;
> * the retry starts from a clean state.
> h3. Suggested fix
> # On timeout, abort the specific repair instead of calling
> {{{}f.cancel(true){}}}. Use the parent session id
> ({{{}RepairCoordinator.state.id{}}}) to:
> ** force-shutdown that parent's \{{RepairSession}}s, so the coordinator
> chain completes exceptionally;
> ** fail the IR {{{}CoordinatorSession{}}}, which sends {{FailSession}} to
> the participants;
> ** remove the {{{}ParentRepairSession{}}}.
> This may need a small {{{}RepairCoordinator{}}}/{{{}ActiveRepairService{}}}
> API for aborting a single repair, like {{terminateSessions()}} but scoped to
> one repair.
> # Log the real outcome of the abort; the current {{cancellationStatus}} is
> always {{false}} and misleading.
> # Optionally: don't start a retry for the same ranges while the previous IR
> session is still non-terminal.
> # Optionally: when incremental auto-repair is enabled while
> {{repair.retries}} (merkle_tree_response) is disabled, warn at startup or
> document it prominently. A single dropped merkle tree otherwise hangs the
> repair indefinitely (CASSANDRA-20560).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
