shashank created CAMEL-25682:
--------------------------------
Summary: camel-resilience4j - Circuit Breaker EIP: a timeout that
fires while the task copies its result into the exchange lets the fallback (or
the timeout exception) and that copy change the exchange at the same time
Key: CAMEL-25682
URL: https://issues.apache.org/jira/browse/CAMEL-25682
Project: Camel
Issue Type: Bug
Components: eip
Reporter: shashank
Assignee: shashank
CAMEL-24134 added a write guard between the task of the circuit breaker (it
runs on the timeout executor and copies its result from the correlated copy
into the exchange) and the fallback that the caller runs after a timeout. The
task claims the guard with {{compareAndSet(false, true)}} and then writes
({{ResilienceProcessor.processTask}}, lines 751-764 at main cd2ce899764e; async
mode {{processTaskAsync}} 812-824). The fallback task only does
{{exchangeWriteGuard.set(true)}} (line 911) and then writes the exchange. So
when the timeout fires after the task has won the guard but before it has
finished its write-back, both write the same exchange at the same time:
* with {{onFallback}}: the task's {{ExchangeHelper.copyResults}} replaces the
body of the fallback by the body of the task, or sets
{{CamelCircuitBreakerResponseFromFallback}} back to {{false}} and
{{CamelCircuitBreakerResponseSuccessfulExecution}} to {{true}} after the
fallback has set them, while the exchange already continues in the route;
* without fallback: {{copyResults}} ends with {{setException(null)}} and clears
the {{TimeoutException}} the caller has just set (a timed-out call continues as
a success);
* the headers and properties maps of the exchange are written by two threads.
The caller does not wait for the task: with a timeout the task runs in
{{CompletableFuture.supplyAsync}} (591) behind
{{TimeLimiter.decorateFutureSupplier}} (599); async mode uses
{{TimeLimiter.decorateCompletionStage}} (672). The window is the time the task
needs for {{copyResults}} and two properties, so it needs a task that ends at
about the moment of the timeout.
h3. Reproduction
* New {{ResilienceTimeoutDuringWriteBackTest}} (4 tests: sync and
{{asynchronous=true}}, with and without {{onFallback}}, timeout 500 ms). The
task's result message holds the first read of its body by
{{ExchangeHelper.copyResults}} (the test checks the stack) until the caller has
returned, or until the caller has not returned 2 seconds after the timeout. On
main all 4 fail: {{expected: <Fallback result> but was: <Task result>}} (sync
and async), and {{expected: <java.util.concurrent.TimeoutException> but was:
<null>}} (sync and async).
* Without the hold: a probe (not part of the change) with 20000 calls from 32
threads, timeout 20 ms and a task of 20 ms +/- 0.3 ms, a 64-thread timeout
executor: on main 26 exchanges ended inconsistent (for example the fallback
body with {{CamelCircuitBreakerResponseFromFallback=false}}); with the fix 0.
One run each.
h3. Proposed fix
The guard becomes a small lock-based guard per call: the task claims it and
writes while holding the lock; the fallback task claims it under the same lock,
so a caller that times out while the task is writing waits until that write is
done (bounded: {{copyResults}} and two properties). It then sets the timeout
result or runs the fallback on the exchange, which then holds the result of the
task (as today when the timeout fires just after the copy). The task writes
only if it claims the exchange before the caller; the caller always writes
after a timeout. In async mode the waiting thread is a thread of the time
limiter's scheduler pool, which already runs the fallback. Sync and async mode.
One {{ReentrantLock}} per call instead of one {{AtomicBoolean}}.
camel-microprofile-fault-tolerance has the same {{set(true)}}
({{FaultToleranceProcessor}} 298) but is not affected: the same test (adapted)
passes on main, because with SmallRye Fault Tolerance 6.11.4 the caller gets
the {{TimeoutException}} only after the task has returned (it waited the whole
hold in the test), so the task and the fallback do not overlap.
camel-resilience4j tests with the fix: 81 run, 0 failures.
Not changed (follow-ups): an exception thrown by {{copyResults}} after the task
claimed the exchange is dropped and the call counts as a success (same on
main); and whether a caller that finds the write-back already claimed should
keep the task's result instead of running the fallback (it would change the
outcome of such calls, so it needs a decision).
How it was found: the backlog of an earlier review round listed the
{{set(true)}}. A small TLA+ model of the guard (task: claim, write body, write
properties; caller: timeout at any time before the task returned, claim,
fallback, continue) violates "the task does not write after the caller took the
exchange over" and "the final exchange tells one story" on main in 18 / 36-49
states, and both hold with the lock (28-31 states). The model is a sketch; the
test and the probe are the evidence. The microprofile part of the model
assumption (the caller continues while the task runs) turned out to be false
for SmallRye, see above.
Affected: main (cd2ce899764e; tested), camel-4.23.x and camel-4.22.x (same
{{set(true)}}; read, not run). camel-4.18.x has no guard at all (before
CAMEL-24134).
Duplicate check (2026-10-15): JIRA text "circuit breaker timeout worker write
back" (CAMEL-24134 only, resolved, the fix this completes),
"exchangeWriteGuard" (CAMEL-25129, CAMEL-24209: other topics); GitHub pull
requests "exchangeWriteGuard", "FaultToleranceProcessor" (#24797 = CAMEL-24134,
merged; nothing open).
_Filed with Claude Code on behalf of allthingssecurity._
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
