shashank created CAMEL-25005:
--------------------------------
Summary: In-memory Saga: a step that joins while the saga is being
completed, compensated or timed out is reported as joined, but its
compensation/completion never runs
Key: CAMEL-25005
URL: https://issues.apache.org/jira/browse/CAMEL-25005
Project: Camel
Issue Type: Bug
Components: camel-core
Reporter: shashank
{{InMemorySagaCoordinator.beginStep}} (core/camel-support,
{{InMemorySagaCoordinator.java}}) is check-then-act without any lock:
# {{:80-85}} reads {{currentStatus}} and continues if it is {{RUNNING}},
# {{:87-104}} evaluates the step's saga options (user expressions),
# {{:105}} adds the step to {{enlistments}} and returns a completed future, so
the step body runs.
{{complete()}}, {{compensate()}} and the timeout task switch the status with
{{compareAndSet(RUNNING, ...)}} ({{:109}}, {{:122}}, {{:147}}) and
{{doFinalize}} then takes a snapshot of the steps with
{{reversed(enlistments)}} ({{:196}}). Nothing orders the snapshot after the
enlistment of a step that already passed the status check. A step that enlists
after the snapshot:
* gets a successful {{beginStep}}, so its action runs (for example, the payment
is taken),
* is never passed to its compensation (or completion) endpoint,
* and the saga ends as COMPENSATED (or COMPLETED) and is removed from the
service.
The window is the whole option evaluation plus the status read. The timeout
task makes it reachable with only synchronous routes: the saga times out while
a joining step ({{MANDATORY}}/{{REQUIRED}}/{{SUPPORTS}}) is between the check
and the add.
*Reproduction* (real classes):
{code:java}
from("direct:owner").saga().timeout(300,
MILLISECONDS).compensation("direct:compOwner")
.to("direct:payment");
from("direct:payment").saga().propagation(SagaPropagation.MANDATORY)
.option("orderId", slowOption) // evaluated inside the window
.compensation("direct:compPayment")
.process(e -> paymentTaken++);
{code}
{{slowOption}} waits until {{direct:compOwner}} has run (it forces the
interleaving; any slow option expression or a GC pause does the same). Output:
{noformat}
[late] owner exchange failed=true exception=Cannot complete: status is
COMPENSATED
[late] payment step action executed=1 (its beginStep returned success)
[late] owner compensation calls=1 payment compensation calls=0
{noformat}
Without any forcing ({{stress}}: timeout 1 ms, owner work 0.7-1.3 ms, plain
{{simple}} option), 1 of 5000 sagas ended compensated with the payment step
executed and never compensated.
The same race exists with {{complete()}}/{{compensate()}} from AUTO completion
when a participant joins from another thread.
TLA+: {{ExactlyOnceAtEnd}} ("every step whose beginStep succeeded is finalized
exactly once") is violated by {{sync_timeout_mandatory}} (PCheck(p1) ->
TimerFire -> FzSnap -> FzRun -> FzFinal -> PEnlist(p1)) and
{{async_mandatory}}. {{NeverBoth}}, {{AtMostOnce}} and {{KindMatches}} hold.
*Proposed fix:* make the status check and the enlistment atomic with respect to
the transition to a final state. For example, a lock in the coordinator:
* {{beginStep}}: evaluate the options first (outside the lock), then under the
lock check {{RUNNING}} and add the enlistment (and register the timeout);
* {{complete()}}, {{compensate()}}, the timeout task: under the same lock do
the CAS and take the {{enlistments}} snapshot that {{doFinalize}} uses (pass it
in instead of reading the list later).
A step that loses the race then fails with {{IllegalStateException("Cannot
begin: status is ...")}} as it does today when it arrives a moment later. The
model with this change ({{fix_*}} configs) satisfies all invariants and
{{Termination}}.
_Filed with Claude Code on behalf of allthingssecurity._
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
