shashank created CAMEL-26569:
--------------------------------
Summary: camel-core - parallel Split/Multicast with
shareUnitOfWork: on completion handover fails with
ConcurrentModificationException
Key: CAMEL-26569
URL: https://issues.apache.org/jira/browse/CAMEL-26569
Project: Camel
Issue Type: Bug
Components: camel-core
Reporter: shashank
Assignee: shashank
Found with a SPIN model of {{DefaultUnitOfWork}} shared by the parts of a
parallel Split (two parts handing over, one part adding an on completion, then
the parent's {{done()}}), then reproduced with a deterministic test on the real
classes.
Since Camel 4.14 the parts of a Split, Multicast or Recipient List with
{{shareUnitOfWork}} use the parent's unit of work
({{MulticastProcessor.prepareSharedUnitOfWork}}, see the 4.14 upgrade guide).
With {{parallelProcessing}} they use it at the same time.
{{DefaultUnitOfWork.addSynchronization}} / {{removeSynchronization}} take the
unit of work lock, but {{handoverSynchronization}} iterates the list and
removes from it with an iterator, without the lock. A part hands over the on
completions whenever it sends to an endpoint that routes a copy on its own, for
example a {{seda}} endpoint with an {{InOnly}} exchange
({{SedaProducer.addToQueue}} -> {{ExchangeHelper.createCorrelatedCopy(exchange,
true, true)}} -> {{handoverCompletions}}). {{Resequencer}} and
{{StreamResequencer}} hand over the same way.
So when one part hands over while another part adds an on completion, or hands
over too:
* the part fails with a {{ConcurrentModificationException}} from
{{it.remove()}}; with {{shareUnitOfWork}} that fails the whole split, and the
copy was never queued;
* an on completion is handed over to two copies and runs twice (for example the
consumer's commit or file move), or, when the part that took it fails, it is
never run.
{code:java}
from("file:inbox") // or any consumer that adds an on completion
.split(body().tokenize("\n")).parallelProcessing().shareUnitOfWork()
.to("seda:lines")
.end();
{code}
{noformat}
java.util.ConcurrentModificationException
at java.base/java.util.ArrayList$Itr.checkForComodification(ArrayList.java)
at java.base/java.util.ArrayList$Itr.remove(ArrayList.java)
at
org.apache.camel.impl.engine.DefaultUnitOfWork.handoverSynchronization(DefaultUnitOfWork.java:258)
at
org.apache.camel.support.AbstractExchange.handoverCompletions(AbstractExchange.java:863)
at
org.apache.camel.support.ExchangeHelper.createCorrelatedCopy(ExchangeHelper.java:320)
at
org.apache.camel.component.seda.SedaProducer.prepareCopy(SedaProducer.java:209)
{noformat}
SPIN trace (short): parts 1 and 2 both create an iterator over [consumerSync];
part 3 adds partSync (under the lock, which the iterators ignore); part 2 hands
consumerSync to its copy and removes it; part 1 gets a CME, its copy is dropped
and the split fails. In another trace part 1 had already taken consumerSync
into its copy, so it is never run. The fixed model (copy under the lock, then
take each synchronization out under the lock before handing it over) holds,
with a witness that the good path is reachable.
On main (1b12eea, 2026-11-02) a deterministic test fails: part A is held inside
the handover (in {{allowHandover()}} of the on completion) while part B adds an
on completion, and two parts are held there at the same time; both splits fail
with the CME, while the same route without {{parallelProcessing}} passes.
Without any hook, 2000 splits of 4 parts each sending to {{seda}} (counted
after a 60 second wait) failed 4 times with the CME and ran the consumer on
completion twice once.
Related: CAMEL-22259 (the shared unit of work, 4.14), CAMEL-22533 and
CAMEL-23681 (PR #23777) (a race with the same setup in the route stack of the
shared unit of work, now under the same lock), CAMEL-16566 (an earlier
single-threaded CME in {{handoverSynchronization}}). Affects 4.14.0 and later;
fix for 4.24.0 and 4.18.6 (the same code is on 4.18.x).
*Proposal*
* {{handoverSynchronization}}: take a copy of the list under the lock, decide
on the copy ({{allowHandover}}, filter), and take each synchronization out of
the list under the lock before handing it over, skipping one another thread has
already taken, so it is handed over once. The target is not called while
holding the lock (no lock order between two units of work).
* {{done()}}, {{beforeRoute()}}, {{afterRoute()}}: run the synchronizations
from a copy taken under the lock (they copy the list without it, so a
concurrent add or handover can leave a {{null}} in the copy). No lock when
there are no synchronizations (hot path, CAMEL-25132).
* For a single thread the order and the result stay the same; an on completion
added during the handover is no longer visited (that was the CME of
CAMEL-16566).
* Not part of this: after a parallel split times out, a late part can still
hand over or add an on completion while {{done()}} runs (it can then run twice,
or not at all).
* Not part of this (could be a separate JIRA): with {{shareUnitOfWork}}, a part
that sends to {{seda}} hands over all the on completions of the shared unit of
work (including the consumer's commit or file move) to its copy, so they run
when that copy is done, not when the split is done (also without parallel
processing).
_Filed with Claude Code on behalf of allthingssecurity._
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
