[
https://issues.apache.org/jira/browse/CAMEL-26569?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on CAMEL-26569 started by shashank.
----------------------------------------
> camel-core - parallel Split/Multicast with shareUnitOfWork: on completion
> handover fails with ConcurrentModificationException
> -----------------------------------------------------------------------------------------------------------------------------
>
> Key: CAMEL-26569
> URL: https://issues.apache.org/jira/browse/CAMEL-26569
> Project: Camel
> Issue Type: Bug
> Components: camel-core
> Reporter: shashank
> Assignee: shashank
> Priority: Major
>
> Found with a SPIN model of {{DefaultUnitOfWork}} shared by the parts of a
> parallel Split (two parts handing over, one part adding an on completion,
> then the parent's {{done()}}), then reproduced with a deterministic test on
> the real classes.
> Since Camel 4.14 the parts of a Split, Multicast or Recipient List with
> {{shareUnitOfWork}} use the parent's unit of work
> ({{MulticastProcessor.prepareSharedUnitOfWork}}, see the 4.14 upgrade guide).
> With {{parallelProcessing}} they use it at the same time.
> {{DefaultUnitOfWork.addSynchronization}} / {{removeSynchronization}} take the
> unit of work lock, but {{handoverSynchronization}} iterates the list and
> removes from it with an iterator, without the lock. A part hands over the on
> completions whenever it sends to an endpoint that routes a copy on its own,
> for example a {{seda}} endpoint with an {{InOnly}} exchange
> ({{SedaProducer.addToQueue}} ->
> {{ExchangeHelper.createCorrelatedCopy(exchange, true, true)}} ->
> {{handoverCompletions}}). {{Resequencer}} and {{StreamResequencer}} hand over
> the same way.
> So when one part hands over while another part adds an on completion, or
> hands over too:
> * the part fails with a {{ConcurrentModificationException}} from
> {{it.remove()}}; with {{shareUnitOfWork}} that fails the whole split, and the
> copy was never queued;
> * an on completion is handed over to two copies and runs twice (for example
> the consumer's commit or file move), or, when the part that took it fails, it
> is never run.
> {code:java}
> from("file:inbox") // or any consumer that adds an on completion
> .split(body().tokenize("\n")).parallelProcessing().shareUnitOfWork()
> .to("seda:lines")
> .end();
> {code}
> {noformat}
> java.util.ConcurrentModificationException
> at
> java.base/java.util.ArrayList$Itr.checkForComodification(ArrayList.java)
> at java.base/java.util.ArrayList$Itr.remove(ArrayList.java)
> at
> org.apache.camel.impl.engine.DefaultUnitOfWork.handoverSynchronization(DefaultUnitOfWork.java:258)
> at
> org.apache.camel.support.AbstractExchange.handoverCompletions(AbstractExchange.java:863)
> at
> org.apache.camel.support.ExchangeHelper.createCorrelatedCopy(ExchangeHelper.java:320)
> at
> org.apache.camel.component.seda.SedaProducer.prepareCopy(SedaProducer.java:209)
> {noformat}
> SPIN trace (short): parts 1 and 2 both create an iterator over
> [consumerSync]; part 3 adds partSync (under the lock, which the iterators
> ignore); part 2 hands consumerSync to its copy and removes it; part 1 gets a
> CME, its copy is dropped and the split fails. In another trace part 1 had
> already taken consumerSync into its copy, so it is never run. The fixed model
> (copy under the lock, then take each synchronization out under the lock
> before handing it over) holds, with a witness that the good path is reachable.
> On main (1b12eea, 2026-11-02) a deterministic test fails: part A is held
> inside the handover (in {{allowHandover()}} of the on completion) while part
> B adds an on completion, and two parts are held there at the same time; both
> splits fail with the CME, while the same route without {{parallelProcessing}}
> passes. Without any hook, 2000 splits of 4 parts each sending to {{seda}}
> (counted after a 60 second wait) failed 4 times with the CME and ran the
> consumer on completion twice once.
> Related: CAMEL-22259 (the shared unit of work, 4.14), CAMEL-22533 and
> CAMEL-23681 (PR #23777) (a race with the same setup in the route stack of the
> shared unit of work, now under the same lock), CAMEL-16566 (an earlier
> single-threaded CME in {{handoverSynchronization}}). Affects 4.14.0 and
> later; fix for 4.24.0 and 4.18.6 (the same code is on 4.18.x).
> *Proposal*
> * {{handoverSynchronization}}: take a copy of the list under the lock, decide
> on the copy ({{allowHandover}}, filter), and take each synchronization out of
> the list under the lock before handing it over, skipping one another thread
> has already taken, so it is handed over once. The target is not called while
> holding the lock (no lock order between two units of work).
> * {{done()}}, {{beforeRoute()}}, {{afterRoute()}}: run the synchronizations
> from a copy taken under the lock (they copy the list without it, so a
> concurrent add or handover can leave a {{null}} in the copy). No lock when
> there are no synchronizations (hot path, CAMEL-25132).
> * For a single thread the order and the result stay the same; an on
> completion added during the handover is no longer visited (that was the CME
> of CAMEL-16566).
> * Not part of this: after a parallel split times out, a late part can still
> hand over or add an on completion while {{done()}} runs (it can then run
> twice, or not at all).
> * Not part of this (could be a separate JIRA): with {{shareUnitOfWork}}, a
> part that sends to {{seda}} hands over all the on completions of the shared
> unit of work (including the consumer's commit or file move) to its copy, so
> they run when that copy is done, not when the split is done (also without
> parallel processing).
> _Filed with Claude Code on behalf of allthingssecurity._
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
