[
https://issues.apache.org/jira/browse/CAMEL-23681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18086275#comment-18086275
]
Claus Ibsen commented on CAMEL-23681:
-------------------------------------
Not sure which of these PRs but we have this race condition
{code:java}
[INFO] Running org.apache.camel.issues.SplitParallelSharedUoWIssueTest
Error: Tests run: 9, Failures: 0, Errors: 6, Skipped: 0, Time elapsed: 1.430 s
<<< FAILURE! -- in org.apache.camel.issues.SplitParallelSharedUoWIssueTest
Error:
org.apache.camel.issues.SplitParallelSharedUoWIssueTest.testSplitParallelShareeUoW()[8]
-- Time elapsed: 0.157 s <<< ERROR!
org.apache.camel.CamelExecutionException: Exception occurred during execution
on the exchange: Exchange[5A2D3B4526A7A8B-0000000000000000]
at
org.apache.camel.CamelExecutionException.wrapCamelExecutionException(CamelExecutionException.java:60)
at
org.apache.camel.support.ExchangeHelper.extractResultBody(ExchangeHelper.java:704)
at
org.apache.camel.impl.engine.DefaultProducerTemplate.extractResultBody(DefaultProducerTemplate.java:594)
at
org.apache.camel.impl.engine.DefaultProducerTemplate.extractResultBody(DefaultProducerTemplate.java:590)
at
org.apache.camel.impl.engine.DefaultProducerTemplate.sendBody(DefaultProducerTemplate.java:188)
at
org.apache.camel.impl.engine.DefaultProducerTemplate.sendBody(DefaultProducerTemplate.java:194)
at
org.apache.camel.issues.SplitParallelSharedUoWIssueTest.testSplitParallelShareeUoW(SplitParallelSharedUoWIssueTest.java:33)
Caused by: java.util.NoSuchElementException
at java.base/java.util.ArrayDeque.removeFirst(ArrayDeque.java:361)
at java.base/java.util.ArrayDeque.pop(ArrayDeque.java:592)
at
org.apache.camel.impl.engine.DefaultUnitOfWork.popRoute(DefaultUnitOfWork.java:394)
at
org.apache.camel.impl.engine.CamelInternalProcessor$UnitOfWorkProcessorAdvice.after(CamelInternalProcessor.java:1189)
at
org.apache.camel.impl.engine.CamelInternalProcessor$UnitOfWorkProcessorAdvice.after(CamelInternalProcessor.java:1121)
at
org.apache.camel.impl.engine.AdviceIterator.runAfterTask(AdviceIterator.java:45)
at
org.apache.camel.impl.engine.AdviceIterator.runAfterTasks(AdviceIterator.java:39)
at
org.apache.camel.impl.engine.CamelInternalProcessor$AsyncAfterTask.done(CamelInternalProcessor.java:263)
at org.apache.camel.AsyncCallback.run(AsyncCallback.java:46)
at
org.apache.camel.impl.engine.DefaultReactiveExecutor$Worker.doRun(DefaultReactiveExecutor.java:202)
at
org.apache.camel.impl.engine.DefaultReactiveExecutor$Worker.executeReactiveWork(DefaultReactiveExecutor.java:192)
at
org.apache.camel.impl.engine.DefaultReactiveExecutor$Worker.tryExecuteReactiveWork(DefaultReactiveExecutor.java:169)
at
org.apache.camel.impl.engine.DefaultReactiveExecutor$Worker.schedule(DefaultReactiveExecutor.java:143)
at
org.apache.camel.impl.engine.DefaultReactiveExecutor.scheduleSync(DefaultReactiveExecutor.java:64)
at
org.apache.camel.processor.MulticastProcessor.lambda$schedule$1(MulticastProcessor.java:401)
at
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
at java.base/java.lang.Thread.run(Thread.java:1583)
Error:
org.apache.camel.issues.SplitParallelSharedUoWIssueTest.testSplitParallelShareeUoW()[11]
-- Time elapsed: 0.162 s <<< ERROR!
org.apache.camel.CamelExecutionException: Exception occurred during execution
on the exchange: Exchange[05444D1200E1BC5-0000000000000000]
at
org.apache.camel.CamelExecutionException.wrapCamelExecutionException(CamelExecutionException.java:60)
at
org.apache.camel.support.ExchangeHelper.extractResultBody(ExchangeHelper.java:704)
at
org.apache.camel.impl.engine.DefaultProducerTemplate.extractResultBody(DefaultProducerTemplate.java:594)
at
org.apache.camel.impl.engine.DefaultProducerTemplate.extractResultBody(DefaultProducerTemplate.java:590)
at
org.apache.camel.impl.engine.DefaultProducerTemplate.sendBody(DefaultProducerTemplate.java:188)
at
org.apache.camel.impl.engine.DefaultProducerTemplate.sendBody(DefaultProducerTemplate.java:194)
at
org.apache.camel.issues.SplitParallelSharedUoWIssueTest.testSplitParallelShareeUoW(SplitParallelSharedUoWIssueTest.java:33)
Caused by: java.util.NoSuchElementException
at java.base/java.util.ArrayDeque.removeFirst(ArrayDeque.java:361)
at java.base/java.util.ArrayDeque.pop(ArrayDeque.java:592)
at
org.apache.camel.impl.engine.DefaultUnitOfWork.popRoute(DefaultUnitOfWork.java:394)
at
org.apache.camel.impl.engine.CamelInternalProcessor$UnitOfWorkProcessorAdvice.after(CamelInternalProcessor.java:1189)
at
org.apache.camel.impl.engine.CamelInternalProcessor$UnitOfWorkProcessorAdvice.after(CamelInternalProcessor.java:1121)
at
org.apache.camel.impl.engine.AdviceIterator.runAfterTask(AdviceIterator.java:45)
at
org.apache.camel.impl.engine.AdviceIterator.runAfterTasks(AdviceIterator.java:39)
at
org.apache.camel.impl.engine.CamelInternalProcessor$AsyncAfterTask.done(CamelInternalProcessor.java:263)
at org.apache.camel.AsyncCallback.run(AsyncCallback.java:46)
at
org.apache.camel.impl.engine.DefaultReactiveExecutor$Worker.doRun(DefaultReactiveExecutor.java:202)
at
org.apache.camel.impl.engine.DefaultReactiveExecutor$Worker.executeReactiveWork(DefaultReactiveExecutor.java:192)
at
org.apache.camel.impl.engine.DefaultReactiveExecutor$Worker.tryExecuteReactiveWork(DefaultReactiveExecutor.java:169)
at
org.apache.camel.impl.engine.DefaultReactiveExecutor$Worker.schedule(DefaultReactiveExecutor.java:143)
at
org.apache.camel.impl.engine.DefaultReactiveExecutor.scheduleSync(DefaultReactiveExecutor.java:64)
at
org.apache.camel.processor.MulticastProcessor.lambda$schedule$1(MulticastProcessor.java:401)
at
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
at java.base/java.lang.Thread.run(Thread.java:1583)
Error:
org.apache.camel.issues.SplitParallelSharedUoWIssueTest.testSplitParallelShareeUoW()[19]
-- Time elapsed: 0.156 s <<< ERROR!
org.apache.camel.CamelExecutionException: Exception occurred during execution
on the exchange: Exchange[041598AB69158D1-0000000000000000]
at
org.apache.camel.CamelExecutionException.wrapCamelExecutionException(CamelExecutionException.java:60)
at
org.apache.camel.support.ExchangeHelper.extractResultBody(ExchangeHelper.java:704)
at
org.apache.camel.impl.engine.DefaultProducerTemplate.extractResultBody(DefaultProducerTemplate.java:594)
at
org.apache.camel.impl.engine.DefaultProducerTemplate.extractResultBody(DefaultProducerTemplate.java:590)
at
org.apache.camel.impl.engine.DefaultProducerTemplate.sendBody(DefaultProducerTemplate.java:188)
at
org.apache.camel.impl.engine.DefaultProducerTemplate.sendBody(DefaultProducerTemplate.java:194)
at
org.apache.camel.issues.SplitParallelSharedUoWIssueTest.testSplitParallelShareeUoW(SplitParallelSharedUoWIssueTest.java:33)
Caused by: java.util.NoSuchElementException
at java.base/java.util.ArrayDeque.removeFirst(ArrayDeque.java:361)
at java.base/java.util.ArrayDeque.pop(ArrayDeque.java:592)
at
org.apache.camel.impl.engine.DefaultUnitOfWork.popRoute(DefaultUnitOfWork.java:394)
at
org.apache.camel.impl.engine.CamelInternalProcessor$UnitOfWorkProcessorAdvice.after(CamelInternalProcessor.java:1189)
at
org.apache.camel.impl.engine.CamelInternalProcessor$UnitOfWorkProcessorAdvice.after(CamelInternalProcessor.java:1121)
at
org.apache.camel.impl.engine.AdviceIterator.runAfterTask(AdviceIterator.java:45)
at
org.apache.camel.impl.engine.AdviceIterator.runAfterTasks(AdviceIterator.java:39)
at
org.apache.camel.impl.engine.CamelInternalProcessor$AsyncAfterTask.done(CamelInternalProcessor.java:263)
at org.apache.camel.AsyncCallback.run(AsyncCallback.java:46)
at
org.apache.camel.impl.engine.DefaultReactiveExecutor$Worker.doRun(DefaultReactiveExecutor.java:202)
at
org.apache.camel.impl.engine.DefaultReactiveExecutor$Worker.executeReactiveWork(DefaultReactiveExecutor.java:192)
at
org.apache.camel.impl.engine.DefaultReactiveExecutor$Worker.tryExecuteReactiveWork(DefaultReactiveExecutor.java:169)
at
org.apache.camel.impl.engine.DefaultReactiveExecutor$Worker.schedule(DefaultReactiveExecutor.java:143)
at
org.apache.camel.impl.engine.DefaultReactiveExecutor.scheduleSync(DefaultReactiveExecutor.java:64)
at
org.apache.camel.processor.MulticastProcessor.lambda$schedule$1(MulticastProcessor.java:401)
at
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
at java.base/java.lang.Thread.run(Thread.java:1583) {code}
> Optimize Exchange memory pressure: copy-on-write headers and lazy
> initialization
> --------------------------------------------------------------------------------
>
> Key: CAMEL-23681
> URL: https://issues.apache.org/jira/browse/CAMEL-23681
> Project: Camel
> Issue Type: Improvement
> Components: camel-core
> Reporter: Guillaume Nodet
> Assignee: Guillaume Nodet
> Priority: Major
> Fix For: 4.21.0
>
>
> h2. Problem
> During Exchange routing, message copies (multicast, splitter, recipient-list,
> pipeline processors) allocate new header maps via
> {{ConcurrentHashMap.putAll()}} on every copy. For high-fanout EIPs this
> creates significant memory pressure and GC load.
> h2. Analysis
> The Exchange lifecycle was analyzed for memory allocation patterns:
> *Header Map Copying:*
> - {{MessageSupport.copyFromWithNewBody()}} copies all headers on every
> message copy
> - Multicast with N branches creates N full header map copies
> - Splitter with 1000 items creates ~1000 header map copies
> - Most copies only read headers — they never modify them
> *Lazy Headers:*
> - {{DefaultMessage.getHeader()}} and related read methods force-create the
> headers map even when headers have never been set
> - Many pipeline processors only interact with the message body, not headers
> - Each unnecessary map creation wastes 200-400 bytes
> *Pooled Exchange Properties:*
> - {{DefaultPooledExchange.done()}} discards and recreates the properties map
> on every reuse
> - For typical small properties maps, clearing and reusing is cheaper than
> reallocating
> h2. Solution
> *1. Copy-on-Write Headers (Highest Impact)*
> Message copies now share the headers map reference until one side modifies
> it. A {{CopyOnWriteHeadersMap}} wrapper defers cloning until the first
> mutation, with lazy iterators for {{keySet()}} and {{entrySet()}} that only
> trigger COW on {{Iterator.remove()}}.
> *2. Lazy Headers Initialization*
> {{getHeader()}} and related read methods return null without allocation when
> headers have never been set.
> *3. Properties Map Reuse in Pooled Exchanges*
> {{DefaultPooledExchange.done()}} reuses cleared properties maps instead of
> discarding them (threshold: 50 entries).
> h2. Expected Impact
> - Significant reduction in memory allocation for high-fanout EIPs (Multicast,
> Splitter, RecipientList)
> - Zero behavioral change for existing routes
> - Backward compatible: no public API changes
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
