[
https://issues.apache.org/jira/browse/CAMEL-22364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Claus Ibsen updated CAMEL-22364:
--------------------------------
Summary: camel-jms - Rare ConcurrentModificationException on message header
map when using multiple camel-jms calls after each other in route (was:
ConcurrentModificationException on message header map when using multiple
camel-jms calls after each other in route)
> camel-jms - Rare ConcurrentModificationException on message header map when
> using multiple camel-jms calls after each other in route
> ------------------------------------------------------------------------------------------------------------------------------------
>
> Key: CAMEL-22364
> URL: https://issues.apache.org/jira/browse/CAMEL-22364
> Project: Camel
> Issue Type: Bug
> Components: camel-jms
> Affects Versions: 4.4.5, 4.5.0, 4.6.0, 4.8.8, 4.9.0, 4.10.6, 4.11.0,
> 4.12.0, 4.13.0, 4.14.0
> Reporter: Vladimir Dobos
> Priority: Major
> Fix For: 4.8.9, 4.10.7, 4.14.1, 4.15.0
>
> Attachments: access_history.txt, camel.log
>
>
> When calling multiple camel-jms endpoints in sequence in one route, there is
> a random possibility of ConcurrentModificationException on header map during
> route execution when using more than 1 thread to run the route (see
> stacktrace on line 12 in attached log file camel.log) for details.
> I wrote simple program to simulate the issue
> (https://github.com/vdobos-tr/TestJmsConcurrentModification), program
> contains both client (RageCallMQ.java using jmh, run by jmh.sh) and backend
> (CamelMain, executed usually from my IDE) part, client uses JMH to tax the
> camel-jms endpoints and cause the exception. We use IBM MQ as a message
> broker (added example compose in directory mq-compose, requires path
> modification to run).
> Probability and time until the exception happens depends heavily on hardware
> and general setup of the host machine. Our customers can easily replicate the
> issue in their test environments running 20 consumers and simple route
> (from(jms, in only).process().to(jms, inout).process(...).to(jms, out only))
> whereas we had to run 50-75 threads and up to 6 orchestrations to get one
> exception within 30 minutes of execution (from(jms, in
> only).process().[to(jms, inout).process(...)<repeat 6 times>].to(jms, out
> only)). As a race condition, unfortunatelly, the timing when the exception
> occuring is very random, sometimes it can occur within first minute,
> sometimes it can take 30+ minutes :(.
> Exception occurs regardless if CachingConnectionFactory is used and regardles
> of asyncConsumer setting. ConcurrentModificationException is caused by
> parallel acces of two threads to headers map on message (I used custom
> HeadersMapFactory and custom map implementation with some Proxy classes to
> trace the issue, see package cz.trask.bi.mq.camel.debugmap in application)
> I was able to trace the issue to change in camel 4.4.0
> (https://issues.apache.org/jira/browse/CAMEL-20338, see file
> access_history.txt, lines 78 to 200 and line 202 created by custom Headers
> Map that shows thread acces causing the issue).
> As far as I understand the problem, it is posible for doSend(...) on line 260
> in JmsProducer to not finish (maybe OS schduler parking threads ?) until next
> thread processing next part of route received response from backend and is
> already sending next request (or final response) in route (header map
> iteration on line 368 of JmsBinfding, where the stacktrace originates),
> during this iteration original doSend(...) on line on line 260 in
> JmsProducer resumes and hits lines 260-262, added by CAMEL-20338, causing
> ConcurrentModificationException on next loop iteration in JmsBinding, line
> 368.
> I tested the issue on camel 4.4.0, 4.8.3 - the version of apache camel we and
> our customers are currently using, 4.10.6, 4.13.0 and also newly relased
> 4.14.0 and it occurs on all of them.
> Apache camel releases before canel 4.4.0 (4.3.0, 4.2.0, 4.1.0, 4.0.0) do not
> manifest the issue.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
