[
https://issues.apache.org/jira/browse/CAMEL-24401?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Omar Atie reassigned CAMEL-24401:
---------------------------------
Assignee: Omar Atie
> Camel-jms InOut pattern on to(...) with temporary queues can enter infinitely
> looping error state
> -------------------------------------------------------------------------------------------------
>
> Key: CAMEL-24401
> URL: https://issues.apache.org/jira/browse/CAMEL-24401
> Project: Camel
> Issue Type: Bug
> Components: camel-jms
> Affects Versions: 4.22.0
> Environment: OpenShift (customer production environemnt, on
> enterprise license)
> Local OpenSuse Linux distribution for reproduciton + Docker
> JDK 21 and JDK 25
> Reporter: Vladimir Dobos
> Assignee: Omar Atie
> Priority: Major
>
> Today I bring you another "juicy" camel-jms threading issue :) (much juicier
> than [last year one|https://issues.apache.org/jira/browse/CAMEL-22364])
> Companion git repo:
> [https://github.com/vdobos-trask/Reproducer_Camel_Jms_Failed_ReplyTo_Destination_2026]
>
> ([https://github.com/vdobos-trask/Reproducer_Camel_Jms_Failed_ReplyTo_Destination_2026.git])
> Overview:
> - This issue describes two use-cases where thread of camel-jms InOut to(...)
> endpoint gets stuck +infinitely+ looping on "Failed to resolve replyTo
> destination on the exchange" error for each processed request, when using
> temporary reply queue (see Exception.txt in linked git repository), unable to
> +ever+ recover (which means the route is theoretically reported as live,
> although it does not work)
> - Issue was reported on IBM MQ, however it should be a general problem for
> any MQ broker
> - This ticket reports 2 issues dubbed scenario A and scenario B, both are
> succession of (different) steps, that will result in the same outcome inside
> of camel-jms
> - The scenario A depicts (most probable) cause of real situation that
> happened on our customers OpenShift running many Camel Main Java route DSL
> (no Spring Boot) based microservices, scenario B was found accidentaly during
> the analysis and search of A, after upgrading to camel 4.21.0/4.22.0
> - Our customer mentioned this issue occuring several times during last
> year+, but they only observed it happening very occasionaly (once every 2-3
> months) on less critical services, only in production environment, never on
> dev or test
> - After this issue occurred on critical services in production with
> side-effect of necessiating regulatory reporting, we were called to fully
> diagnose the issue
> Natural occurrence background (for last occurrance, we don't know about
> previous ones):
> - OpenShift cluster
> - Issue happened on one OpenShift worker after destabilization of storage
> caused by live patching
> - More than one service on same worker were affected, however not all of
> them. With no found common characteristics besides camel-jms usage (affectees
> and survivors were pointed to both same and different MQ managers randomly
> between them)
> - The messaging broker is IBM MQ
> - Routes in affected microservices are called by http using rest(...) and
> call mix of camel-jms and http/rest backends
> - The camel version is 4.8.9 (with planned update to 4.22.0 this fall),
> again to reiterate - java code camel routes, Camel Main without Spring Boot
> - The logging level in production is set to ERROR, there were unfortunately
> no abnormal "leading cause" log lines before the critical "Failed to resolve
> replyTo destination on the exchange" exception, we saw only end state in the
> logs, nothing that lead to it was logged. This is why the scenario A is
> depicted as "most probable" cause, as i was unable to find different scenario
> on camel-jms 4.8.9 that can cause the issue
> - There was no reproduction of this issue outside of our customers
> production environment before I started the analysis
> AI Usage
> - Most of exact diagnosis and reproduction was performed with heavy AI
> assistance
> - AI used was mostly Claude Fable, Calude Opus 5 with small parts being done
> by GPT 5.6 Sol
> - Claude Fable was used mostly due to frequent hitting of content-filters
> for cybersecurity on 5.6 Sol, Opus 5 was used when we hit content filtering
> on Fable (tweaking of networking and CPU resources and specific cli commands
> required for reproduction unfortunately cause quite frequent false-positive
> flagging, especialy on GPT 5.6 Sol)
> - We use Github Copilot as AI harness (enterprise ghe domain as we need EU
> data residency)
> - Analysis was heavily human-guided as AI had propensity to dig itself into
> very deep hole it could not escape from ;)
> - This text is 100% human created ;)
> Diagnosis steps:
> - After manually reviewing the code, my lead theory was some sort of out-of
> order execution with thread parking, similar in spirit to issue from last
> year (happened for the same customer - their environments are clearly
> -cursed- blessed), but clearly much more complex in setup and causality chain
> - The analysis impled that replyTo got permanently stuck on null which on
> normal inspection of recovery steps on any error should not happen (it did
> unfortunately)
> - Reproduction was performed on my personal laptop running Linux OS, there
> is a possibility that different CPU/OS will cause different outcomes on
> reproducer
> - Focus of the reproduction was reproduction using unmodified
> camel/camel-jms to prove the issue can happen in the wild and is not only
> theoretical
> - AI confirmed potential presence of candidate causality chain that can lead
> to described stuck state and created a reproducer
> - Reproducer required changes to CPU settings for containers, thus necessity
> of root privileges. I used rootful docker with temporary facl on docker
> socket to run the reproducer (I usually use rootless podman)
> - After succesfull reproduction on camel 4.8.9, the reproducer was updated
> to 4.21.0 and subsequently to locally built and regularly updated snapshot of
> master from camel git repo for 4.22.0 (this was in weeks pre-release)
> - The reproduction of issue on 4.22.0 snapshot hit much faster than 4.8.9,
> however after querying AI to confirm the reproduced issue really matches
> theoretized scenario A, AI identified scenario B which is caused by Spring
> 7.0.x adding verification step to recovery inside of spring-jms, which causes
> setting of varaiable in camel-jms prematurely while MQ is still unstable
> - After isolating Scenario B I refocused AI to focus on reproduction of
> Scenario A and was succesfull with reproduction on camel 4.22.0 snapshot.
> Scenario A is complicated thread-parking induced issue. According to AI, it
> is much more likely to happen on old 4.8.9 than new 4.22.0, however it is
> still possible (as is seen in succesfull reproducer), thus Murphys laws would
> make it happen within 12 hours of deployed upgrade ;)
> - I instructed AI to double-confirm the reproduction of both scenarios by
> creating probes in source code using java agent and new Java 25 classfile
> API. it chose injection of custom JFR events and verified the scenarios
> - Both issues were also successfully reproduced on camel 4.22.0 final release
> - Due to complexity, I don't include exact description of issues here in
> ticket but created companion git repository with documentation and reproducer
> *Artifacts (AI generated, I assume availability of AI assistance, per AI
> activity on camel JIRA/Github):*
> - Public GitHub repository link:
> [https://github.com/vdobos-trask/Reproducer_Camel_Jms_Failed_ReplyTo_Destination_2026]
>
> ([https://github.com/vdobos-trask/Reproducer_Camel_Jms_Failed_ReplyTo_Destination_2026.git])
> - Reproducer: folder ~/Reproducer (see ~/Reproducer/README.md), REQUIRES
> rootfull containers, expects docker with rw facl set on docker socket
> - Human-friendly interactive overview of both scenarios (html, created also
> for me so i can better understand the scenarios): primary scenario-based,
> based on theoretical best-case scenario -
> "/explanation_docs/interactive/scenario/index.html". For complex scenario A
> there is also {~}"{~}/explanation_docs/interactive/debugger/index.html",
> based on real reproducer log.
> - AI (and human ;)) readable markdown resources:
> "~/explanation_docs/markdown/*.md"
> Postscript:
> - Athough I mention version 4.8.9, that is only for full context as
> originator of the issue as it is long out of support. I reported the issue
> for 4.22.0 only, as that is the planned version to use forward for our
> customer and we are interested in. Older versions of camel between 4.8.9 and
> 4.22.0 most probably also suffer from at least scenario A, and if using
> Spring 7.0.X (i think its 7.0.4 and higher) they also suffer from scenario B,
> however I didnt have enough time to test historical versions.
> - I instructed AI to theoretize similar scenarios on camel-sjms. It
> successfully proposed several scenarios of unknown viability, even claimed
> some are worse than ones in camel-jms. AI readable markdown can be found in
> git in file ""~/explanation_docs/markdown/Unproven_camel_sjms.md". As this
> was outside of my assignment, i unfortunately didn't have time to explore it
> further, however I noticed you employed AI to do checks and fix several
> issues for 4.22.0 so I included it so you can explore this.
> - When trying to reproduce scenario A on camel 4.22.0, I instructed AI to
> perform minimal fix of scenario B as it shadowed A in most use-cases, I was
> warned that naive fix for B might introduce scenario C that will cause the
> same issue in different steps (scenario is briefly described in resources)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
