[
https://issues.apache.org/jira/browse/CAMEL-24401?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vladimir Dobos updated CAMEL-24401:
-----------------------------------
Description:
Today I bring you another "juicy" camel-jms threading issue :) (much juicier
than last year one)
Overview:
- This issue describes two use-cases where thread of camel-jms InOut to(...)
endpoint gets stuck +infinitely+ looping on "Failed to resolve replyTo
destination on the exchange" error for each processed request, when using
temporary reply queue (see Exception.txt in linked git repository), unable to
+ever+ recover (which means the route is theoretically reported as live,
although it does not work)
- The issue implies that replyTo got stuck on null permanently somehow, which
on first glance should never happen :/
- Issue was reported on IBM MQ, however it should be a general problem for any
MQ broker
- This ticket reports 2 issues dubbed scenario A and scenario B, both are
succession of (different) steps, that will result in the same outcome inside of
camel-jms
- The scenario A depicts (most probable) cause of real situation that happened
on our customers OpenShift running many Camel Main Java route DSL (no Spring
Boot) based microservices, scenario B was found accidentaly during the analysis
and search of A, after upgrading to camel 4.21.0/4.22.0
- Our customer mentioned this issue occuring several times during last year+,
but they only observed it happening very occasionaly (once every 2-3 months) on
less critical services, only in production environment, never on dev or test
- After this issue occurred on critical services in production with
side-effect of necessiating regulatory reporting, we were called to fully
diagnose the issue
Natural occurrence background (for last occurrance, we don't know about
previous ones):
- OpenShift cluster
- Issue happened on one OpenShift worker after destabilization of storage
caused by live patching
- More than one service on same worker were affected, however not all of them.
With no found common characteristics besides camel-jms usage (affectees and
survivors were pointed to both same and different MQ managers randomly between
them)
- The messaging broker is IBM MQ
- Rotes in affected microservices are called by http using rest(...) and call
mix of camel-jms and http/rest backends
- The camel version is 4.8.9 (with planned update to 4.22.0 this fall), again
to reiterate - java code camel routes, Camel Main without Spring Boot
- The logging level in production is set to ERROR, there were unfortunately no
abnormal "leading cause" log lines before the critical "Failed to resolve
replyTo destination on the exchange" exception, we saw only end state in the
logs, nothing that lead to it was logged. This is why the scenario A is
depicted as "most probable" cause, as i was unable to find different scenario
on camel-jms 4.8.9 that can cause the issue
- There was no reproduction of this issue outside of our customers production
environment before I started the analysis
AI Usage
- Most of exact diagnosis and reproduction was performed with heavy AI
assistance
- AI used was mostly Claude Fable, Calude Opus 5 with small parts being done
by GPT 5.6 Sol
- Claude Fable was used mostly due to frequent hitting of content-filters for
cybersecurity on 5.6 Sol, Opus 5 was used when we hit content filtering on
Fable (tweaking of networking and CPU resources and specific cli commands
required for reproduction unfortunately cause quite frequent false-positive
flagging, especialy on GPT 5.6 Sol)
- We use Github Copilot as AI harness (enterprise ghe domain as we need EU
data residency)
- Analysis was heavily human-guided as AI had propensity to dig itself into
very deep hole it could not escape from ;)
- This text is 100% human created ;)
Diagnosis steps:
- After manually reviewing the code, my lead theory was some sort of out-of
order execution with thread parking, similar in spirit to issue from last year
last year (happened for the same customer - their environments are clearly
-cursed- blessed), but clearly much more complex in setup and causality chain
- Reproduction was performed on my personal laptop running Linux OS, there is
a possibility that different CPU/OS will cause different outcomes on reproducer
- Focus of the reproduction was reproduction using unmodified camel/camel-jms
to prove the issue can happen in the wild and is not only theoretical
- AI confirmed potential presence of candidate causality chain that can lead
to described stuck state and created a reproducer
- Reproducer required changes to CPU settings for containers, thus necessity
of root privileges. I used rootful docker with temporary facl on docker socket
to run the reproducer (I usually use rootless podman)
- After succesfull reproduction on camel 4.8.9, the reproducer was updated to
4.21.0 and subsequently to locally built and regularly updated snapshot of
master from camel git repo for 4.22.0 (this was in weeks pre-release)
- The reproduction of issue on 4.22.0 snapshot hit much faster than 4.8.9,
however after querying AI to confirm the reproduced issue really matches
theoretized scenario A, AI identified scenario B which is caused by Spring
7.0.x adding verification step to recovery inside of spring-jms, which causes
setting of varaiable in camel-jms prematurely while MQ is still unstable
- After isolating Scenario B I refocused AI to focus on reproduction of
Scenario A and was succesfull with reproduction camel 4.22.0 snapshot. Scenario
A is complicated thread-parking induced issue. According to AI, it is much more
likely to happen on old 4.8.9 than new 4.22.0, however it is still possible (as
is seen in succesfull reproducer), thus Murphys laws would make it happen
within 12 hours of deployed upgrade ;)
- I instructed AI to double-confirm the reproduction of both scenarios by
creating probes in source code using java agent and new Java 25 classfile API.
it chose injection of custom JFR events and verified the scenarios
- Both issues were also successfully reproduced on camel 4.22.0 final release
Artifacts (AI generated, I assume availability of AI assistance, per AI
activity on camel JIRA/Github):
- Public GitHub repository link:
[https://github.com/vdobos-trask/Reproducer_Camel_Jms_Failed_ReplyTo_Destination_2026]
([https://github.com/vdobos-trask/Reproducer_Camel_Jms_Failed_ReplyTo_Destination_2026.git])
- Reproducer: folder ~/Reproducer (see ~/Reproducer/README.md), REQUIRES
rootfull containers, expects docker with rw facl set on docker socket
- Human-friendly interactive overview of both scenarios (html, created also
for me so i can better understand the scenarios): primary scenario-based, based
on theoretical best-case scenario -
"{~}/explanation_docs/interactive/scenario/index.html", for complex scenario A
there is also "{~}/explanation_docs/interactive/debugger/index.html", based on
real reproducer log.
- AI (and human ;)) readable markdown resources:
"~/explanation_docs/markdown/*.md"
Postscript:
- I instructed AI to theoretize similar scenarios on camel-sjms. It
successfully proposed several scenarios of unknown viability, even claimed some
are worse than ones in camel-jms. AI readable markdown can be found in git in
file ""~/explanation_docs/markdown/Unproven_camel_sjms.md". As this was outside
of my assignment, i unfortunately didn't have time to explore it further,
however I noticed you employed AI to do checks and fix several issues for
4.22.0 so I included it so you can explore this.
- When trying to reproduce scenario A on camel 4.22.0, I instructed AI to
perform minimal fix of scenario B as it shadowed A in most use-cases, I was
warned that naive fix for B might introduce scenario C that will cause the same
issue in different steps (scenario is briefly described in resources)
was:
Today I bring you another "juicy" camel-jms threading issue :) (much juicier
than [last year one|https://issues.apache.org/jira/browse/CAMEL-22364])
Overview:
- This issue describes two use-cases where thread of camel-jms InOut to(...)
endpoint gets stuck +infinitely+ looping on "Failed to resolve replyTo
destination on the exchange" error for each processed request, when using
temporary reply queue (see Exception.txt in linked git repository), unable to
+ever+ recover (which means the route is theoretically reported as live,
although it does not work)
- The issue implies that replyTo got stuck on null permanently somehow, which
on first glance should never happen :/
- Issue was reported on IBM MQ, however it should be a general problem for any
MQ broker
- This ticket reports 2 issues dubbed scenario A and scenario B, both are
succession of (different) steps, that will result in the same outcome inside of
camel-jms
- The scenario A depicts (most probable) cause of real situation that happened
on our customers OpenShift running many Camel Main Java route DSL (no Spring
Boot) based microservices, scenario B was found accidentaly during the analysis
and search of A, after upgrading to camel 4.21.0/4.22.0
- Our customer mentioned this issue occuring several times during last year+,
but they only observed it happening very occasionaly (once every 2-3 months) on
less critical services, only in production environment, never on dev or test
- After this issue occurred on critical services in production with side-effect
of necessiating regulatory reporting, we were called to fully diagnose the issue
Natural occurrence background (for last occurrance, we dont know about previous
ones):
- OpenShift cluster
- Issue happened on one OpenShift worker after destabilization of storage
caused by live patching
- More than one service on same worker were affected, however not all of them.
With no found common characteristics besides camel-jms usage (affectees and
survivors were pointed to both same and different MQ managers randomly between
them)
- The messaging broker is IBM MQ
- Rotes in affected microservices are called by http using rest(...) and call
mix of camel-jms and http/rest backends
- The camel version is 4.8.9 (with planned update to 4.22.0 this fall), again
to reiterate - java code camel routes, Camel Main without Spring Boot
- The logging level in production is set to ERROR, there were unfortunately no
abnormal "leading cause" log lines before the critical "Failed to resolve
replyTo destination on the exchange" exception, we saw only end state in the
logs, nothing that lead to it was logged. This is why the scenario A is
depicted as "most probable" cause, as i was unable to find different scenario
on camel-jms 4.8.9 that can cause the issue
- There was no reproduction of this issue outside of our customers production
environment before I started the analysis
AI Usage
- Most of exact diagnosis and reproduction was performed with heavy AI
assistance
- AI used was mostly Claude Fable, Calude Opus 5 with small parts being done by
GPT 5.6 Sol
- Claude Fable was used mostly due to frequent hitting of content-filters for
cybersecurity on 5.6 Sol, Opus 5 was used when we hit content filtering on
Fable (tweaking of networking and CPU resources and specific cli commands
required for reproduction unfortunately cause quite frequent false-positive
flagging, especialy on GPT 5.6 Sol)
- We use Github Copilot as AI harness (enterprise ghe domain as we need EU data
residency)
- Analysis was heavily human-guided as AI had propensity to dig itself into
very deep hole it could not escape from ;)
- This text is 100% human created ;)
Diagnosis steps:
- After manually reviewing the code, my lead theory was some sort of out-of
order execution with thread parking, similar in spirit to issue from last year
[last year|https://issues.apache.org/jira/browse/CAMEL-22364] (happened for the
same customer - their environments are clearly -cursed- blessed), but clearly
much more complex in setup and causality chain
- Reproduction was performed on my personal laptop running Linux OS, there is a
possibility that different CPU/OS will cause different outcomes on reproducer
- Focus of the reproduction was reproduction using unmodified camel/camel-jms
to prove the issue can happen in the wild and is not only theoretical
- AI confirmed potential presence of candidate causality chain that can lead to
described stuck state and created a reproducer
- Reproducer required changes to CPU settings for containers, thus necessity of
root privileges. I used rootful docker with temporary facl on docker socket to
run the reproducer (I usually use rootless podman)
- After succesfull reproduction on camel 4.8.9, the reproducer was updated to
4.21.0 and subsequently to locally built and regularly updated snapshot of
master from camel git repo for 4.22.0 (this was in weeks pre-release)
- The reproduction of issue on 4.22.0 snapshot hit much faster than 4.8.9,
however after querying AI to confirm the reproduced issue really matches
theoretized scenario A, AI identified scenario B which is caused by Spring
7.0.x adding verification step to recovery inside of spring-jms, which causes
setting of varaiable in camel-jms prematurely while MQ is still unstable
- After isolating Scenario B I refocused AI to focus on reproduction of
Scenario A and was succesfull with reproduction camel 4.22.0 snapshot. Scenario
A is complicated thread-parking induced issue. According to AI, it is much more
likely to happen on old 4.8.9 than new 4.22.0, however it is still possible (as
is seen in succesfull reproducer), thus Murphys laws would make it happen
within 12 hours of deployed upgrade ;)
- I instructed AI to double-confirm the reproduction of both scenarios by
creating probes in source code using java agent and new Java 25 classfile API.
it chose injection of custom JFR events and verified the scenarios
- Both issues were also successfully reproduced on camel 4.22.0 final release
Artifacts (AI generated, I assume availability of AI assistance, per AI
activity on camel JIRA/Github):
- Public GitHub repository link:
https://github.com/vdobos-trask/Reproducer_Camel_Jms_Failed_ReplyTo_Destination_2026
(https://github.com/vdobos-trask/Reproducer_Camel_Jms_Failed_ReplyTo_Destination_2026.git)
- Reproducer: folder ~/Reproducer (see ~/Reproducer/README.md), REQUIRES
rootfull containers, expects docker with rw facl set on docker socket
- Human-friendly interactive overview of both scenarios (html, created also for
me so i can better understand the scenarios): primary scenario-based, based on
theoretical best-case scenario -
"~/explanation_docs/interactive/scenario/index.html", for complex scenario A
there is also "~/explanation_docs/interactive/debugger/index.html", based on
real reproducer log.
- AI (and human ;)) readable markdown resources:
"~/explanation_docs/markdown/*.md"
Postscript:
- I instructed AI to theoretize similar scenarios on camel-sjms. It
successfully proposed several scenarios of unknown viability, even claimed some
are worse than ones in camel-jms. AI readable markdown can be found in git in
file ""~/explanation_docs/markdown/Unproven_camel_sjms.md". As this was outside
of my assignment, i unfortunately didn't have time to explore it further,
however I noticed you employed AI to do checks and fix several issues for
4.22.0 so I included it so you can explore this.
- When trying to reproduce scenario A on camel 4.22.0, I instructed AI to
perform minimal fix of scenario B as it shadowed A in most use-cases, I was
warned that naive fix for B might introduce scenario C that will cause the same
issue in different steps (scenario is briefly described in resources)
> Camel-jms InOut pattern on to(...) with temporary queues can enter infinitely
> looping error state
> -------------------------------------------------------------------------------------------------
>
> Key: CAMEL-24401
> URL: https://issues.apache.org/jira/browse/CAMEL-24401
> Project: Camel
> Issue Type: Bug
> Components: camel-jms
> Affects Versions: 4.22.0
> Environment: OpenShift (customer production environemnt, on
> enterprise license)
> Local OpenSuse Linux distribution for reproduciton + Docker
> JDK 21 and JDK 25
> Reporter: Vladimir Dobos
> Priority: Major
>
> Today I bring you another "juicy" camel-jms threading issue :) (much juicier
> than last year one)
> Overview:
> - This issue describes two use-cases where thread of camel-jms InOut to(...)
> endpoint gets stuck +infinitely+ looping on "Failed to resolve replyTo
> destination on the exchange" error for each processed request, when using
> temporary reply queue (see Exception.txt in linked git repository), unable to
> +ever+ recover (which means the route is theoretically reported as live,
> although it does not work)
> - The issue implies that replyTo got stuck on null permanently somehow,
> which on first glance should never happen :/
> - Issue was reported on IBM MQ, however it should be a general problem for
> any MQ broker
> - This ticket reports 2 issues dubbed scenario A and scenario B, both are
> succession of (different) steps, that will result in the same outcome inside
> of camel-jms
> - The scenario A depicts (most probable) cause of real situation that
> happened on our customers OpenShift running many Camel Main Java route DSL
> (no Spring Boot) based microservices, scenario B was found accidentaly during
> the analysis and search of A, after upgrading to camel 4.21.0/4.22.0
> - Our customer mentioned this issue occuring several times during last
> year+, but they only observed it happening very occasionaly (once every 2-3
> months) on less critical services, only in production environment, never on
> dev or test
> - After this issue occurred on critical services in production with
> side-effect of necessiating regulatory reporting, we were called to fully
> diagnose the issue
> Natural occurrence background (for last occurrance, we don't know about
> previous ones):
> - OpenShift cluster
> - Issue happened on one OpenShift worker after destabilization of storage
> caused by live patching
> - More than one service on same worker were affected, however not all of
> them. With no found common characteristics besides camel-jms usage (affectees
> and survivors were pointed to both same and different MQ managers randomly
> between them)
> - The messaging broker is IBM MQ
> - Rotes in affected microservices are called by http using rest(...) and
> call mix of camel-jms and http/rest backends
> - The camel version is 4.8.9 (with planned update to 4.22.0 this fall),
> again to reiterate - java code camel routes, Camel Main without Spring Boot
> - The logging level in production is set to ERROR, there were unfortunately
> no abnormal "leading cause" log lines before the critical "Failed to resolve
> replyTo destination on the exchange" exception, we saw only end state in the
> logs, nothing that lead to it was logged. This is why the scenario A is
> depicted as "most probable" cause, as i was unable to find different scenario
> on camel-jms 4.8.9 that can cause the issue
> - There was no reproduction of this issue outside of our customers
> production environment before I started the analysis
> AI Usage
> - Most of exact diagnosis and reproduction was performed with heavy AI
> assistance
> - AI used was mostly Claude Fable, Calude Opus 5 with small parts being done
> by GPT 5.6 Sol
> - Claude Fable was used mostly due to frequent hitting of content-filters
> for cybersecurity on 5.6 Sol, Opus 5 was used when we hit content filtering
> on Fable (tweaking of networking and CPU resources and specific cli commands
> required for reproduction unfortunately cause quite frequent false-positive
> flagging, especialy on GPT 5.6 Sol)
> - We use Github Copilot as AI harness (enterprise ghe domain as we need EU
> data residency)
> - Analysis was heavily human-guided as AI had propensity to dig itself into
> very deep hole it could not escape from ;)
> - This text is 100% human created ;)
> Diagnosis steps:
> - After manually reviewing the code, my lead theory was some sort of out-of
> order execution with thread parking, similar in spirit to issue from last
> year last year (happened for the same customer - their environments are
> clearly -cursed- blessed), but clearly much more complex in setup and
> causality chain
> - Reproduction was performed on my personal laptop running Linux OS, there
> is a possibility that different CPU/OS will cause different outcomes on
> reproducer
> - Focus of the reproduction was reproduction using unmodified
> camel/camel-jms to prove the issue can happen in the wild and is not only
> theoretical
> - AI confirmed potential presence of candidate causality chain that can lead
> to described stuck state and created a reproducer
> - Reproducer required changes to CPU settings for containers, thus necessity
> of root privileges. I used rootful docker with temporary facl on docker
> socket to run the reproducer (I usually use rootless podman)
> - After succesfull reproduction on camel 4.8.9, the reproducer was updated
> to 4.21.0 and subsequently to locally built and regularly updated snapshot of
> master from camel git repo for 4.22.0 (this was in weeks pre-release)
> - The reproduction of issue on 4.22.0 snapshot hit much faster than 4.8.9,
> however after querying AI to confirm the reproduced issue really matches
> theoretized scenario A, AI identified scenario B which is caused by Spring
> 7.0.x adding verification step to recovery inside of spring-jms, which causes
> setting of varaiable in camel-jms prematurely while MQ is still unstable
> - After isolating Scenario B I refocused AI to focus on reproduction of
> Scenario A and was succesfull with reproduction camel 4.22.0 snapshot.
> Scenario A is complicated thread-parking induced issue. According to AI, it
> is much more likely to happen on old 4.8.9 than new 4.22.0, however it is
> still possible (as is seen in succesfull reproducer), thus Murphys laws would
> make it happen within 12 hours of deployed upgrade ;)
> - I instructed AI to double-confirm the reproduction of both scenarios by
> creating probes in source code using java agent and new Java 25 classfile
> API. it chose injection of custom JFR events and verified the scenarios
> - Both issues were also successfully reproduced on camel 4.22.0 final release
> Artifacts (AI generated, I assume availability of AI assistance, per AI
> activity on camel JIRA/Github):
> - Public GitHub repository link:
> [https://github.com/vdobos-trask/Reproducer_Camel_Jms_Failed_ReplyTo_Destination_2026]
>
> ([https://github.com/vdobos-trask/Reproducer_Camel_Jms_Failed_ReplyTo_Destination_2026.git])
> - Reproducer: folder ~/Reproducer (see ~/Reproducer/README.md), REQUIRES
> rootfull containers, expects docker with rw facl set on docker socket
> - Human-friendly interactive overview of both scenarios (html, created also
> for me so i can better understand the scenarios): primary scenario-based,
> based on theoretical best-case scenario -
> "{~}/explanation_docs/interactive/scenario/index.html", for complex scenario
> A there is also "{~}/explanation_docs/interactive/debugger/index.html", based
> on real reproducer log.
> - AI (and human ;)) readable markdown resources:
> "~/explanation_docs/markdown/*.md"
> Postscript:
> - I instructed AI to theoretize similar scenarios on camel-sjms. It
> successfully proposed several scenarios of unknown viability, even claimed
> some are worse than ones in camel-jms. AI readable markdown can be found in
> git in file ""~/explanation_docs/markdown/Unproven_camel_sjms.md". As this
> was outside of my assignment, i unfortunately didn't have time to explore it
> further, however I noticed you employed AI to do checks and fix several
> issues for 4.22.0 so I included it so you can explore this.
> - When trying to reproduce scenario A on camel 4.22.0, I instructed AI to
> perform minimal fix of scenario B as it shadowed A in most use-cases, I was
> warned that naive fix for B might introduce scenario C that will cause the
> same issue in different steps (scenario is briefly described in resources)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
