[ 
https://issues.apache.org/jira/browse/CAMEL-27105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18146263#comment-18146263
 ] 

Torsten Mielke edited comment on CAMEL-27105 at 11/17/26, 2:58 PM:
-------------------------------------------------------------------

h3. Root cause confirmed via local reproduction: Artemis wildcard-queue/address 
binding race

Summary: this is *not* caused by CI resource contention (the original working 
hypothesis). It's a race, reproducible locally, between the Camel route's 
wildcard JMS consumer queue registering on the Artemis broker and a producer's 
first-ever touch of a brand-new matching address. Evidence points to the actual 
defect living in Apache ActiveMQ Artemis, not in Camel.
h4. Reproduction

Reproduced locally by repeatedly running the full {{camel-jms}} integration 
test suite (isolating just this one test removes the scheduling jitter that 
triggers the race, so it
won't reproduce that way). On a failing run, {{target/camel-jms-test.log}} 
captures the embedded Artemis broker's own internal logging (address/queue 
creation, routing, consumption — all with millisecond timestamps), which is 
*not* visible in the CI console log: {{{}camel-jms{}}}'s 
{{log4j2-test.properties}} routes this detail to that file only, never to 
stdout.
h4. Evidence

Timestamped sequence from a captured failing run (Artemis 2.44.0):
{code:java}
  15:11:46,444  Started route8 (activemq://queue:sport.%23)        <- Camel 
reports the wildcard consumer started
  15:11:46,457  Artemis creates address "sport.pl.manu"             <- test's 
1st send (message A) touches its address
  15:11:46,465  Artemis creates queue "sport.#"                     <- the 
wildcard consumer's queue (8ms *after* the address)
  15:11:46,465  Artemis creates queue "sport.pl.manu" (auto-created)
  15:11:46,497  message A sent/routed to address "sport.pl.manu"    <- wildcard 
queue has existed for 32ms already
  {code}
Message A (messageID=94) never appears in a "consuming ... from sport.#" log 
line anywhere afterwards — it is delivered only to its own private auto-created 
queue and never reaches the wildcard consumer. Messages B/C/D, sent moments 
later (after the wildcard queue already existed), each show an immediate, 
successful "consuming ... from sport.#" line. There is no ongoing per-message 
race: once the wildcard queue exists, every subsequent new address binds to it 
correctly and instantly.

Critically, the wildcard consumer was already fully subscribed (since 465ms) 
*before* the message was actually sent (497ms) — this isn't simply "producer 
beat an absent consumer."
Artemis appears to resolve/cache an address's routing bindings at the address's 
creation time and does not re-resolve them against a wildcard queue that is 
created shortly afterward, even when that happens well before the message is 
actually routed.

This also explains the ~20s elapsed time seen in CI: {{{}mock:other{}}}'s 
{{assertIsSatisfied()}} hits the full 10s default latch timeout, and then 
{{{}MockEndpoint{}}}'s {{safeLatchReset()}} cleanup waits on the same 
never-satisfied latch for *another* ~10s during teardown (two separate "The 
latch did not reach 0" warnings, ~10.0s apart in the log).
10s + 10s plus overhead matches the 20.29–21.34s elapsed times observed in CI.
h4. Why it's flaky rather than always failing

CI's Failsafe configuration includes 
{{{}-Dfailsafe.rerunFailingTestsCount=1{}}}, so a failing attempt is 
automatically retried once; a fresh context/route start gets a new roll of the 
race and usually wins, so the build stays green while Develocity still records 
the flake. The race itself is timing-sensitive (single-digit milliseconds in 
the capture above), so it does not reproduce on every run — including an 
initial clean local run — but is made easier to hit by scheduling jitter from 
concurrently running tests ({{{}camel.failsafe.parallel=true{}}} for this 
module).
h4. Assessment: likely an upstream Artemis defect, not a Camel bug

JMS point-to-point semantics guarantee a message sent to a queue is retained 
until a consumer takes it, regardless of producer/consumer timing. Wildcard 
queue consumption is an Artemis extension on top of that guarantee, whose 
entire value is that producers and consumers stay unaware of each other's 
lifecycle. The race above breaks that transparency due to what looks like a 
stale/uncached routing decision inside Artemis, not anything Camel or this test 
does incorrectly.

This isn't unprecedented for this feature area in Artemis:
 * ARTEMIS-2304 — messages lost with (default) 
auto-delete-queues/auto-delete-addresses, fixed as a broker bug.
 * ARTEMIS-5337 — wildcard queue vs. new-consumer auto-create mishandling, 
fixed in 2.41.0 (we're on 2.44.0, so not the same ticket, but the same general 
feature area).

Neither is an exact match for this specific ordering, so this looks like a 
distinct, previously unreported race. A report against Apache ActiveMQ Artemis 
will follow separately.

Related Camel prior art: CAMEL-24034 hit the same general race *class* 
(consumer not yet registered with the broker when sends started) for a topic 
subscription, fixed there with a hard-coded warm-up.
h4. Possible production impact

This is unlikely to affect a long-running, already-deployed consumer on an 
ordinary restart (a durable wildcard queue binding, once established, persists 
independently of consumer connect/disconnect). It could plausibly still occur 
for: a first-ever rollout of a new wildcard-consuming route while producers are 
already live, a non-durable/auto-deleted wildcard queue configuration (rebuilt 
on every consumer restart), or a dynamic/per-tenant addressing scheme where 
brand-new addresses are created on an ongoing basis. Not empirically verified 
against a standalone (non-embedded) broker yet.
h4. Suggested next steps
 # File a report against Apache ActiveMQ Artemis with the timeline above 
(checking ARTEMIS-2304/5337/5672 first for overlap).
 # Separately decide whether {{{}camel-jms{}}}'s test (and/or the component) 
should carry an interim, defensive mitigation while the upstream fix is pending 
— not because the defect is Camel's, but because CI is flaky today and 
production users on current Artemis versions are already exposed.

_Claude Code on behalf of Torsten Mielke_


was (Author: tmielke):
h3. Root cause confirmed via local reproduction: Artemis wildcard-queue/address 
binding race

  Summary: this is *not* caused by CI resource contention (the original working 
hypothesis). It's a race, reproducible locally, between the Camel route's 
wildcard JMS consumer queue
  registering on the Artemis broker and a producer's first-ever touch of a 
brand-new matching address. Evidence points to the actual defect living in 
Apache ActiveMQ Artemis, not in
  Camel.

  h4. Reproduction

  Reproduced locally by repeatedly running the full {{camel-jms}} integration 
test suite (isolating just this one test removes the scheduling jitter that 
triggers the race, so it
  won't reproduce that way). On a failing run, {{target/camel-jms-test.log}} 
captures the embedded Artemis broker's own internal logging (address/queue 
creation, routing, consumption
  — all with millisecond timestamps), which is *not* visible in the CI console 
log: {{camel-jms}}'s {{log4j2-test.properties}} routes this detail to that file 
only, never to stdout.

  h4. Evidence

  Timestamped sequence from a captured failing run (Artemis 2.44.0):
  {code}
  15:11:46,444  Started route8 (activemq://queue:sport.%23)        <- Camel 
reports the wildcard consumer started
  15:11:46,457  Artemis creates address "sport.pl.manu"             <- test's 
1st send (message A) touches its address
  15:11:46,465  Artemis creates queue "sport.#"                     <- the 
wildcard consumer's queue (8ms *after* the address)
  15:11:46,465  Artemis creates queue "sport.pl.manu" (auto-created)
  15:11:46,497  message A sent/routed to address "sport.pl.manu"    <- wildcard 
queue has existed for 32ms already
  {code}

  Message A (messageID=94) never appears in a "consuming ... from sport.#" log 
line anywhere afterwards — it is delivered only to its own private auto-created 
queue and never reaches
  the wildcard consumer. Messages B/C/D, sent moments later (after the wildcard 
queue already existed), each show an immediate, successful "consuming ... from 
sport.#" line. There is
  no ongoing per-message race: once the wildcard queue exists, every subsequent 
new address binds to it correctly and instantly.

  Critically, the wildcard consumer was already fully subscribed (since 465ms) 
*before* the message was actually sent (497ms) — this isn't simply "producer 
beat an absent consumer."
  Artemis appears to resolve/cache an address's routing bindings at the 
address's creation time and does not re-resolve them against a wildcard queue 
that is created shortly
  afterward, even when that happens well before the message is actually routed.

  This also explains the ~20s elapsed time seen in CI: {{mock:other}}'s 
{{assertIsSatisfied()}} hits the full 10s default latch timeout, and then 
{{MockEndpoint}}'s
  {{safeLatchReset()}} cleanup waits on the same never-satisfied latch for 
*another* ~10s during teardown (two separate "The latch did not reach 0" 
warnings, ~10.0s apart in the log).
  10s + 10s plus overhead matches the 20.29–21.34s elapsed times observed in CI.

  h4. Why it's flaky rather than always failing

  CI's Failsafe configuration includes {{-Dfailsafe.rerunFailingTestsCount=1}}, 
so a failing attempt is automatically retried once; a fresh context/route start 
gets a new roll of the
  race and usually wins, so the build stays green while Develocity still 
records the flake. The race itself is timing-sensitive (single-digit 
milliseconds in the capture above), so it
  does not reproduce on every run — including an initial clean local run — but 
is made easier to hit by scheduling jitter from concurrently running tests
  ({{camel.failsafe.parallel=true}} for this module).

  h4. Assessment: likely an upstream Artemis defect, not a Camel bug

  JMS point-to-point semantics guarantee a message sent to a queue is retained 
until a consumer takes it, regardless of producer/consumer timing. Wildcard 
queue consumption is an
  Artemis extension on top of that guarantee, whose entire value is that 
producers and consumers stay unaware of each other's lifecycle. The race above 
breaks that transparency due to
  what looks like a stale/uncached routing decision inside Artemis, not 
anything Camel or this test does incorrectly.

  This isn't unprecedented for this feature area in Artemis:
  * ARTEMIS-2304 — messages lost with (default) 
auto-delete-queues/auto-delete-addresses, fixed as a broker bug.
  * ARTEMIS-5337 — wildcard queue vs. new-consumer auto-create mishandling, 
fixed in 2.41.0 (we're on 2.44.0, so not the same ticket, but the same general 
feature area).

  Neither is an exact match for this specific ordering, so this looks like a 
distinct, previously unreported race. A report against Apache ActiveMQ Artemis 
will follow separately.

  Related Camel prior art: CAMEL-24034 hit the same general race *class* 
(consumer not yet registered with the broker when sends started) for a topic 
subscription, fixed there with a
  hard-coded warm-up.

  h4. Possible production impact

  This is unlikely to affect a long-running, already-deployed consumer on an 
ordinary restart (a durable wildcard queue binding, once established, persists 
independently of consumer
  connect/disconnect). It could plausibly still occur for: a first-ever rollout 
of a new wildcard-consuming route while producers are already live, a 
non-durable/auto-deleted wildcard
  queue configuration (rebuilt on every consumer restart), or a 
dynamic/per-tenant addressing scheme where brand-new addresses are created on 
an ongoing basis. Not empirically
  verified against a standalone (non-embedded) broker yet.

  h4. Suggested next steps

  # File a report against Apache ActiveMQ Artemis with the timeline above 
(checking ARTEMIS-2304/5337/5672 first for overlap).
  # Separately decide whether {{camel-jms}}'s test (and/or the component) 
should carry an interim, defensive mitigation while the upstream fix is pending 
— not because the defect is
  Camel's, but because CI is flaky today and production users on current 
Artemis versions are already exposed.

  _Claude Code on behalf of Torsten Mielke_

> Fix flaky test ActiveMQConsumeWildcardQueuesIT
> ----------------------------------------------
>
>                 Key: CAMEL-27105
>                 URL: https://issues.apache.org/jira/browse/CAMEL-27105
>             Project: Camel
>          Issue Type: Test
>          Components: camel-jms
>    Affects Versions: 4.24.0
>            Reporter: Aurélien Pupier
>            Assignee: Torsten Mielke
>            Priority: Minor
>
> [76 flaky in last 7 
> days|https://develocity.apache.org/scans/tests?search.startTimeMax=1794956399999&search.startTimeMin=1794265200000&search.timeZoneId=Europe%2FParis&tests.container=org.apache.camel.component.jms.integration.activemq.ActiveMQConsumeWildcardQueuesIT&tests.test=testWildcard]
> {noformat}
> java.lang.AssertionError: mock://other Received message count. Expected: <2> 
> but was: <1>     
>       at 
> org.apache.camel.component.mock.MockEndpoint.fail(MockEndpoint.java:2079)    
>       at 
> org.apache.camel.component.mock.MockEndpoint.assertEquals(MockEndpoint.java:2014)
>     
>       at 
> org.apache.camel.component.mock.MockEndpoint.doAssertIsSatisfied(MockEndpoint.java:502)
>       
>       at 
> org.apache.camel.component.mock.MockEndpoint.assertIsSatisfied(MockEndpoint.java:480)
>         
>       at 
> org.apache.camel.component.mock.MockEndpoint.assertIsSatisfied(MockEndpoint.java:470)
>         
>       at 
> org.apache.camel.component.jms.integration.activemq.ActiveMQConsumeWildcardQueuesIT.testWildcard(ActiveMQConsumeWildcardQueuesIT.java:59)
>     
> {noformat}



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

Reply via email to