[ 
https://issues.apache.org/jira/browse/CAMEL-27105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18146263#comment-18146263
 ] 

Torsten Mielke commented on CAMEL-27105:
----------------------------------------

h3. Root cause confirmed via local reproduction: Artemis wildcard-queue/address 
binding race

  Summary: this is *not* caused by CI resource contention (the original working 
hypothesis). It's a race, reproducible locally, between the Camel route's 
wildcard JMS consumer queue
  registering on the Artemis broker and a producer's first-ever touch of a 
brand-new matching address. Evidence points to the actual defect living in 
Apache ActiveMQ Artemis, not in
  Camel.

  h4. Reproduction

  Reproduced locally by repeatedly running the full {{camel-jms}} integration 
test suite (isolating just this one test removes the scheduling jitter that 
triggers the race, so it
  won't reproduce that way). On a failing run, {{target/camel-jms-test.log}} 
captures the embedded Artemis broker's own internal logging (address/queue 
creation, routing, consumption
  — all with millisecond timestamps), which is *not* visible in the CI console 
log: {{camel-jms}}'s {{log4j2-test.properties}} routes this detail to that file 
only, never to stdout.

  h4. Evidence

  Timestamped sequence from a captured failing run (Artemis 2.44.0):
  {code}
  15:11:46,444  Started route8 (activemq://queue:sport.%23)        <- Camel 
reports the wildcard consumer started
  15:11:46,457  Artemis creates address "sport.pl.manu"             <- test's 
1st send (message A) touches its address
  15:11:46,465  Artemis creates queue "sport.#"                     <- the 
wildcard consumer's queue (8ms *after* the address)
  15:11:46,465  Artemis creates queue "sport.pl.manu" (auto-created)
  15:11:46,497  message A sent/routed to address "sport.pl.manu"    <- wildcard 
queue has existed for 32ms already
  {code}

  Message A (messageID=94) never appears in a "consuming ... from sport.#" log 
line anywhere afterwards — it is delivered only to its own private auto-created 
queue and never reaches
  the wildcard consumer. Messages B/C/D, sent moments later (after the wildcard 
queue already existed), each show an immediate, successful "consuming ... from 
sport.#" line. There is
  no ongoing per-message race: once the wildcard queue exists, every subsequent 
new address binds to it correctly and instantly.

  Critically, the wildcard consumer was already fully subscribed (since 465ms) 
*before* the message was actually sent (497ms) — this isn't simply "producer 
beat an absent consumer."
  Artemis appears to resolve/cache an address's routing bindings at the 
address's creation time and does not re-resolve them against a wildcard queue 
that is created shortly
  afterward, even when that happens well before the message is actually routed.

  This also explains the ~20s elapsed time seen in CI: {{mock:other}}'s 
{{assertIsSatisfied()}} hits the full 10s default latch timeout, and then 
{{MockEndpoint}}'s
  {{safeLatchReset()}} cleanup waits on the same never-satisfied latch for 
*another* ~10s during teardown (two separate "The latch did not reach 0" 
warnings, ~10.0s apart in the log).
  10s + 10s plus overhead matches the 20.29–21.34s elapsed times observed in CI.

  h4. Why it's flaky rather than always failing

  CI's Failsafe configuration includes {{-Dfailsafe.rerunFailingTestsCount=1}}, 
so a failing attempt is automatically retried once; a fresh context/route start 
gets a new roll of the
  race and usually wins, so the build stays green while Develocity still 
records the flake. The race itself is timing-sensitive (single-digit 
milliseconds in the capture above), so it
  does not reproduce on every run — including an initial clean local run — but 
is made easier to hit by scheduling jitter from concurrently running tests
  ({{camel.failsafe.parallel=true}} for this module).

  h4. Assessment: likely an upstream Artemis defect, not a Camel bug

  JMS point-to-point semantics guarantee a message sent to a queue is retained 
until a consumer takes it, regardless of producer/consumer timing. Wildcard 
queue consumption is an
  Artemis extension on top of that guarantee, whose entire value is that 
producers and consumers stay unaware of each other's lifecycle. The race above 
breaks that transparency due to
  what looks like a stale/uncached routing decision inside Artemis, not 
anything Camel or this test does incorrectly.

  This isn't unprecedented for this feature area in Artemis:
  * ARTEMIS-2304 — messages lost with (default) 
auto-delete-queues/auto-delete-addresses, fixed as a broker bug.
  * ARTEMIS-5337 — wildcard queue vs. new-consumer auto-create mishandling, 
fixed in 2.41.0 (we're on 2.44.0, so not the same ticket, but the same general 
feature area).

  Neither is an exact match for this specific ordering, so this looks like a 
distinct, previously unreported race. A report against Apache ActiveMQ Artemis 
will follow separately.

  Related Camel prior art: CAMEL-24034 hit the same general race *class* 
(consumer not yet registered with the broker when sends started) for a topic 
subscription, fixed there with a
  hard-coded warm-up.

  h4. Possible production impact

  This is unlikely to affect a long-running, already-deployed consumer on an 
ordinary restart (a durable wildcard queue binding, once established, persists 
independently of consumer
  connect/disconnect). It could plausibly still occur for: a first-ever rollout 
of a new wildcard-consuming route while producers are already live, a 
non-durable/auto-deleted wildcard
  queue configuration (rebuilt on every consumer restart), or a 
dynamic/per-tenant addressing scheme where brand-new addresses are created on 
an ongoing basis. Not empirically
  verified against a standalone (non-embedded) broker yet.

  h4. Suggested next steps

  # File a report against Apache ActiveMQ Artemis with the timeline above 
(checking ARTEMIS-2304/5337/5672 first for overlap).
  # Separately decide whether {{camel-jms}}'s test (and/or the component) 
should carry an interim, defensive mitigation while the upstream fix is pending 
— not because the defect is
  Camel's, but because CI is flaky today and production users on current 
Artemis versions are already exposed.

  _Claude Code on behalf of Torsten Mielke_

> Fix flaky test ActiveMQConsumeWildcardQueuesIT
> ----------------------------------------------
>
>                 Key: CAMEL-27105
>                 URL: https://issues.apache.org/jira/browse/CAMEL-27105
>             Project: Camel
>          Issue Type: Test
>          Components: camel-jms
>    Affects Versions: 4.24.0
>            Reporter: Aurélien Pupier
>            Assignee: Torsten Mielke
>            Priority: Minor
>
> [76 flaky in last 7 
> days|https://develocity.apache.org/scans/tests?search.startTimeMax=1794956399999&search.startTimeMin=1794265200000&search.timeZoneId=Europe%2FParis&tests.container=org.apache.camel.component.jms.integration.activemq.ActiveMQConsumeWildcardQueuesIT&tests.test=testWildcard]
> {noformat}
> java.lang.AssertionError: mock://other Received message count. Expected: <2> 
> but was: <1>     
>       at 
> org.apache.camel.component.mock.MockEndpoint.fail(MockEndpoint.java:2079)    
>       at 
> org.apache.camel.component.mock.MockEndpoint.assertEquals(MockEndpoint.java:2014)
>     
>       at 
> org.apache.camel.component.mock.MockEndpoint.doAssertIsSatisfied(MockEndpoint.java:502)
>       
>       at 
> org.apache.camel.component.mock.MockEndpoint.assertIsSatisfied(MockEndpoint.java:480)
>         
>       at 
> org.apache.camel.component.mock.MockEndpoint.assertIsSatisfied(MockEndpoint.java:470)
>         
>       at 
> org.apache.camel.component.jms.integration.activemq.ActiveMQConsumeWildcardQueuesIT.testWildcard(ActiveMQConsumeWildcardQueuesIT.java:59)
>     
> {noformat}



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

Reply via email to