[
https://issues.apache.org/jira/browse/CAMEL-27105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18146263#comment-18146263
]
Torsten Mielke commented on CAMEL-27105:
----------------------------------------
h3. Root cause confirmed via local reproduction: Artemis wildcard-queue/address
binding race
Summary: this is *not* caused by CI resource contention (the original working
hypothesis). It's a race, reproducible locally, between the Camel route's
wildcard JMS consumer queue
registering on the Artemis broker and a producer's first-ever touch of a
brand-new matching address. Evidence points to the actual defect living in
Apache ActiveMQ Artemis, not in
Camel.
h4. Reproduction
Reproduced locally by repeatedly running the full {{camel-jms}} integration
test suite (isolating just this one test removes the scheduling jitter that
triggers the race, so it
won't reproduce that way). On a failing run, {{target/camel-jms-test.log}}
captures the embedded Artemis broker's own internal logging (address/queue
creation, routing, consumption
— all with millisecond timestamps), which is *not* visible in the CI console
log: {{camel-jms}}'s {{log4j2-test.properties}} routes this detail to that file
only, never to stdout.
h4. Evidence
Timestamped sequence from a captured failing run (Artemis 2.44.0):
{code}
15:11:46,444 Started route8 (activemq://queue:sport.%23) <- Camel
reports the wildcard consumer started
15:11:46,457 Artemis creates address "sport.pl.manu" <- test's
1st send (message A) touches its address
15:11:46,465 Artemis creates queue "sport.#" <- the
wildcard consumer's queue (8ms *after* the address)
15:11:46,465 Artemis creates queue "sport.pl.manu" (auto-created)
15:11:46,497 message A sent/routed to address "sport.pl.manu" <- wildcard
queue has existed for 32ms already
{code}
Message A (messageID=94) never appears in a "consuming ... from sport.#" log
line anywhere afterwards — it is delivered only to its own private auto-created
queue and never reaches
the wildcard consumer. Messages B/C/D, sent moments later (after the wildcard
queue already existed), each show an immediate, successful "consuming ... from
sport.#" line. There is
no ongoing per-message race: once the wildcard queue exists, every subsequent
new address binds to it correctly and instantly.
Critically, the wildcard consumer was already fully subscribed (since 465ms)
*before* the message was actually sent (497ms) — this isn't simply "producer
beat an absent consumer."
Artemis appears to resolve/cache an address's routing bindings at the
address's creation time and does not re-resolve them against a wildcard queue
that is created shortly
afterward, even when that happens well before the message is actually routed.
This also explains the ~20s elapsed time seen in CI: {{mock:other}}'s
{{assertIsSatisfied()}} hits the full 10s default latch timeout, and then
{{MockEndpoint}}'s
{{safeLatchReset()}} cleanup waits on the same never-satisfied latch for
*another* ~10s during teardown (two separate "The latch did not reach 0"
warnings, ~10.0s apart in the log).
10s + 10s plus overhead matches the 20.29–21.34s elapsed times observed in CI.
h4. Why it's flaky rather than always failing
CI's Failsafe configuration includes {{-Dfailsafe.rerunFailingTestsCount=1}},
so a failing attempt is automatically retried once; a fresh context/route start
gets a new roll of the
race and usually wins, so the build stays green while Develocity still
records the flake. The race itself is timing-sensitive (single-digit
milliseconds in the capture above), so it
does not reproduce on every run — including an initial clean local run — but
is made easier to hit by scheduling jitter from concurrently running tests
({{camel.failsafe.parallel=true}} for this module).
h4. Assessment: likely an upstream Artemis defect, not a Camel bug
JMS point-to-point semantics guarantee a message sent to a queue is retained
until a consumer takes it, regardless of producer/consumer timing. Wildcard
queue consumption is an
Artemis extension on top of that guarantee, whose entire value is that
producers and consumers stay unaware of each other's lifecycle. The race above
breaks that transparency due to
what looks like a stale/uncached routing decision inside Artemis, not
anything Camel or this test does incorrectly.
This isn't unprecedented for this feature area in Artemis:
* ARTEMIS-2304 — messages lost with (default)
auto-delete-queues/auto-delete-addresses, fixed as a broker bug.
* ARTEMIS-5337 — wildcard queue vs. new-consumer auto-create mishandling,
fixed in 2.41.0 (we're on 2.44.0, so not the same ticket, but the same general
feature area).
Neither is an exact match for this specific ordering, so this looks like a
distinct, previously unreported race. A report against Apache ActiveMQ Artemis
will follow separately.
Related Camel prior art: CAMEL-24034 hit the same general race *class*
(consumer not yet registered with the broker when sends started) for a topic
subscription, fixed there with a
hard-coded warm-up.
h4. Possible production impact
This is unlikely to affect a long-running, already-deployed consumer on an
ordinary restart (a durable wildcard queue binding, once established, persists
independently of consumer
connect/disconnect). It could plausibly still occur for: a first-ever rollout
of a new wildcard-consuming route while producers are already live, a
non-durable/auto-deleted wildcard
queue configuration (rebuilt on every consumer restart), or a
dynamic/per-tenant addressing scheme where brand-new addresses are created on
an ongoing basis. Not empirically
verified against a standalone (non-embedded) broker yet.
h4. Suggested next steps
# File a report against Apache ActiveMQ Artemis with the timeline above
(checking ARTEMIS-2304/5337/5672 first for overlap).
# Separately decide whether {{camel-jms}}'s test (and/or the component)
should carry an interim, defensive mitigation while the upstream fix is pending
— not because the defect is
Camel's, but because CI is flaky today and production users on current
Artemis versions are already exposed.
_Claude Code on behalf of Torsten Mielke_
> Fix flaky test ActiveMQConsumeWildcardQueuesIT
> ----------------------------------------------
>
> Key: CAMEL-27105
> URL: https://issues.apache.org/jira/browse/CAMEL-27105
> Project: Camel
> Issue Type: Test
> Components: camel-jms
> Affects Versions: 4.24.0
> Reporter: Aurélien Pupier
> Assignee: Torsten Mielke
> Priority: Minor
>
> [76 flaky in last 7
> days|https://develocity.apache.org/scans/tests?search.startTimeMax=1794956399999&search.startTimeMin=1794265200000&search.timeZoneId=Europe%2FParis&tests.container=org.apache.camel.component.jms.integration.activemq.ActiveMQConsumeWildcardQueuesIT&tests.test=testWildcard]
> {noformat}
> java.lang.AssertionError: mock://other Received message count. Expected: <2>
> but was: <1>
> at
> org.apache.camel.component.mock.MockEndpoint.fail(MockEndpoint.java:2079)
> at
> org.apache.camel.component.mock.MockEndpoint.assertEquals(MockEndpoint.java:2014)
>
> at
> org.apache.camel.component.mock.MockEndpoint.doAssertIsSatisfied(MockEndpoint.java:502)
>
> at
> org.apache.camel.component.mock.MockEndpoint.assertIsSatisfied(MockEndpoint.java:480)
>
> at
> org.apache.camel.component.mock.MockEndpoint.assertIsSatisfied(MockEndpoint.java:470)
>
> at
> org.apache.camel.component.jms.integration.activemq.ActiveMQConsumeWildcardQueuesIT.testWildcard(ActiveMQConsumeWildcardQueuesIT.java:59)
>
> {noformat}
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)