This is an automated email from the ASF dual-hosted git repository.
apupier pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/camel.git
The following commit(s) were added to refs/heads/main by this push:
new a5900e000771 CAMEL-26373: camel-opentelemetry-metrics - Fix flaky
OpenTelemetryRoutePolicyTest
a5900e000771 is described below
commit a5900e000771c00037411d769b3592675052c543
Author: Aurélien Pupier <[email protected]>
AuthorDate: Wed Oct 28 14:25:30 2026 +0100
CAMEL-26373: camel-opentelemetry-metrics - Fix flaky
OpenTelemetryRoutePolicyTest
OpenTelemetryRoutePolicyTest asserts that the maximum duration recorded
in the camel.route.policy histogram is below the route delay plus a
TOLERANCE of 20 ms. The routes only delay 20 ms (foo) and 50 ms (bar),
so on a loaded CI machine a single scheduling pause, GC pause or JIT
warm-up of about 20 ms is enough to break the upper bound and fail with:
AssertionFailedError: max value ==> expected: <true> but was: <false>
at OpenTelemetryRoutePolicyTest.verifyHistogramMetric
The policy starts its timer in onExchangeBegin and records the duration
in onExchangeDone, so any stall of the route thread between these two
points is part of the recorded max. The lower bound assertions (min and
sum) cannot fail this way, only the upper bound is timing sensitive.
Raise TOLERANCE from 20 ms to 100 ms, the value already used by
OpenTelemetryRoutePolicyLongTaskTest and
OpenTelemetryExchangeEventNotifierTest in the same module.
How the problem was reproduced
Running the test 10 times with the original tolerance, while 2x nproc
(44) busy loops saturated the CPUs, did not reproduce the flakiness
(10/10 passed), and 10/10 also passed with 100 ms. Local CPU load alone
was not enough to trigger a stall.
The failure was then reproduced deterministically with a temporary copy
of the test (not committed) whose route foo had an artificial stall
right after the delay, for the first message only:
from("direct:foo").routeId("foo")
.delay(DELAY_FOO)
.process(e -> {
if ("Hello 0".equals(e.getIn().getBody(String.class))) {
Thread.sleep(60);
}
})
.to("mock:result");
The sleep happens between onExchangeBegin and onExchangeDone, so it is
included in the recorded max, simulating a one-off pause on a busy CI
machine. The max of route foo becomes about 80 ms:
- TOLERANCE = 20 ms (limit 40 ms): fails every time (3 of 3 runs) with
the same "max value ==> expected: <true> but was: <false>" as reported
- TOLERANCE = 100 ms (limit 120 ms): passes
This confirms the root cause is a too tight upper bound, and that the
new tolerance absorbs stalls of that magnitude.
Co-authored-by: IBM Bob 2.2.1
Signed-off-by: Aurélien Pupier <[email protected]>
---
.../opentelemetry/metrics/routepolicy/OpenTelemetryRoutePolicyTest.java | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git
a/components/camel-opentelemetry-metrics/src/test/java/org/apache/camel/opentelemetry/metrics/routepolicy/OpenTelemetryRoutePolicyTest.java
b/components/camel-opentelemetry-metrics/src/test/java/org/apache/camel/opentelemetry/metrics/routepolicy/OpenTelemetryRoutePolicyTest.java
index 072e85136715..c91fff9b40f4 100644
---
a/components/camel-opentelemetry-metrics/src/test/java/org/apache/camel/opentelemetry/metrics/routepolicy/OpenTelemetryRoutePolicyTest.java
+++
b/components/camel-opentelemetry-metrics/src/test/java/org/apache/camel/opentelemetry/metrics/routepolicy/OpenTelemetryRoutePolicyTest.java
@@ -33,7 +33,7 @@ public class OpenTelemetryRoutePolicyTest extends
AbstractOpenTelemetryRoutePoli
private static final long DELAY_FOO = 20;
private static final long DELAY_BAR = 50;
- private static final long TOLERANCE = 20L;
+ private static final long TOLERANCE = 100L;
@Test
public void testMetricsRoutePolicy() throws Exception {
