apupier opened a new pull request, #28632:
URL: https://github.com/apache/camel/pull/28632
OpenTelemetryRoutePolicyTest asserts that the maximum duration recorded in
the camel.route.policy histogram is below the route delay plus a TOLERANCE of
20 ms. The routes only delay 20 ms (foo) and 50 ms (bar), so on a loaded CI
machine a single scheduling pause, GC pause or JIT warm-up of about 20 ms is
enough to break the upper bound and fail with:
AssertionFailedError: max value ==> expected: <true> but was: <false>
at OpenTelemetryRoutePolicyTest.verifyHistogramMetric
The policy starts its timer in onExchangeBegin and records the duration in
onExchangeDone, so any stall of the route thread between these two points is
part of the recorded max. The lower bound assertions (min and sum) cannot fail
this way, only the upper bound is timing sensitive.
Raise TOLERANCE from 20 ms to 100 ms, the value already used by
OpenTelemetryRoutePolicyLongTaskTest and
OpenTelemetryExchangeEventNotifierTest in the same module.
How the problem was reproduced
Running the test 10 times with the original tolerance, while 2x nproc (44)
busy loops saturated the CPUs, did not reproduce the flakiness (10/10 passed),
and 10/10 also passed with 100 ms. Local CPU load alone was not enough to
trigger a stall.
The failure was then reproduced deterministically with a temporary copy of
the test (not committed) whose route foo had an artificial stall right after
the delay, for the first message only:
from("direct:foo").routeId("foo")
.delay(DELAY_FOO)
.process(e -> {
if ("Hello 0".equals(e.getIn().getBody(String.class))) {
Thread.sleep(60);
}
})
.to("mock:result");
The sleep happens between onExchangeBegin and onExchangeDone, so it is
included in the recorded max, simulating a one-off pause on a busy CI machine.
The max of route foo becomes about 80 ms:
- TOLERANCE = 20 ms (limit 40 ms): fails every time (3 of 3 runs) with
the same "max value ==> expected: <true> but was: <false>" as reported
- TOLERANCE = 100 ms (limit 120 ms): passes
This confirms the root cause is a too tight upper bound, and that the new
tolerance absorbs stalls of that magnitude.
Co-authored-by: IBM Bob 2.2.1
# Description
<!--
- Write a pull request description that is detailed enough to understand
what the pull request does, how, and why.
-->
# Target
- [ ] I checked that the commit is targeting the correct branch (Camel 4
uses the `main` branch)
# Tracking
- [ ] If this is a large change, bug fix, or code improvement, I checked
there is a [JIRA issue](https://issues.apache.org/jira/browse/CAMEL) filed for
the change (usually before you start working on it).
<!--
# *Note*: trivial changes like, typos, minor documentation fixes and other
small items do not require a JIRA issue. In this case your pull request should
address just this issue, without pulling in other changes.
-->
# Apache Camel coding standards and style
- [ ] I checked that each commit in the pull request has a meaningful
subject line and body.
<!--
If you're unsure, you can format the pull request title like `[CAMEL-XXX]
Fixes bug in camel-file component`, where you replace `CAMEL-XXX` with the
appropriate JIRA issue.
-->
- [ ] I have run `mvn clean install -DskipTests` locally from root folder
and I have committed all auto-generated changes.
<!--
You can run the aforementioned command in your module so that the build
auto-formats your code. This will also be verified as part of the checks and
your PR may be rejected if if there are uncommited changes after running `mvn
clean install -DskipTests`.
You can learn more about the contribution guidelines at
https://github.com/apache/camel/blob/main/CONTRIBUTING.md
-->
# AI-assisted contributions
- [ ] If this PR includes AI-generated code, commits have proper
co-authorship attribution (e.g., `Co-authored-by` trailers) and the PR
description identifies the AI tool used.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
