ammachado opened a new pull request, #30316:
URL: https://github.com/apache/camel/pull/30316

   # Description
   
   Round 3 of the camel-core test time work 
([CAMEL-27095](https://issues.apache.org/jira/browse/CAMEL-27095)), after 
#30002 and #30214. **Draft:** the parallelism value is tuned on an 18-core 
machine, and this PR build is meant to show how it behaves on the 4 vCPU 
runners.
   
   ## Why the CPUs were idle: the JUnit ForkJoinPool saturated
   
   A JUnit `TestExecutionListener` on the test classpath (not part of this PR) 
gave the real start and end of each class. The camel-core log and the surefire 
times are misleading: 271 classes never log, and FJP join stacking skews the 
surefire times. The timeline showed the classes starting in waves, with one 
class running for up to 7 s while 1000+ waited. The log has the cause:
   
   ```
   Junit ForkJoinPool saturated: running threads=0, pool size=512, queued 
tasks=1342
   ```
   
   A thread dump at the first saturation showed 505 of the 512 workers parked 
in `CompositeLock.acquireAllLocks` on the **read** lock of a 
`ReentrantReadWriteLock`, and 3 on its write lock:
   
   - `TestSupport` declares `@ResourceLock(value = SYSTEM_PROPERTIES, mode = 
READ)`, so almost every class reads that resource.
   - 6 classes write it (`@ResourceLock(SYSTEM_PROPERTIES)` or JUnit Pioneer 
`@SetSystemProperty`). Once one queues for the write lock, 
`ReentrantReadWriteLock` makes every later reader wait behind it until the 
longest running reader ends. Each blocked worker made the pool add a 
compensation thread, up to the 512 cap.
   
   The 6 writers are now `@Isolated`, which JUnit runs after the concurrent 
classes. They take 0.1 s in all.
   
   ## Parallelism 4 → 32
   
   With the stalls gone, the pool kept to ~13 threads and the run went to **244 
s**: the bursts of hundreds of compensation threads that followed each stall 
were what made the old run fast. The tests mostly wait (timers, latches, 
timeouts), so the parallelism goes to 32: steady, without the swings from 1 to 
500 classes at once.
   
   | Local run (18 cores) | `mvn test` wall time |
   |---|---|
   | baseline (4 runs) | 125-148 s |
   | writers isolated, parallelism 4 | 244 s |
   | writers isolated, parallelism 16 | 125 s |
   | writers isolated, parallelism 32 | 106 s |
   | this PR (2 runs) | 97.0 s, 96.6 s |
   
   Also tried: the `WORKER_THREAD_POOL` executor (experimental in JUnit 6.1) at 
parallelism 4 took 310 s with 2 failures, and `maxPoolSize=2048` still 
saturated at 1764 threads.
   
   The pool size was lowered in CAMEL-18988 because of OOMs under `-Xmx2G`. 32 
classes at once is far below the old bursts, and no run here hit an OOM, but 
the CI heap and flakiness on 4 vCPUs are what this draft is meant to check.
   
   ## Shorter serial tail
   
   - `RouteStopCutOffLogTest`: the 3 processed cut-off tests used a 5 s 
shutdown timeout, which they always wait out since the processor is blocked. 
They now use 1 s like the other tests of the class and 
`RouteStopCutOffLogNestedTest` (19.4 s → 7.1 s). This class ended the parallel 
phase.
   - `throttle.requests.ThrottlingGroupingTest`: the route holding the 3 
throttled messages is stopped with a 100 ms timeout at the end of the test, 
instead of the CamelContext shutdown waiting out the 5 s period (6.7 s → 1.8 
s). The 5000 ms period of CAMEL-26489 is kept.
   - `FileChangedReadLockTest` (and its 3 subclasses), `FutureConverterTest`, 
`FileConsumerRestartNotLeakThreadTest`: wait for an empty `InflightRepository` 
(the pattern of #30002), instead of the shutdown polling 1 s for the exchange 
in flight.
   
   All the changed classes also pass with `-Psanitizer`.
   
   Not changed, worth a look by reviewers:
   - `MarkerFileExclusiveReadLockStrategyRecursiveCleanupTest` (5 s): 
`initialDelay=1000` comes from the CAMEL-23495 flaky test fix. Its files are 
written before the routes are added, so the delay may not be needed, but I did 
not revert a deliberate fix without reproducing the flakiness.
   - `BackgroundTaskTest` (25 s) and `ClusteredEdgeCasesTest` (33 s) now bound 
the parallel phase. Their tests are built around time budgets.
   
   # Target
   
   - [x] I checked that the commit is targeting the correct branch (Camel 4 
uses the `main` branch)
   
   # Tracking
   - [x] If this is a large change, bug fix, or code improvement, I checked 
there is a [JIRA issue](https://issues.apache.org/jira/browse/CAMEL) filed for 
the change (usually before you start working on it).
   
   # Apache Camel coding standards and style
   
   - [x] I checked that each commit in the pull request has a meaningful 
subject line and body.
   
   - [ ] I have run `mvn clean install -DskipTests` locally from root folder 
and I have committed all auto-generated changes.
   
   Only test sources and a test resource of `core/camel-core` change: I ran 
`mvn formatter:format impsort:sort` and the camel-core tests, and `git status` 
is clean.
   
   # AI-assisted contributions
   
   - [x] If this PR includes AI-generated code, commits have proper 
co-authorship attribution (e.g., `Co-authored-by` trailers) and the PR 
description identifies the AI tool used.
   
   🤖 Generated with [Claude Code](https://claude.com/claude-code) on behalf of 
Adriano Machado (@ammachado)
   
   _This was generated by an AI agent and may contain inaccuracies. Please 
verify before relying on it._


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to