ammachado opened a new pull request, #30316: URL: https://github.com/apache/camel/pull/30316
# Description Round 3 of the camel-core test time work ([CAMEL-27095](https://issues.apache.org/jira/browse/CAMEL-27095)), after #30002 and #30214. **Draft:** the parallelism value is tuned on an 18-core machine, and this PR build is meant to show how it behaves on the 4 vCPU runners. ## Why the CPUs were idle: the JUnit ForkJoinPool saturated A JUnit `TestExecutionListener` on the test classpath (not part of this PR) gave the real start and end of each class. The camel-core log and the surefire times are misleading: 271 classes never log, and FJP join stacking skews the surefire times. The timeline showed the classes starting in waves, with one class running for up to 7 s while 1000+ waited. The log has the cause: ``` Junit ForkJoinPool saturated: running threads=0, pool size=512, queued tasks=1342 ``` A thread dump at the first saturation showed 505 of the 512 workers parked in `CompositeLock.acquireAllLocks` on the **read** lock of a `ReentrantReadWriteLock`, and 3 on its write lock: - `TestSupport` declares `@ResourceLock(value = SYSTEM_PROPERTIES, mode = READ)`, so almost every class reads that resource. - 6 classes write it (`@ResourceLock(SYSTEM_PROPERTIES)` or JUnit Pioneer `@SetSystemProperty`). Once one queues for the write lock, `ReentrantReadWriteLock` makes every later reader wait behind it until the longest running reader ends. Each blocked worker made the pool add a compensation thread, up to the 512 cap. The 6 writers are now `@Isolated`, which JUnit runs after the concurrent classes. They take 0.1 s in all. ## Parallelism 4 → 32 With the stalls gone, the pool kept to ~13 threads and the run went to **244 s**: the bursts of hundreds of compensation threads that followed each stall were what made the old run fast. The tests mostly wait (timers, latches, timeouts), so the parallelism goes to 32: steady, without the swings from 1 to 500 classes at once. | Local run (18 cores) | `mvn test` wall time | |---|---| | baseline (4 runs) | 125-148 s | | writers isolated, parallelism 4 | 244 s | | writers isolated, parallelism 16 | 125 s | | writers isolated, parallelism 32 | 106 s | | this PR (2 runs) | 97.0 s, 96.6 s | Also tried: the `WORKER_THREAD_POOL` executor (experimental in JUnit 6.1) at parallelism 4 took 310 s with 2 failures, and `maxPoolSize=2048` still saturated at 1764 threads. The pool size was lowered in CAMEL-18988 because of OOMs under `-Xmx2G`. 32 classes at once is far below the old bursts, and no run here hit an OOM, but the CI heap and flakiness on 4 vCPUs are what this draft is meant to check. ## Shorter serial tail - `RouteStopCutOffLogTest`: the 3 processed cut-off tests used a 5 s shutdown timeout, which they always wait out since the processor is blocked. They now use 1 s like the other tests of the class and `RouteStopCutOffLogNestedTest` (19.4 s → 7.1 s). This class ended the parallel phase. - `throttle.requests.ThrottlingGroupingTest`: the route holding the 3 throttled messages is stopped with a 100 ms timeout at the end of the test, instead of the CamelContext shutdown waiting out the 5 s period (6.7 s → 1.8 s). The 5000 ms period of CAMEL-26489 is kept. - `FileChangedReadLockTest` (and its 3 subclasses), `FutureConverterTest`, `FileConsumerRestartNotLeakThreadTest`: wait for an empty `InflightRepository` (the pattern of #30002), instead of the shutdown polling 1 s for the exchange in flight. All the changed classes also pass with `-Psanitizer`. Not changed, worth a look by reviewers: - `MarkerFileExclusiveReadLockStrategyRecursiveCleanupTest` (5 s): `initialDelay=1000` comes from the CAMEL-23495 flaky test fix. Its files are written before the routes are added, so the delay may not be needed, but I did not revert a deliberate fix without reproducing the flakiness. - `BackgroundTaskTest` (25 s) and `ClusteredEdgeCasesTest` (33 s) now bound the parallel phase. Their tests are built around time budgets. # Target - [x] I checked that the commit is targeting the correct branch (Camel 4 uses the `main` branch) # Tracking - [x] If this is a large change, bug fix, or code improvement, I checked there is a [JIRA issue](https://issues.apache.org/jira/browse/CAMEL) filed for the change (usually before you start working on it). # Apache Camel coding standards and style - [x] I checked that each commit in the pull request has a meaningful subject line and body. - [ ] I have run `mvn clean install -DskipTests` locally from root folder and I have committed all auto-generated changes. Only test sources and a test resource of `core/camel-core` change: I ran `mvn formatter:format impsort:sort` and the camel-core tests, and `git status` is clean. # AI-assisted contributions - [x] If this PR includes AI-generated code, commits have proper co-authorship attribution (e.g., `Co-authored-by` trailers) and the PR description identifies the AI tool used. 🤖 Generated with [Claude Code](https://claude.com/claude-code) on behalf of Adriano Machado (@ammachado) _This was generated by an AI agent and may contain inaccuracies. Please verify before relying on it._ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
