allthingssecurity opened a new pull request, #30459: URL: https://github.com/apache/camel/pull/30459
# Description [CAMEL-27212](https://issues.apache.org/jira/browse/CAMEL-27212) Two problems with the thread pool of the Circuit Breaker EIP (camel-resilience4j) when the timeout is enabled: - Asynchronous mode (`asynchronous=true`): the reifier creates one `ScheduledThreadPoolExecutor` (or takes the `timeoutExecutorService`) and the processor uses it both to run the calls and as the scheduler of the `TimeLimiter`, so the timeout of a call is a task of the same pool. When the calls block their threads (a synchronous http call to a server that does not answer, which is the case the timeout is for) and there are as many of them as threads in the pool (10 with the default profile), no thread is left to run the timeouts: no call times out, no fallback runs, the exchanges wait for as long as the server does not answer, and the circuit breaker never records the failures, so it does not open. The docs example (`asynchronous(true).timeoutEnabled(true).timeoutDuration(2000)` around an `http:` call) runs into it under load. `ResilienceAsyncTimeoutTest` uses `delay()`, which gives the thread back, so it does not show it. - Route restart (any mode): the reifier creates the thread pool once, with the route, but `ResilienceProcessor.doStop` shuts it down. After `stopRoute` + `startRoute` (JMX, supervising route controller, CLI) the pool is terminated. The rejection policy of the pool (CallerRuns) discards a task once the pool is shut down, without an exception: in synchronous mode every call times out without running and gets the fallback (and the circuit breaker opens); in asynchronous mode the call and its timeout are both discarded, so the exchange never completes and the caller waits for ever. It stays so until the route is removed and added again. Fix: - In asynchronous mode the processor schedules the timeouts on threads of its own: a scheduled thread pool (default profile) created when the processor starts and shut down when the route is removed. The thread pool of the reifier (or `timeoutExecutorService`) runs only the calls. A `scheduledExecutorService` set on the processor by code is still used for the timeouts. The fallback, and the rest of the route, run on the timeout thread when the call times out, as they ran on a thread of the shared pool before (one more sentence in the docs). - The thread pools are shut down in `doShutdown` (route removal, CamelContext stop) instead of `doStop`, as the other EIPs with their own thread pool do (Threads, Multicast, WireTap). Model: a small TLA+ model (`R4jAsyncTimeout.tla`: two calls that hang, a pool of one thread, the timeouts in the same pool or on their own threads, a route stop + start) checks "every call gets an answer" and "a call is not answered without running because of a restart". On main TLC reports a deadlock in 4 states (the call blocks the only thread, its timeout waits behind it) and the restart invariant is violated in 3 states; with the change both hold (32 and 64 distinct states). Tests: - `ResilienceAsyncTimeoutBlockedPoolTest`: asynchronous mode, timeout 200 ms, a `timeoutExecutorService` with one thread, a step that blocks: the fallback must come within 5 seconds. - `ResilienceRouteRestartTest`: synchronous and asynchronous mode with a timeout; after `stopRoute` + `startRoute` the steps of the circuit breaker must still run. On main all three fail: no fallback within 5 seconds (blocked pool); after the restart the synchronous route answers `Fallback` instead of `Bye World`, and the asynchronous one never answers. With the change they pass, and the camel-resilience4j tests pass (101 run, 0 failures, built with camel-management of main for `ResilienceFallbackManagementTest`). The two new classes 20 times in one JVM: 60 run, 0 failures. Not changed: the reifier still requires a `ScheduledExecutorService` for `timeoutExecutorService` in asynchronous mode (it now only runs the calls, so the check could be relaxed in a follow-up). camel-microprofile-fault-tolerance is not affected by the first problem (SmallRye uses its own timer), and its synchronous guard does not use the thread pool that its `doStop` shuts down. # Target - [x] I checked that the commit is targeting the correct branch (Camel 4 uses the `main` branch) # Tracking - [x] If this is a large change, bug fix, or code improvement, I checked there is a [JIRA issue](https://issues.apache.org/jira/browse/CAMEL) filed for the change (usually before you start working on it): [CAMEL-27212](https://issues.apache.org/jira/browse/CAMEL-27212). # Apache Camel coding standards and style - [x] I checked that each commit in the pull request has a meaningful subject line and body. - [ ] I have run `mvn clean install -DskipTests` locally from root folder and I have committed all auto-generated changes. (I built the touched modules with the formatter and import-sort plugins; the docs change is in the EIP page and its catalog copy. I did not run the full root build.) # AI-assisted contributions - [x] If this PR includes AI-generated code, commits have proper co-authorship attribution (e.g., `Co-authored-by` trailers) and the PR description identifies the AI tool used. This PR was prepared with Claude Code (Claude Opus 5.5). The commit carries a `Co-Authored-By` trailer. _Claude Code on behalf of allthingssecurity_ 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
