allthingssecurity opened a new pull request, #29666: URL: https://github.com/apache/camel/pull/29666
# Description [CAMEL-26865](https://issues.apache.org/jira/browse/CAMEL-26865) Builds on #29589 (CAMEL-26799, `ClusteredRoutePolicy` starts a route when the leadership was taken before the route could be started) and #29529 (CAMEL-26754), both merged: one commit on main. When the node takes the leadership, `ClusteredRoutePolicy` starts its routes in a loop inside one `try` block, after it has set its leader flag: - a route whose start failed (an endpoint that cannot connect yet at takeover) was only logged, and nothing tried again: the leadership was already taken, so only a later leadership change would start it. The node held the leadership and did not run the route, and no other node did; - the routes after it in the loop were not started at all. The stop on a leadership loss had the same single `try`: a route that failed to stop kept the routes after it running on a node that is no longer the leader, and stayed recorded as started, so it was not started again on the next leadership. `MasterRoutePolicy` (CAMEL-26540, #29123) and the camel-zookeeper-master consumer (CAMEL-25315) already retry a failed start while the node is the leader. A supervising route controller does not cover these routes when the policy has set them to `autoStartup=false` (it skips them). A route added at runtime while the node already is the leader stays supervised: its own back-off retry is not tied to the leadership; that is unchanged here. Change: - each route is started in its own `try`; a route that fails goes to the exception handler (WARN by default, with the route id and the retry delay) and stays with the stopped routes, and the other routes are started; - after a failed start, the policy schedules a retry: after the retry delay the policy thread applies the leadership again (the same hand-over to the policy thread as a leadership change, so the retry never starts a route on its timer thread): it starts the stopped routes if this node is still the leader, and does nothing to them if not. Each failed attempt goes to the exception handler with its number and the delay, and the start after failures is logged at INFO. The delay is 5 seconds by default, `setRetryDelay(Duration)` (positive; only on a policy created by hand, the factory and `ClusteredRouteController` use the default). One retry is scheduled at a time, on a timer thread that ends once it fired, so a policy keeps no thread (a policy per route with `ClusteredRoutePolicyFactory`); - each route is stopped in its own `try`; a route that is no longer running after a failed stop is recorded as stopped (so it is started again on the next leadership); one still running stays with the started routes and its stop is tried again after the retry delay while the node is not the leader. The stop retry (a route still running after a failed stop) has no test of its own: it needs a shutdown strategy that fails before the route stops. Not changed here: setting the delay through `ClusteredRoutePolicyFactory` or `ClusteredRouteController`; routes added while the CamelContext is suspended (the start does nothing and the route is recorded as started; the follow-up on suspend and resume handles the policy on resume); a route added at runtime on the leader whose start fails is started by the route controller, not by the policy, so the policy does not retry it. Docs: the retry and `setRetryDelay` in the clustering page. Tests, in `ClusteredEdgeCasesTest` (in-JVM test cluster view, no sleeps): - `testRouteThatFailsToStartIsStartedAgain`: two routes on one policy, the consumer of one fails its first start: both must end up started; - `testRetryDoesNotStartARouteOnAFollower`: a route that keeps failing is tried again; once the node lost the leadership, a retry must not start it, also when it could start; - `testRouteThatFailsToStopIsStartedAgainOnTheNextLeadership`: a second route policy throws in `onStop` of one route once: the other route must stop when the node loses the leadership, and both must start on the next leadership. On main without the change the three fail (a no-op `setRetryDelay` was added for the tests to compile). With the change the camel-core cluster tests pass (64, 0 failures, with the tests of #29529, #29589 and the other merged cluster fixes), the camel-file cluster tests too (16, 0 failures), and `ClusteredEdgeCasesTest` (12 tests) passes 240/240 over 20 runs in one JVM. A backport would take #29529 and #29589 first (same methods). Found with Specula (TLA+ model checked against traces of the real code) and confirmed with the tests above. # Target - [x] I checked that the commit is targeting the correct branch (Camel 4 uses the `main` branch) # Tracking - [x] If this is a large change, bug fix, or code improvement, I checked there is a [JIRA issue](https://issues.apache.org/jira/browse/CAMEL) filed for the change (usually before you start working on it): [CAMEL-26865](https://issues.apache.org/jira/browse/CAMEL-26865). # Apache Camel coding standards and style - [x] I checked that each commit in the pull request has a meaningful subject line and body. - [ ] I have run `mvn clean install -DskipTests` locally from root folder and I have committed all auto-generated changes. (I built the core modules with the formatter and import-sort plugins and ran the camel-core cluster tests; no generated file changes. I did not run the full root build.) # AI-assisted contributions - [x] If this PR includes AI-generated code, commits have proper co-authorship attribution (e.g., `Co-authored-by` trailers) and the PR description identifies the AI tool used. This PR was prepared with Claude Code (Claude Opus 5.5). The commit carries a `Co-Authored-By` trailer. _Claude Code on behalf of allthingssecurity_ 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
