shashank created CAMEL-25010:
--------------------------------
Summary: Supervising route controller: a restart attempt that is
already running starts the route after stopRoute() cancelled it
Key: CAMEL-25010
URL: https://issues.apache.org/jira/browse/CAMEL-25010
Project: Camel
Issue Type: Bug
Components: camel-core
Reporter: shashank
The restart attempt of {{DefaultSupervisingRouteController.RouteManager.start}}
({{DefaultSupervisingRouteController.java:677-703}}) checks only
{{getCamelContext().isRunAllowed()}} ({{:681}}) and then calls
{{doStartRoute(r, false, ...)}} ({{:691}}), which takes the controller lock and
starts the route. It does not check, once it holds the lock, whether its task
is still the route's restart task and still active.
{{stopRoute(id)}} ({{doStopRoute}}, {{:443-463}}) takes the same lock, cancels
the task ({{routeManager.release}}, {{:751-760}}) and stops the route.
{{BackOffTimerTask.cancel}} only prevents future runs ({{run()}} checks the
status once on entry, {{BackOffTimerTask.java:171}}). An attempt that has
entered {{run()}} before the cancel goes on after the stop and starts the route
the user just stopped. The route then runs, is not supervised (no task in
{{routes}}), and nothing reports it.
The attempt reaches the lock after the stop whenever it is waiting for the
controller lock during the stop. The lock is shared by all routes, so a slow
stop of any other route is enough.
*Reproduction*:
{{natural}} (no thread is held by the harness): route r1 fails to start (broker
down), back-off 400 ms; route r2 has a consumer that takes 600 ms to stop.
Thread 1 calls {{stopRoute("r2")}} at +150 ms; the broker comes back at +250
ms; main calls {{stopRoute("r1")}} at +300 ms. r1's attempt fires at +400 ms
and waits for the lock behind {{stopRoute("r1")}}:
{noformat}
[natural] stopRoute(r1) returned at +765ms: r1=Stopped restartAttempts=1
[natural] 1.5s later: r1=Started restartAttempts=1 consumerStarts=1
{noformat}
(3 of 3 runs.)
{{stop}} (forced): the timer thread is held at {{RouteRestartingEvent}}, which
it fires right before {{doStartRoute}}; main calls {{stopRoute("r1")}}, then
releases it:
{noformat}
[stop] stopRoute(r1) returned: status=Stopped restarting=0
[stop] 1.5s later: status=Started consumerStarts=1 restartAttempts=1
{noformat}
TLA+: {{NoRestartAfterStop}} is violated by TRun(1) -> TCheck(1) ->
UAcquire("stop") -> URelease -> UAct -> TStart(1, TRUE). With the fix below
({{fix_user_ops2}}, {{fix_user_ops3}}: up to 3 user operations and 3 tasks),
{{NoRestartAfterStop}}, {{NoOrphanTask}}, {{AttemptsBounded}}, {{NoDeadlock}}
and the liveness property {{UserTerminates}} hold.
*Proposed fix:* in the attempt, inside the controller lock (a {{doStartRoute}}
variant for the supervisor), give up without starting when {{routes.get(r) !=
task}}, when {{task.getStatus() != Active}}, or when
{{!getCamelContext().isRunAllowed()}}, and return {{false}} so the task
completes. Together with the callback change of CAMEL-25007 ({{routes.remove(r,
task)}}), a cancelled task can no longer start the route.
_Filed with Claude Code on behalf of allthingssecurity._
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
