Federico Mariani created CAMEL-27009:
----------------------------------------

             Summary: camel-core - the shutdown timeout does not bound 
CamelContext.stop() or a route stop when a consumer's suspend or stop blocks: 
the forced shutdown waits for the service lock held by the graceful shutdown 
task
                 Key: CAMEL-27009
                 URL: https://issues.apache.org/jira/browse/CAMEL-27009
             Project: Camel
          Issue Type: Bug
          Components: camel-core
            Reporter: Federico Mariani
         Attachments: DefaultShutdownStrategyBlockedConsumerTest.java

The shutdown timeout is meant to bound the graceful shutdown: when it elapses, 
the remaining route consumers are shut down "now" ({{DefaultShutdownStrategy}} 
javadoc: "This ensures that when shutting down Camel it at some point 
eventually will shutdown"). It does not when the suspend or stop of a consumer 
blocks and does not react to the interrupt, such as a client call waiting for a 
reply on a connection that stopped answering. {{CamelContext.stop()}}, or a 
route stop with a timeout, then waits until that call returns, however long it 
takes.

Found with fault-injection tests (a proxy that silently drops the traffic of an 
open connection, then {{CamelContext.stop()}} with a 5 s shutdown timeout):
* camel-spring-rabbitmq: the graceful suspend runs 
{{SpringRabbitMQConsumer.doSuspend()}} -> {{listenerContainer.stop()}} -> 
{{ChannelN.basicCancel}}, which waits for basic.cancel-ok up to 
{{channelRpcTimeout}} (amqp-client default 10 minutes); {{CamelContext.stop()}} 
waits until the cancel returns.
* camel-jms with Artemis: the suspend/stop of the 
{{DefaultMessageListenerContainer}} ({{stopSharedConnection}}, 
{{clearResources}}) waits for the client's {{callTimeout}} (30 s).
* camel-ftp: {{RemoteFileConsumer.doStop()}} waited {{soTimeout}} for the reply 
to QUIT (fixed in the component by CAMEL-26999).

h3. Mechanism (main, 4.24.0-SNAPSHOT)

All paths are in 
{{core/camel-base-engine/src/main/java/org/apache/camel/impl/engine/}}.

# {{DefaultShutdownStrategy.doShutdown()}} runs the graceful shutdown in a 
{{ShutdownTask}} on its own thread: the task suspends a {{Suspendable}} 
consumer ({{suspendNow()}}, line 659) or stops the others ({{shutdownNow()}}, 
line 668), and stops the suspended ones at the end ({{shutdownNow()}}, line 
820). {{BaseService.suspend()}} / {{stop()}} 
({{core/camel-api/.../support/service/BaseService.java}}, lines 195 / 159) hold 
the service lock of the consumer while {{doSuspend()}} / {{doStop()}} run.
# When the timeout elapses, the calling thread cancels the task 
({{future.cancel(true)}}, line 244): the interrupt is ignored by the blocked 
call.
# The forced shutdown then stops the same consumer on the calling thread: 
{{shutdownRoutesNow()}} -> {{shutdownNow()}} -> 
{{ServiceHelper.stopService(consumer)}} (line 457) -> {{BaseService.stop()}} -> 
{{lock.lock()}} (line 159), held by the task thread.
# Even if the forced path skipped that consumer, the next step waits for the 
same lock: {{AbstractCamelContext.doStop()}} stops the route services (line 
3463) -> {{RouteService.doStop()}} -> {{stopChildServices()}} (line 296) -> 
{{ServiceHelper.stopAndShutdownService(consumer)}} (line 447) -> 
{{BaseService.stop()}} line 159. Checked on main: with a shutdown strategy 
whose {{shutdownNow(String, Consumer)}} skips a consumer that is still 
suspending/stopping, all five tests below still fail, now waiting in 
{{RouteService.stopChildServices}} (lines 447 / 449).

The same holds for a route stop with a timeout ({{RouteController.stopRoute(id, 
timeout, unit)}}): with {{shutdownNowOnTimeout=true}} (default) it waits in the 
forced path, with {{shutdownNowOnTimeout=false}} in 
{{RouteService.stopChildServices}} ({{AbstractCamelContext.doShutdownRoute()}}, 
line 1503), with {{abortAfterTimeout=true}} in {{BaseService.start()}} when the 
aborted route is started again (line 1506; seen with the same consumer on 
4.23.0).

Camel Main (and the Camel CLI) are bounded anyway: the JVM shutdown hook of 
{{DefaultMainShutdownStrategy}} waits at most {{shutdownTimeout + 
extraShutdownTimeout}} and the JVM exits. The wait is unbounded for anything 
that calls {{CamelContext.stop()}} or stops a route itself (an embedding 
application, a runtime that stops the CamelContext in its own lifecycle, tests, 
a route stop through the route controller or JMX).

h3. Steps to reproduce

Unit test {{DefaultShutdownStrategyBlockedConsumerTest}} in camel-core 
({{org.apache.camel.impl.engine}}, attached): a consumer whose {{doSuspend()}} 
or {{doStop()}} waits on a latch and ignores the interrupt, shutdown timeout 1 
s, the stop called from a thread, asserting it returns within 5 s; the latch is 
released afterwards.

h3. Actual (main at 21b8ed8e9613)

||Test||Consumer||Blocks in||Stop||Result||
|testContextStopWhenSuspendBlocks|Suspendable|{{doSuspend()}} (graceful 
suspend)|{{CamelContext.stop()}}|not returned after 5 s|
|testContextStopWhenStopOfSuspendableConsumerBlocks|Suspendable|{{doStop()}} 
(deferred stop by the task)|{{CamelContext.stop()}}|not returned after 5 s|
|testContextStopWhenStopBlocks|not 
Suspendable|{{doStop()}}|{{CamelContext.stop()}}|not returned after 5 s|
|testRouteStopWhenSuspendBlocks|Suspendable|{{doSuspend()}}|{{stopRoute(id, 1, 
SECONDS)}}|not returned after 5 s|
|testRouteStopWithoutShutdownNowOnTimeoutWhenSuspendBlocks|Suspendable|{{doSuspend()}}|{{stopRoute(id,
 1, SECONDS)}}, {{shutdownNowOnTimeout=false}}|not returned after 5 s|

The first four wait here (the stop thread is WAITING on the {{ReentrantLock}} 
owned by the {{ShutdownTask}} thread):

{noformat}
at org.apache.camel.support.service.BaseService.stop(BaseService.java:159)
at 
org.apache.camel.support.service.ServiceHelper.stopService(ServiceHelper.java:203)
at 
org.apache.camel.impl.engine.DefaultShutdownStrategy.shutdownNow(DefaultShutdownStrategy.java:457)
at 
org.apache.camel.impl.engine.DefaultShutdownStrategy.shutdownRoutesNow(DefaultShutdownStrategy.java:430)
at 
org.apache.camel.impl.engine.DefaultShutdownStrategy.doShutdown(DefaultShutdownStrategy.java:271)
at 
org.apache.camel.impl.engine.DefaultShutdownStrategy.shutdownForced(DefaultShutdownStrategy.java:154)
{noformat}

the last one here:

{noformat}
at org.apache.camel.support.service.BaseService.stop(BaseService.java:159)
at 
org.apache.camel.support.service.ServiceHelper.stopService(ServiceHelper.java:203)
at 
org.apache.camel.impl.engine.RouteService.stopChildServices(RouteService.java:449)
at org.apache.camel.impl.engine.RouteService.doStop(RouteService.java:296)
at 
org.apache.camel.support.ChildServiceSupport.stop(ChildServiceSupport.java:90)
at 
org.apache.camel.impl.engine.AbstractCamelContext.stopRouteService(AbstractCamelContext.java:3907)
{noformat}

Each stop returns as soon as the latch is released.

h3. Expected

The stop returns shortly after the shutdown timeout, with a WARN that names the 
consumer it could not stop, and the blocked call is left behind on the (daemon) 
ShutdownTask thread, as {{shutdownNow}} already warns that "some resources may 
still be running".

h3. Possible fixes

A fix contained in {{DefaultShutdownStrategy}}'s forced path is not enough 
(point 4 above), and each of these changes what a stop guarantees, so it needs 
a decision first:
# *{{BaseService.stop()}} returns when another thread is already stopping the 
service* (check {{status == STOPPING}} before taking the lock, as 
{{build()}}/{{init()}} already do). Small, but it changes the semantics of 
every service: {{stop()}} would no longer mean "stopped when it returns" under 
a concurrent stop. And it does not cover a suspend that blocks (the 
spring-rabbitmq and camel-jms cases), where the consumer is {{SUSPENDING}}, not 
{{STOPPING}}.
# *The shutdown strategy gives up on the consumer that its task is still 
suspending/stopping*: the task records the consumer it is working on; on 
timeout the forced path skips it with a WARN, the {{RouteService}} stop skips 
stopping/shutting down that consumer while its suspend/stop still runs, and the 
task thread stops it once the call returns. Contained to camel-base-engine (two 
classes), but {{CamelContext.stop()}} / {{stopRoute}} then return while a 
consumer is not stopped yet, and restarting that route waits for it (as the 
stop does today). Needs an upgrade-guide note.
# *Bound every forced consumer stop*: run the forced stops on a helper thread 
and wait at most a bound (the shutdown timeout, or a new option), plus the 
{{RouteService}} part of option 2. Also covers a consumer whose stop blocks on 
the calling thread itself (the graceful shutdown timed out because of inflight 
exchanges, and the forced {{doStop()}} then blocks, as in CAMEL-7623), which no 
lock change covers. Largest change.
# *Components only*: bound the stop/suspend of each component (CAMEL-26999 and 
CAMEL-27001 did so for camel-ftp and camel-sftp). Leaves the core gap for the 
next component.

h3. Related

* CAMEL-7623 (Not A Problem) and CAMEL-13310 (Invalid): a forced shutdown that 
blocks in a component's stop; the answer then was that forced shutdown is best 
effort and the component must release its resources.
* CAMEL-26555, CAMEL-26449: the forced path and the cancelled task (the task no 
longer stops deferred consumers after a timeout).
* CAMEL-25036: the same pattern (service lifecycle lock held while blocked) in 
{{EventDrivenPollingConsumer}}.
* CAMEL-26999, CAMEL-27001: camel-ftp / camel-sftp stops bounded in the 
component.

_Claude Code on behalf of Croway_




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

Reply via email to