Federico Mariani created CAMEL-27009:
----------------------------------------
Summary: camel-core - the shutdown timeout does not bound
CamelContext.stop() or a route stop when a consumer's suspend or stop blocks:
the forced shutdown waits for the service lock held by the graceful shutdown
task
Key: CAMEL-27009
URL: https://issues.apache.org/jira/browse/CAMEL-27009
Project: Camel
Issue Type: Bug
Components: camel-core
Reporter: Federico Mariani
Attachments: DefaultShutdownStrategyBlockedConsumerTest.java
The shutdown timeout is meant to bound the graceful shutdown: when it elapses,
the remaining route consumers are shut down "now" ({{DefaultShutdownStrategy}}
javadoc: "This ensures that when shutting down Camel it at some point
eventually will shutdown"). It does not when the suspend or stop of a consumer
blocks and does not react to the interrupt, such as a client call waiting for a
reply on a connection that stopped answering. {{CamelContext.stop()}}, or a
route stop with a timeout, then waits until that call returns, however long it
takes.
Found with fault-injection tests (a proxy that silently drops the traffic of an
open connection, then {{CamelContext.stop()}} with a 5 s shutdown timeout):
* camel-spring-rabbitmq: the graceful suspend runs
{{SpringRabbitMQConsumer.doSuspend()}} -> {{listenerContainer.stop()}} ->
{{ChannelN.basicCancel}}, which waits for basic.cancel-ok up to
{{channelRpcTimeout}} (amqp-client default 10 minutes); {{CamelContext.stop()}}
waits until the cancel returns.
* camel-jms with Artemis: the suspend/stop of the
{{DefaultMessageListenerContainer}} ({{stopSharedConnection}},
{{clearResources}}) waits for the client's {{callTimeout}} (30 s).
* camel-ftp: {{RemoteFileConsumer.doStop()}} waited {{soTimeout}} for the reply
to QUIT (fixed in the component by CAMEL-26999).
h3. Mechanism (main, 4.24.0-SNAPSHOT)
All paths are in
{{core/camel-base-engine/src/main/java/org/apache/camel/impl/engine/}}.
# {{DefaultShutdownStrategy.doShutdown()}} runs the graceful shutdown in a
{{ShutdownTask}} on its own thread: the task suspends a {{Suspendable}}
consumer ({{suspendNow()}}, line 659) or stops the others ({{shutdownNow()}},
line 668), and stops the suspended ones at the end ({{shutdownNow()}}, line
820). {{BaseService.suspend()}} / {{stop()}}
({{core/camel-api/.../support/service/BaseService.java}}, lines 195 / 159) hold
the service lock of the consumer while {{doSuspend()}} / {{doStop()}} run.
# When the timeout elapses, the calling thread cancels the task
({{future.cancel(true)}}, line 244): the interrupt is ignored by the blocked
call.
# The forced shutdown then stops the same consumer on the calling thread:
{{shutdownRoutesNow()}} -> {{shutdownNow()}} ->
{{ServiceHelper.stopService(consumer)}} (line 457) -> {{BaseService.stop()}} ->
{{lock.lock()}} (line 159), held by the task thread.
# Even if the forced path skipped that consumer, the next step waits for the
same lock: {{AbstractCamelContext.doStop()}} stops the route services (line
3463) -> {{RouteService.doStop()}} -> {{stopChildServices()}} (line 296) ->
{{ServiceHelper.stopAndShutdownService(consumer)}} (line 447) ->
{{BaseService.stop()}} line 159. Checked on main: with a shutdown strategy
whose {{shutdownNow(String, Consumer)}} skips a consumer that is still
suspending/stopping, all five tests below still fail, now waiting in
{{RouteService.stopChildServices}} (lines 447 / 449).
The same holds for a route stop with a timeout ({{RouteController.stopRoute(id,
timeout, unit)}}): with {{shutdownNowOnTimeout=true}} (default) it waits in the
forced path, with {{shutdownNowOnTimeout=false}} in
{{RouteService.stopChildServices}} ({{AbstractCamelContext.doShutdownRoute()}},
line 1503), with {{abortAfterTimeout=true}} in {{BaseService.start()}} when the
aborted route is started again (line 1506; seen with the same consumer on
4.23.0).
Camel Main (and the Camel CLI) are bounded anyway: the JVM shutdown hook of
{{DefaultMainShutdownStrategy}} waits at most {{shutdownTimeout +
extraShutdownTimeout}} and the JVM exits. The wait is unbounded for anything
that calls {{CamelContext.stop()}} or stops a route itself (an embedding
application, a runtime that stops the CamelContext in its own lifecycle, tests,
a route stop through the route controller or JMX).
h3. Steps to reproduce
Unit test {{DefaultShutdownStrategyBlockedConsumerTest}} in camel-core
({{org.apache.camel.impl.engine}}, attached): a consumer whose {{doSuspend()}}
or {{doStop()}} waits on a latch and ignores the interrupt, shutdown timeout 1
s, the stop called from a thread, asserting it returns within 5 s; the latch is
released afterwards.
h3. Actual (main at 21b8ed8e9613)
||Test||Consumer||Blocks in||Stop||Result||
|testContextStopWhenSuspendBlocks|Suspendable|{{doSuspend()}} (graceful
suspend)|{{CamelContext.stop()}}|not returned after 5 s|
|testContextStopWhenStopOfSuspendableConsumerBlocks|Suspendable|{{doStop()}}
(deferred stop by the task)|{{CamelContext.stop()}}|not returned after 5 s|
|testContextStopWhenStopBlocks|not
Suspendable|{{doStop()}}|{{CamelContext.stop()}}|not returned after 5 s|
|testRouteStopWhenSuspendBlocks|Suspendable|{{doSuspend()}}|{{stopRoute(id, 1,
SECONDS)}}|not returned after 5 s|
|testRouteStopWithoutShutdownNowOnTimeoutWhenSuspendBlocks|Suspendable|{{doSuspend()}}|{{stopRoute(id,
1, SECONDS)}}, {{shutdownNowOnTimeout=false}}|not returned after 5 s|
The first four wait here (the stop thread is WAITING on the {{ReentrantLock}}
owned by the {{ShutdownTask}} thread):
{noformat}
at org.apache.camel.support.service.BaseService.stop(BaseService.java:159)
at
org.apache.camel.support.service.ServiceHelper.stopService(ServiceHelper.java:203)
at
org.apache.camel.impl.engine.DefaultShutdownStrategy.shutdownNow(DefaultShutdownStrategy.java:457)
at
org.apache.camel.impl.engine.DefaultShutdownStrategy.shutdownRoutesNow(DefaultShutdownStrategy.java:430)
at
org.apache.camel.impl.engine.DefaultShutdownStrategy.doShutdown(DefaultShutdownStrategy.java:271)
at
org.apache.camel.impl.engine.DefaultShutdownStrategy.shutdownForced(DefaultShutdownStrategy.java:154)
{noformat}
the last one here:
{noformat}
at org.apache.camel.support.service.BaseService.stop(BaseService.java:159)
at
org.apache.camel.support.service.ServiceHelper.stopService(ServiceHelper.java:203)
at
org.apache.camel.impl.engine.RouteService.stopChildServices(RouteService.java:449)
at org.apache.camel.impl.engine.RouteService.doStop(RouteService.java:296)
at
org.apache.camel.support.ChildServiceSupport.stop(ChildServiceSupport.java:90)
at
org.apache.camel.impl.engine.AbstractCamelContext.stopRouteService(AbstractCamelContext.java:3907)
{noformat}
Each stop returns as soon as the latch is released.
h3. Expected
The stop returns shortly after the shutdown timeout, with a WARN that names the
consumer it could not stop, and the blocked call is left behind on the (daemon)
ShutdownTask thread, as {{shutdownNow}} already warns that "some resources may
still be running".
h3. Possible fixes
A fix contained in {{DefaultShutdownStrategy}}'s forced path is not enough
(point 4 above), and each of these changes what a stop guarantees, so it needs
a decision first:
# *{{BaseService.stop()}} returns when another thread is already stopping the
service* (check {{status == STOPPING}} before taking the lock, as
{{build()}}/{{init()}} already do). Small, but it changes the semantics of
every service: {{stop()}} would no longer mean "stopped when it returns" under
a concurrent stop. And it does not cover a suspend that blocks (the
spring-rabbitmq and camel-jms cases), where the consumer is {{SUSPENDING}}, not
{{STOPPING}}.
# *The shutdown strategy gives up on the consumer that its task is still
suspending/stopping*: the task records the consumer it is working on; on
timeout the forced path skips it with a WARN, the {{RouteService}} stop skips
stopping/shutting down that consumer while its suspend/stop still runs, and the
task thread stops it once the call returns. Contained to camel-base-engine (two
classes), but {{CamelContext.stop()}} / {{stopRoute}} then return while a
consumer is not stopped yet, and restarting that route waits for it (as the
stop does today). Needs an upgrade-guide note.
# *Bound every forced consumer stop*: run the forced stops on a helper thread
and wait at most a bound (the shutdown timeout, or a new option), plus the
{{RouteService}} part of option 2. Also covers a consumer whose stop blocks on
the calling thread itself (the graceful shutdown timed out because of inflight
exchanges, and the forced {{doStop()}} then blocks, as in CAMEL-7623), which no
lock change covers. Largest change.
# *Components only*: bound the stop/suspend of each component (CAMEL-26999 and
CAMEL-27001 did so for camel-ftp and camel-sftp). Leaves the core gap for the
next component.
h3. Related
* CAMEL-7623 (Not A Problem) and CAMEL-13310 (Invalid): a forced shutdown that
blocks in a component's stop; the answer then was that forced shutdown is best
effort and the component must release its resources.
* CAMEL-26555, CAMEL-26449: the forced path and the cancelled task (the task no
longer stops deferred consumers after a timeout).
* CAMEL-25036: the same pattern (service lifecycle lock held while blocked) in
{{EventDrivenPollingConsumer}}.
* CAMEL-26999, CAMEL-27001: camel-ftp / camel-sftp stops bounded in the
component.
_Claude Code on behalf of Croway_
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)