Federico Mariani created CAMEL-27001:
----------------------------------------
Summary: camel-sftp - CamelContext.stop() waits until a hung
upload ends: the producer stop waits for the SftpOperations lock that the
upload holds, also after the shutdown timeout
Key: CAMEL-27001
URL: https://issues.apache.org/jira/browse/CAMEL-27001
Project: Camel
Issue Type: Bug
Components: camel-ftp
Affects Versions: 4.14.9, 4.18.5, 4.23.0
Reporter: Federico Mariani
Fix For: 4.24.0
{{SftpOperations}} runs every operation under one {{ReentrantLock}} ({{lock}},
line 96 of
{{components/camel-ftp/src/main/java/org/apache/camel/component/file/remote/SftpOperations.java}}
on main; {{synchronized}} methods from CAMEL-8887 until CAMEL-20199).
{{storeFile()}} (line 1275) holds it for the whole upload, including
{{ChannelSftp.put()}} (lines 1389, 1393). When the server stops answering
during the upload, {{put()}} waits for the status replies of the server: over a
dead (half-open) network path until JSch drops the session, which with the
defaults is about twice {{soTimeout}} (a keepalive after 300 s of silence, then
one more timeout: about 10 minutes); for good when the SSH transport is alive
but the SFTP subsystem of the server hangs.
When Camel stops, the shutdown strategy waits for the in-flight exchange up to
its timeout, logs "Timeout occurred during graceful shutdown. Forcing the
routes to be shutdown now", and stops the routes. Stopping the route stops the
producer: {{RemoteFileProducer.doStop()}}
({{components/camel-ftp-common/.../RemoteFileProducer.java}}, line 192) calls
{{disconnect()}} (line 102), whose first call, {{SftpOperations.isConnected()}}
(line 614), waits for the lock without a timeout;
{{SftpOperations.disconnect()}} and {{forceDisconnect()}} (lines 624, 639) take
the same lock. Nothing interrupts the thread that runs the upload: it is the
thread of the caller (a {{ProducerTemplate}} caller, an HTTP or JMS listener
thread, ...). So {{CamelContext.stop()}} does not return before the upload
ends, whatever the shutdown timeout.
The sftp consumer is not affected the same way: its poll thread holds the same
lock during a download, but the forced shutdown interrupts it, JSch's read of
the channel pipe is interruptible, the download fails and releases the lock.
Found with fault-injection tests (a half-open connection to the SFTP server
during an upload).
h3. Steps to reproduce
An embedded Apache SSHD server whose SFTP subsystem blocks on the first write
request ({{SftpEventListener.writing()}}) while the SSH transport stays alive.
Route {{from("direct:upload").to("sftp://...")}}, shutdown timeout of a few
seconds. Send a 64 KB body from another thread; once the upload reached the
server, call {{CamelContext.stop()}}.
h3. Actual (4.23.0 and 4.24.0-SNAPSHOT)
{{stop()}} is still blocked long after the shutdown timeout, and returns only
when the server answers again. Stack at 4.23.0:
{noformat}
"ShutdownTask" WAITING
at
org.apache.camel.component.file.remote.SftpOperations.isConnected(SftpOperations.java:610)
at
org.apache.camel.component.file.remote.RemoteFileProducer.disconnect(RemoteFileProducer.java:104)
at
org.apache.camel.component.file.remote.RemoteFileProducer.doStop(RemoteFileProducer.java:198)
... SendProcessor.doStop <- DirectConsumer.doStop
"camel-stop" (the caller of CamelContext.stop()) WAITING
at org.apache.camel.support.service.BaseService.stop(BaseService.java:159)
at
org.apache.camel.impl.engine.DefaultShutdownStrategy.shutdownNow(DefaultShutdownStrategy.java:434)
at
org.apache.camel.impl.engine.DefaultShutdownStrategy.shutdownRoutesNow(DefaultShutdownStrategy.java:407)
{noformat}
The same with an sftp consumer whose download hangs: {{stop()}} returns right
after the shutdown timeout.
h3. Expected
After the shutdown timeout the shutdown is forced and {{stop()}} returns; the
hung upload fails (it does not report success).
h3. Suggested fix
When the producer stops, do not wait for the lock without bound: if an
operation of another thread still holds it after a short wait, disconnect the
JSch session without the lock. {{Session.disconnect()}} can be called from
another thread (JSch's own session thread does it): it closes the channel and
its pipe, so the blocked {{put()}} fails and releases the lock. This is what a
forced shutdown implies; it does not need the transfers to become abortable
(CAMEL-6902).
h3. Related
* CAMEL-6902 (Won't Fix), CAMEL-6582: abort a download or upload in progress
when stopping.
* CAMEL-8887 (2.16): made {{SftpOperations}} synchronized, as JSch's
{{ChannelSftp}} is not thread-safe; CAMEL-20199 (4.8): {{synchronized}}
replaced with the {{ReentrantLock}}.
* CAMEL-25036: the same pattern in {{EventDrivenPollingConsumer}}, a lock held
while waiting made the stop hang.
_Claude Code on behalf of Croway_
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)