Federico Mariani created CAMEL-27001:
----------------------------------------

             Summary: camel-sftp - CamelContext.stop() waits until a hung 
upload ends: the producer stop waits for the SftpOperations lock that the 
upload holds, also after the shutdown timeout
                 Key: CAMEL-27001
                 URL: https://issues.apache.org/jira/browse/CAMEL-27001
             Project: Camel
          Issue Type: Bug
          Components: camel-ftp
    Affects Versions: 4.14.9, 4.18.5, 4.23.0
            Reporter: Federico Mariani
             Fix For: 4.24.0


{{SftpOperations}} runs every operation under one {{ReentrantLock}} ({{lock}}, 
line 96 of 
{{components/camel-ftp/src/main/java/org/apache/camel/component/file/remote/SftpOperations.java}}
 on main; {{synchronized}} methods from CAMEL-8887 until CAMEL-20199). 
{{storeFile()}} (line 1275) holds it for the whole upload, including 
{{ChannelSftp.put()}} (lines 1389, 1393). When the server stops answering 
during the upload, {{put()}} waits for the status replies of the server: over a 
dead (half-open) network path until JSch drops the session, which with the 
defaults is about twice {{soTimeout}} (a keepalive after 300 s of silence, then 
one more timeout: about 10 minutes); for good when the SSH transport is alive 
but the SFTP subsystem of the server hangs.

When Camel stops, the shutdown strategy waits for the in-flight exchange up to 
its timeout, logs "Timeout occurred during graceful shutdown. Forcing the 
routes to be shutdown now", and stops the routes. Stopping the route stops the 
producer: {{RemoteFileProducer.doStop()}} 
({{components/camel-ftp-common/.../RemoteFileProducer.java}}, line 192) calls 
{{disconnect()}} (line 102), whose first call, {{SftpOperations.isConnected()}} 
(line 614), waits for the lock without a timeout; 
{{SftpOperations.disconnect()}} and {{forceDisconnect()}} (lines 624, 639) take 
the same lock. Nothing interrupts the thread that runs the upload: it is the 
thread of the caller (a {{ProducerTemplate}} caller, an HTTP or JMS listener 
thread, ...). So {{CamelContext.stop()}} does not return before the upload 
ends, whatever the shutdown timeout.

The sftp consumer is not affected the same way: its poll thread holds the same 
lock during a download, but the forced shutdown interrupts it, JSch's read of 
the channel pipe is interruptible, the download fails and releases the lock.

Found with fault-injection tests (a half-open connection to the SFTP server 
during an upload).

h3. Steps to reproduce

An embedded Apache SSHD server whose SFTP subsystem blocks on the first write 
request ({{SftpEventListener.writing()}}) while the SSH transport stays alive. 
Route {{from("direct:upload").to("sftp://...";)}}, shutdown timeout of a few 
seconds. Send a 64 KB body from another thread; once the upload reached the 
server, call {{CamelContext.stop()}}.

h3. Actual (4.23.0 and 4.24.0-SNAPSHOT)

{{stop()}} is still blocked long after the shutdown timeout, and returns only 
when the server answers again. Stack at 4.23.0:
{noformat}
"ShutdownTask" WAITING
    at 
org.apache.camel.component.file.remote.SftpOperations.isConnected(SftpOperations.java:610)
    at 
org.apache.camel.component.file.remote.RemoteFileProducer.disconnect(RemoteFileProducer.java:104)
    at 
org.apache.camel.component.file.remote.RemoteFileProducer.doStop(RemoteFileProducer.java:198)
    ... SendProcessor.doStop <- DirectConsumer.doStop
"camel-stop" (the caller of CamelContext.stop()) WAITING
    at org.apache.camel.support.service.BaseService.stop(BaseService.java:159)
    at 
org.apache.camel.impl.engine.DefaultShutdownStrategy.shutdownNow(DefaultShutdownStrategy.java:434)
    at 
org.apache.camel.impl.engine.DefaultShutdownStrategy.shutdownRoutesNow(DefaultShutdownStrategy.java:407)
{noformat}
The same with an sftp consumer whose download hangs: {{stop()}} returns right 
after the shutdown timeout.

h3. Expected

After the shutdown timeout the shutdown is forced and {{stop()}} returns; the 
hung upload fails (it does not report success).

h3. Suggested fix

When the producer stops, do not wait for the lock without bound: if an 
operation of another thread still holds it after a short wait, disconnect the 
JSch session without the lock. {{Session.disconnect()}} can be called from 
another thread (JSch's own session thread does it): it closes the channel and 
its pipe, so the blocked {{put()}} fails and releases the lock. This is what a 
forced shutdown implies; it does not need the transfers to become abortable 
(CAMEL-6902).

h3. Related

* CAMEL-6902 (Won't Fix), CAMEL-6582: abort a download or upload in progress 
when stopping.
* CAMEL-8887 (2.16): made {{SftpOperations}} synchronized, as JSch's 
{{ChannelSftp}} is not thread-safe; CAMEL-20199 (4.8): {{synchronized}} 
replaced with the {{ReentrantLock}}.
* CAMEL-25036: the same pattern in {{EventDrivenPollingConsumer}}, a lock held 
while waiting made the stop hang.

_Claude Code on behalf of Croway_




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

Reply via email to