Federico Mariani created CAMEL-26999:
----------------------------------------
Summary: camel-ftp - disconnecting waits up to soTimeout (5
minutes by default) for the reply to QUIT: CamelContext.stop() hangs when the
FTP server stopped answering, and a failed poll waits once more before it
connects again
Key: CAMEL-26999
URL: https://issues.apache.org/jira/browse/CAMEL-26999
Project: Camel
Issue Type: Bug
Components: camel-ftp
Affects Versions: 4.14.9, 4.18.5, 4.23.0
Reporter: Federico Mariani
Fix For: 4.24.0
{{FtpOperations.doDisconnect()}}
({{components/camel-ftp/src/main/java/org/apache/camel/component/file/remote/FtpOperations.java}},
lines 311-333 at {{camel-4.23.0}}, the same on main) calls {{client.logout()}}
(line 316) before {{client.disconnect()}}. In commons-net 3.13.0,
{{FTPClient.logout()}} sends QUIT and reads the reply, so the wait is bounded
only by the SO_TIMEOUT of the control connection: {{soTimeout}}, 300000 ms by
default (set in {{doConnect()}}, line 187). {{forceDisconnect()}} (line 307)
calls the same {{doDisconnect()}}, so the "hard disconnect" of
{{RemoteFileOperations.forceDisconnect()}} waits as well
({{SftpOperations.forceDisconnect()}} closes the session without waiting).
When the server, or the network path to it, has stopped answering (a hung
server, a connection that a firewall dropped without a reset),
{{client.isConnected()}} is still true, and:
* *Stop*: {{RemoteFileConsumer.doStop()}} and {{RemoteFileProducer.doStop()}}
(camel-ftp-common) disconnect, and wait {{soTimeout}} for the reply to QUIT,
also when nothing is in flight. The shutdown timeout does not bound it: the
consumer is stopped in the shutdown task thread; when the shutdown strategy
times out, it stops the consumer again from the calling thread
({{DefaultShutdownStrategy.shutdownRoutesNow()}}), which waits on the lock in
{{BaseService.stop()}} held by the first stop. So {{CamelContext.stop()}} takes
{{soTimeout}} per connected endpoint: 5 minutes with the defaults, more than
the termination grace period of a container.
* *Recovery after a failed poll*: when a poll fails (e.g. the LIST timed out),
{{RemoteFilePollingConsumerPollStrategy.rollback()}} calls
{{forceDisconnect()}} (line 45), which sends QUIT on the dead connection and
waits {{soTimeout}} again before the next poll can connect.
Found with fault-injection tests (a proxy that stops forwarding on an open
connection); reproduced with unit tests and an in-process fake FTP server that
stops answering on its open connections.
h3. Steps to reproduce
* An ftp consumer ({{passiveMode=true&soTimeout=8000}}) has consumed a file and
is idle; the server stops answering on the open connection;
{{CamelContext.stop()}} with a shutdown timeout of 2 s.
* The same with an ftp producer that has uploaded a file (it keeps its
connection, {{disconnect=false}}).
* An ftp consumer ({{soTimeout=2000}}) whose server stops answering at the LIST
of the first poll: the time until it connects again.
h3. Actual (4.23.0 and 4.24.0-SNAPSHOT)
||Case||Actual||
|idle consumer, soTimeout 8 s|{{stop()}} takes 8 s; the only command sent is
QUIT; stack: {{FTPClient.logout}} <- {{FtpOperations.doDisconnect}} <-
{{RemoteFileConsumer.doStop}} (shutdown task thread), the caller waits in
{{BaseService.stop}}|
|idle producer, soTimeout 8 s|{{stop()}} takes 8 s; stack: {{FTPClient.logout}}
<- {{RemoteFileProducer.doStop}}|
|failed poll, soTimeout 2 s|connects again after 4.1 s (the LIST timeout, then
the QUIT timeout)|
h3. Expected
{{stop()}} returns within a few seconds; after a failed poll the consumer
connects again after about one {{soTimeout}}.
h3. Suggested fix
* {{forceDisconnect()}}: close the connection without sending QUIT. It is used
to recover from a failure, when the connection may no longer answer.
* {{disconnect()}}: keep sending QUIT, but wait for its reply only a short time
(e.g. at most 2 s), as the connection is closed anyway.
h3. Related
* CAMEL-12061: added {{forceDisconnect()}}, for the rollback of a failed poll.
* CAMEL-8088, CAMEL-12830: earlier unbounded waits on a silent control
connection (during connect).
* CAMEL-6902 (Won't Fix): stop a download/upload in progress when shutting
down; here nothing is in flight.
_Claude Code on behalf of Croway_
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)