Federico Mariani created CAMEL-27000:
----------------------------------------
Summary: camel-ftp - after a read timeout the consumer keeps using
the dead connection for the rest of the batch: every remaining file waits
soTimeout and fails
Key: CAMEL-27000
URL: https://issues.apache.org/jira/browse/CAMEL-27000
Project: Camel
Issue Type: Bug
Components: camel-ftp
Affects Versions: 4.14.9, 4.18.5, 4.23.0
Reporter: Federico Mariani
Fix For: 4.24.0
When the FTP server, or the network path to it, stops answering in the middle
of a poll (a hung server, a connection dropped by a firewall without a reset),
the command or download in progress fails after {{soTimeout}} (control
connection) or {{timeout}} (data connection). The consumer then goes on with
the next file of the batch on the same connection:
* {{GenericFileConsumer.processBatch()}} (camel-file, line 237 at
{{camel-4.23.0}}) only ends the batch when the consumer is stopping
({{isBatchAllowed()}}). A failed download does not leave the loop:
{{retrieveFileOrAbort()}} (lines 533-566) aborts the file, reports it to the
exception handler (line 559), and the loop goes on.
* commons-net (3.13.0) does not close the connection after a
{{SocketTimeoutException}}, and after a failed transfer the reply to RETR is
still unread, so the control connection cannot be used any more (a late reply
would be read as the reply to the next command). Each next file starts with a
command on it: with the default {{stepwise=true}}, the PWD of
{{FtpOperations.retrieveFileToStreamInBody()}} (camel-ftp, line 483), which
waits {{soTimeout}} for a reply that never comes. The file fails and is
reported (a WARN with a stack trace with the default exception handler), and so
on for every remaining file.
* The poll itself succeeds. The next poll's
{{RemoteFileConsumer.connectIfNecessary()}} (camel-ftp-common, lines 256-277)
sends a NOOP on the same connection (line 260) and waits {{soTimeout}} once
more, then connects again (line 272) without closing the old connection:
commons-net {{SocketClient._connect()}} replaces the socket without closing it,
so the dead control connection stays open until the socket is garbage collected.
So the consumer recovers after (remaining files + 2) x {{soTimeout}} instead of
about one {{soTimeout}}. With the defaults ({{soTimeout}} 300 s), a server that
stops answering at the 4th file of a batch of 20 costs 18 x 5 minutes = 1.5
hours and 17 WARN stack traces; the consumer health check stays UP meanwhile,
as the poll does not fail.
On main, CAMEL-25915 sends a NOOP before the next poll after an I/O failure
when {{sendNoop=false}}; it does not end the batch, and the result is the same.
Found with fault-injection tests (a proxy that stops forwarding on an open
connection); reproduced with a unit test and an in-process fake FTP server that
stops answering on one connection while it serves new ones.
h3. Steps to reproduce
10 files,
{{ftp://...?passiveMode=true&noop=true&sortBy=file:name&delay=100&soTimeout=1000&timeout=1000}};
the server stops answering at {{RETR f-03.txt}}, either before the reply (the
client times out on the control connection) or after it sent half of the file
(the client times out on the data connection).
h3. Expected vs actual (4.23.0 and 4.24.0-SNAPSHOT, soTimeout 1 s)
|| ||Expected||Actual||
|Files that fail|f-03 only|f-03 to f-10 (8 files, 8 WARN)|
|Commands sent on the dead connection after the timeout|none|PWD x 7, NOOP|
|Time from the stall until all 10 files are delivered|about 1 x soTimeout|9.2 s
= (7 + 2) x soTimeout|
|Dead control connection|closed|never closed|
h3. Suggested fix
After a file failed with an I/O error, close the connection without waiting on
it (without the QUIT of CAMEL-26999) and end the batch: {{processBatch()}}
releases the remaining files, as when the consumer stops, and the next poll
connects again and picks them up. Close a connection that is still open before
connecting again.
h3. Related
* CAMEL-26999: disconnecting waits {{soTimeout}} for the reply to QUIT.
* CAMEL-25915, CAMEL-25896: the NOOP before the next poll.
* CAMEL-10440 (Won't Fix): errors during a download do not reach the poll
strategy.
* CAMEL-26657: the PWD/CWD per file with {{stepwise=true}}.
* CAMEL-8718: an earlier connection leak on reconnect (invalid credentials).
_Claude Code on behalf of Croway_
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)