[
https://issues.apache.org/jira/browse/CAMEL-27000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on CAMEL-27000 started by Federico Mariani.
------------------------------------------------
> camel-ftp - after a read timeout the consumer keeps using the dead connection
> for the rest of the batch: every remaining file waits soTimeout and fails
> -------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: CAMEL-27000
> URL: https://issues.apache.org/jira/browse/CAMEL-27000
> Project: Camel
> Issue Type: Bug
> Components: camel-ftp
> Affects Versions: 4.14.9, 4.18.5, 4.23.0
> Reporter: Federico Mariani
> Assignee: Federico Mariani
> Priority: Major
> Fix For: 4.24.0
>
>
> When the FTP server, or the network path to it, stops answering in the middle
> of a poll (a hung server, a connection dropped by a firewall without a
> reset), the command or download in progress fails after {{soTimeout}}
> (control connection) or {{timeout}} (data connection). The consumer then goes
> on with the next file of the batch on the same connection:
> * {{GenericFileConsumer.processBatch()}} (camel-file, line 237 at
> {{camel-4.23.0}}) only ends the batch when the consumer is stopping
> ({{isBatchAllowed()}}). A failed download does not leave the loop:
> {{retrieveFileOrAbort()}} (lines 533-566) aborts the file, reports it to the
> exception handler (line 559), and the loop goes on.
> * commons-net (3.13.0) does not close the connection after a
> {{SocketTimeoutException}}, and after a failed transfer the reply to RETR is
> still unread, so the control connection cannot be used any more (a late reply
> would be read as the reply to the next command). Each next file starts with a
> command on it: with the default {{stepwise=true}}, the PWD of
> {{FtpOperations.retrieveFileToStreamInBody()}} (camel-ftp, line 483), which
> waits {{soTimeout}} for a reply that never comes. The file fails and is
> reported (a WARN with a stack trace with the default exception handler), and
> so on for every remaining file.
> * The poll itself succeeds. The next poll's
> {{RemoteFileConsumer.connectIfNecessary()}} (camel-ftp-common, lines 256-277)
> sends a NOOP on the same connection (line 260) and waits {{soTimeout}} once
> more, then connects again (line 272) without closing the old connection:
> commons-net {{SocketClient._connect()}} replaces the socket without closing
> it, so the dead control connection stays open until the socket is garbage
> collected.
> So the consumer recovers after (remaining files + 2) x {{soTimeout}} instead
> of about one {{soTimeout}}. With the defaults ({{soTimeout}} 300 s), a server
> that stops answering at the 4th file of a batch of 20 costs 18 x 5 minutes =
> 1.5 hours and 17 WARN stack traces; the consumer health check stays UP
> meanwhile, as the poll does not fail.
> On main, CAMEL-25915 sends a NOOP before the next poll after an I/O failure
> when {{sendNoop=false}}; it does not end the batch, and the result is the
> same.
> Found with fault-injection tests (a proxy that stops forwarding on an open
> connection); reproduced with a unit test and an in-process fake FTP server
> that stops answering on one connection while it serves new ones.
> h3. Steps to reproduce
> 10 files,
> {{ftp://...?passiveMode=true&noop=true&sortBy=file:name&delay=100&soTimeout=1000&timeout=1000}};
> the server stops answering at {{RETR f-03.txt}}, either before the reply
> (the client times out on the control connection) or after it sent half of the
> file (the client times out on the data connection).
> h3. Expected vs actual (4.23.0 and 4.24.0-SNAPSHOT, soTimeout 1 s)
> || ||Expected||Actual||
> |Files that fail|f-03 only|f-03 to f-10 (8 files, 8 WARN)|
> |Commands sent on the dead connection after the timeout|none|PWD x 7, NOOP|
> |Time from the stall until all 10 files are delivered|about 1 x soTimeout|9.2
> s = (7 + 2) x soTimeout|
> |Dead control connection|closed|never closed|
> h3. Suggested fix
> After a file failed with an I/O error, close the connection without waiting
> on it (without the QUIT of CAMEL-26999) and end the batch: {{processBatch()}}
> releases the remaining files, as when the consumer stops, and the next poll
> connects again and picks them up. Close a connection that is still open
> before connecting again.
> h3. Related
> * CAMEL-26999: disconnecting waits {{soTimeout}} for the reply to QUIT.
> * CAMEL-25915, CAMEL-25896: the NOOP before the next poll.
> * CAMEL-10440 (Won't Fix): errors during a download do not reach the poll
> strategy.
> * CAMEL-26657: the PWD/CWD per file with {{stepwise=true}}.
> * CAMEL-8718: an earlier connection leak on reconnect (invalid credentials).
> _Claude Code on behalf of Croway_
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)