[ 
https://issues.apache.org/jira/browse/CAMEL-27000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Federico Mariani updated CAMEL-27000:
-------------------------------------
    Description: 
When the FTP server, or the network path to it, stops answering in the middle 
of a poll (a hung server, a connection dropped by a firewall without a reset), 
the command or download in progress fails after {{soTimeout}} (control 
connection) or {{timeout}} (data connection). The consumer then goes on with 
the next file of the batch on the same connection:
* {{GenericFileConsumer.processBatch()}} (camel-file, line 237 at 
{{camel-4.23.0}}) only ends the batch when the consumer is stopping 
({{isBatchAllowed()}}). A failed download does not leave the loop: 
{{retrieveFileOrAbort()}} (lines 533-566) aborts the file, reports it to the 
exception handler (line 559), and the loop goes on.
* commons-net (3.13.0) does not close the connection after a 
{{SocketTimeoutException}}, and after a failed transfer the reply to RETR is 
still unread, so the control connection cannot be used any more (a late reply 
would be read as the reply to the next command). Each next file starts with a 
command on it: with the default {{stepwise=true}}, the PWD of 
{{FtpOperations.retrieveFileToStreamInBody()}} (camel-ftp, line 483), which 
waits {{soTimeout}} for a reply that never comes. The file fails and is 
reported (a WARN with a stack trace with the default exception handler), and so 
on for every remaining file.
* The poll itself succeeds. The next poll's 
{{RemoteFileConsumer.connectIfNecessary()}} (camel-ftp-common, lines 256-277) 
sends a NOOP on the same connection (line 260) and waits {{soTimeout}} once 
more, then connects again (line 272) without closing the old connection: 
commons-net {{SocketClient._connect()}} replaces the socket without closing it, 
so the dead control connection stays open until the socket is garbage collected.

So the consumer recovers after (remaining files + 2) x {{soTimeout}} instead of 
about one {{soTimeout}}. With the defaults ({{soTimeout}} 300 s), a server that 
stops answering at the 4th file of a batch of 20 costs 18 x 5 minutes = 1.5 
hours and 17 WARN stack traces; the consumer health check stays UP meanwhile, 
as the poll does not fail.

On main, CAMEL-25915 sends a NOOP before the next poll after an I/O failure 
when {{sendNoop=false}}; it does not end the batch, and the result is the same.

Found with fault-injection tests (a proxy that stops forwarding on an open 
connection); reproduced with a unit test and an in-process fake FTP server that 
stops answering on one connection while it serves new ones.

h3. Steps to reproduce

10 files, 
{{ftp://...?passiveMode=true&noop=true&sortBy=file:name&delay=100&soTimeout=1000&timeout=1000}};
 the server stops answering at {{RETR f-03.txt}}, either before the reply (the 
client times out on the control connection) or after it sent half of the file 
(the client times out on the data connection).

h3. Expected vs actual (4.23.0 and 4.24.0-SNAPSHOT, soTimeout 1 s)

|| ||Expected||Actual||
|Files that fail|f-03 only|f-03 to f-10 (8 files, 8 WARN)|
|Commands sent on the dead connection after the timeout|none|PWD x 7, NOOP|
|Time from the stall until all 10 files are delivered|about 1 x soTimeout|9.2 s 
= (7 + 2) x soTimeout|
|Dead control connection|closed|never closed|

h3. Suggested fix

After a file failed because the server did not answer in time (a 
{{SocketTimeoutException}}: {{soTimeout}} on the control connection, 
{{timeout}} on the data connection), close the connection without waiting on it 
(without the QUIT of CAMEL-26999) and end the batch: {{processBatch()}} 
releases the remaining files, as when the consumer stops, and the next poll 
connects again and picks them up. Close a connection that is still open before 
connecting again.

h3. Related

* CAMEL-26999: disconnecting waits {{soTimeout}} for the reply to QUIT.
* CAMEL-25915, CAMEL-25896: the NOOP before the next poll.
* CAMEL-10440 (Won't Fix): errors during a download do not reach the poll 
strategy.
* CAMEL-26657: the PWD/CWD per file with {{stepwise=true}}.
* CAMEL-8718: an earlier connection leak on reconnect (invalid credentials).

_Claude Code on behalf of Croway_

  was:
When the FTP server, or the network path to it, stops answering in the middle 
of a poll (a hung server, a connection dropped by a firewall without a reset), 
the command or download in progress fails after {{soTimeout}} (control 
connection) or {{timeout}} (data connection). The consumer then goes on with 
the next file of the batch on the same connection:
* {{GenericFileConsumer.processBatch()}} (camel-file, line 237 at 
{{camel-4.23.0}}) only ends the batch when the consumer is stopping 
({{isBatchAllowed()}}). A failed download does not leave the loop: 
{{retrieveFileOrAbort()}} (lines 533-566) aborts the file, reports it to the 
exception handler (line 559), and the loop goes on.
* commons-net (3.13.0) does not close the connection after a 
{{SocketTimeoutException}}, and after a failed transfer the reply to RETR is 
still unread, so the control connection cannot be used any more (a late reply 
would be read as the reply to the next command). Each next file starts with a 
command on it: with the default {{stepwise=true}}, the PWD of 
{{FtpOperations.retrieveFileToStreamInBody()}} (camel-ftp, line 483), which 
waits {{soTimeout}} for a reply that never comes. The file fails and is 
reported (a WARN with a stack trace with the default exception handler), and so 
on for every remaining file.
* The poll itself succeeds. The next poll's 
{{RemoteFileConsumer.connectIfNecessary()}} (camel-ftp-common, lines 256-277) 
sends a NOOP on the same connection (line 260) and waits {{soTimeout}} once 
more, then connects again (line 272) without closing the old connection: 
commons-net {{SocketClient._connect()}} replaces the socket without closing it, 
so the dead control connection stays open until the socket is garbage collected.

So the consumer recovers after (remaining files + 2) x {{soTimeout}} instead of 
about one {{soTimeout}}. With the defaults ({{soTimeout}} 300 s), a server that 
stops answering at the 4th file of a batch of 20 costs 18 x 5 minutes = 1.5 
hours and 17 WARN stack traces; the consumer health check stays UP meanwhile, 
as the poll does not fail.

On main, CAMEL-25915 sends a NOOP before the next poll after an I/O failure 
when {{sendNoop=false}}; it does not end the batch, and the result is the same.

Found with fault-injection tests (a proxy that stops forwarding on an open 
connection); reproduced with a unit test and an in-process fake FTP server that 
stops answering on one connection while it serves new ones.

h3. Steps to reproduce

10 files, 
{{ftp://...?passiveMode=true&noop=true&sortBy=file:name&delay=100&soTimeout=1000&timeout=1000}};
 the server stops answering at {{RETR f-03.txt}}, either before the reply (the 
client times out on the control connection) or after it sent half of the file 
(the client times out on the data connection).

h3. Expected vs actual (4.23.0 and 4.24.0-SNAPSHOT, soTimeout 1 s)

|| ||Expected||Actual||
|Files that fail|f-03 only|f-03 to f-10 (8 files, 8 WARN)|
|Commands sent on the dead connection after the timeout|none|PWD x 7, NOOP|
|Time from the stall until all 10 files are delivered|about 1 x soTimeout|9.2 s 
= (7 + 2) x soTimeout|
|Dead control connection|closed|never closed|

h3. Suggested fix

After a file failed with an I/O error, close the connection without waiting on 
it (without the QUIT of CAMEL-26999) and end the batch: {{processBatch()}} 
releases the remaining files, as when the consumer stops, and the next poll 
connects again and picks them up. Close a connection that is still open before 
connecting again.

h3. Related

* CAMEL-26999: disconnecting waits {{soTimeout}} for the reply to QUIT.
* CAMEL-25915, CAMEL-25896: the NOOP before the next poll.
* CAMEL-10440 (Won't Fix): errors during a download do not reach the poll 
strategy.
* CAMEL-26657: the PWD/CWD per file with {{stepwise=true}}.
* CAMEL-8718: an earlier connection leak on reconnect (invalid credentials).

_Claude Code on behalf of Croway_


> camel-ftp - after a read timeout the consumer keeps using the dead connection 
> for the rest of the batch: every remaining file waits soTimeout and fails
> -------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: CAMEL-27000
>                 URL: https://issues.apache.org/jira/browse/CAMEL-27000
>             Project: Camel
>          Issue Type: Bug
>          Components: camel-ftp
>    Affects Versions: 4.14.9, 4.18.5, 4.23.0
>            Reporter: Federico Mariani
>            Assignee: Federico Mariani
>            Priority: Major
>             Fix For: 4.24.0
>
>
> When the FTP server, or the network path to it, stops answering in the middle 
> of a poll (a hung server, a connection dropped by a firewall without a 
> reset), the command or download in progress fails after {{soTimeout}} 
> (control connection) or {{timeout}} (data connection). The consumer then goes 
> on with the next file of the batch on the same connection:
> * {{GenericFileConsumer.processBatch()}} (camel-file, line 237 at 
> {{camel-4.23.0}}) only ends the batch when the consumer is stopping 
> ({{isBatchAllowed()}}). A failed download does not leave the loop: 
> {{retrieveFileOrAbort()}} (lines 533-566) aborts the file, reports it to the 
> exception handler (line 559), and the loop goes on.
> * commons-net (3.13.0) does not close the connection after a 
> {{SocketTimeoutException}}, and after a failed transfer the reply to RETR is 
> still unread, so the control connection cannot be used any more (a late reply 
> would be read as the reply to the next command). Each next file starts with a 
> command on it: with the default {{stepwise=true}}, the PWD of 
> {{FtpOperations.retrieveFileToStreamInBody()}} (camel-ftp, line 483), which 
> waits {{soTimeout}} for a reply that never comes. The file fails and is 
> reported (a WARN with a stack trace with the default exception handler), and 
> so on for every remaining file.
> * The poll itself succeeds. The next poll's 
> {{RemoteFileConsumer.connectIfNecessary()}} (camel-ftp-common, lines 256-277) 
> sends a NOOP on the same connection (line 260) and waits {{soTimeout}} once 
> more, then connects again (line 272) without closing the old connection: 
> commons-net {{SocketClient._connect()}} replaces the socket without closing 
> it, so the dead control connection stays open until the socket is garbage 
> collected.
> So the consumer recovers after (remaining files + 2) x {{soTimeout}} instead 
> of about one {{soTimeout}}. With the defaults ({{soTimeout}} 300 s), a server 
> that stops answering at the 4th file of a batch of 20 costs 18 x 5 minutes = 
> 1.5 hours and 17 WARN stack traces; the consumer health check stays UP 
> meanwhile, as the poll does not fail.
> On main, CAMEL-25915 sends a NOOP before the next poll after an I/O failure 
> when {{sendNoop=false}}; it does not end the batch, and the result is the 
> same.
> Found with fault-injection tests (a proxy that stops forwarding on an open 
> connection); reproduced with a unit test and an in-process fake FTP server 
> that stops answering on one connection while it serves new ones.
> h3. Steps to reproduce
> 10 files, 
> {{ftp://...?passiveMode=true&noop=true&sortBy=file:name&delay=100&soTimeout=1000&timeout=1000}};
>  the server stops answering at {{RETR f-03.txt}}, either before the reply 
> (the client times out on the control connection) or after it sent half of the 
> file (the client times out on the data connection).
> h3. Expected vs actual (4.23.0 and 4.24.0-SNAPSHOT, soTimeout 1 s)
> || ||Expected||Actual||
> |Files that fail|f-03 only|f-03 to f-10 (8 files, 8 WARN)|
> |Commands sent on the dead connection after the timeout|none|PWD x 7, NOOP|
> |Time from the stall until all 10 files are delivered|about 1 x soTimeout|9.2 
> s = (7 + 2) x soTimeout|
> |Dead control connection|closed|never closed|
> h3. Suggested fix
> After a file failed because the server did not answer in time (a 
> {{SocketTimeoutException}}: {{soTimeout}} on the control connection, 
> {{timeout}} on the data connection), close the connection without waiting on 
> it (without the QUIT of CAMEL-26999) and end the batch: {{processBatch()}} 
> releases the remaining files, as when the consumer stops, and the next poll 
> connects again and picks them up. Close a connection that is still open 
> before connecting again.
> h3. Related
> * CAMEL-26999: disconnecting waits {{soTimeout}} for the reply to QUIT.
> * CAMEL-25915, CAMEL-25896: the NOOP before the next poll.
> * CAMEL-10440 (Won't Fix): errors during a download do not reach the poll 
> strategy.
> * CAMEL-26657: the PWD/CWD per file with {{stepwise=true}}.
> * CAMEL-8718: an earlier connection leak on reconnect (invalid credentials).
> _Claude Code on behalf of Croway_



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

Reply via email to