Bruno Gonçalves created CAMEL-27264:
---------------------------------------

             Summary: camel-ftp - files polled with pollEnrich from sftp stay 
in the in-progress repository and are skipped until restart when the enriched 
exchange completes on another thread
                 Key: CAMEL-27264
                 URL: https://issues.apache.org/jira/browse/CAMEL-27264
             Project: Camel
          Issue Type: Bug
          Components: camel-ftp
    Affects Versions: 4.23.0, 4.10.9
            Reporter: Bruno Gonçalves
         Attachments: sftp-pollenrich-stuck-inprogress-reproducer.tar.gz

h3. Problem

A route polls an sftp folder with {{pollEnrich}} (one file per call, {{move}} 
to an archive folder) and then sends the exchange with JMS request-reply 
({{InOut}}, {{requestTimeout=20000}}) to a second route. After a while some 
files are no longer picked up: they stay in the folder, are never processed, 
and every later poll skips them. Nothing is logged above TRACE ("Skipping as 
file is already in progress"). Only a JVM restart, or renaming the file, makes 
them visible again.

Their names stay in the endpoint's in-progress repository 
({{GenericFileEndpoint.getInProgressRepository()}}, a 
{{MemoryIdempotentRepository}}). Because the {{pollEnrich}} endpoint is 
resolved dynamically, it stays in the endpoint registry when the route is 
removed, so stopping or reinstalling the route does not clear the repository 
either.

h3. Consumer options

{noformat} 
sftp:host/data/outbound/inbox?autoCreate=false&delete=false&recursive=false&passiveMode=true
 
&readLock=changed&fastExistsCheck=true&disconnect=true&throwExceptionOnConnectFailed=true
 &bridgeErrorHandler=true&binary=true&move=archive&sortBy=file:modified 
&maxMessagesPerPoll=1&eagerMaxMessagesPerPoll=false {noformat}

{{pollEnrich}} uses {{timeout=5000}} and an aggregation strategy that copies 
the polled body and headers onto the original exchange.

h3. What we see

At the same time as the files get stuck, the sftp operations of the endpoint 
fail with errors like these (Camel 4.23.0):

{noformat} org.apache.camel.component.file.GenericFileOperationFailedException: 
Cannot list directory: data/outbound/inbox/ProductChanges_...offset_300....xml 
at 
org.apache.camel.component.file.remote.SftpOperations.listFiles(SftpOperations.java:936)
 at 
org.apache.camel.component.file.remote.strategy.SftpExclusiveReadLockCheck.listFilesFast(SftpExclusiveReadLockCheck.java:120)
 at 
org.apache.camel.component.file.remote.strategy.SftpExclusiveReadLockCheck.tryAcquireExclusiveReadLock(SftpExclusiveReadLockCheck.java:55)
 at 
org.apache.camel.component.file.remote.strategy.SftpChangedExclusiveReadLockStrategy.acquireExclusiveReadLock(SftpChangedExclusiveReadLockStrategy.java:73)
 at 
org.apache.camel.component.file.GenericFileConsumer.processExchange(GenericFileConsumer.java:430)
 at 
org.apache.camel.component.file.remote.RemoteFileConsumer.processExchange(RemoteFileConsumer.java:197)
 at 
org.apache.camel.component.file.GenericFileConsumer.poll(GenericFileConsumer.java:216)
 at 
org.apache.camel.component.file.GenericFilePollingConsumer.doReceive(GenericFilePollingConsumer.java:152)
 at 
org.apache.camel.component.file.GenericFilePollingConsumer.receive(GenericFilePollingConsumer.java:102)
 Caused by: 4: java.io.IOException: Pipe closed

org.apache.camel.CamelExchangeException: Error during poll. Caused by: 
[GenericFileOperationFailedException - Cannot change directory to: ..] at 
org.apache.camel.processor.PollEnricher.process(PollEnricher.java:379) Caused 
by: org.apache.camel.component.file.GenericFileOperationFailedException: Cannot 
change directory to: .. at 
org.apache.camel.component.file.remote.SftpOperations.doChangeDirectory(SftpOperations.java:887)
 at 
org.apache.camel.component.file.remote.SftpOperations.changeCurrentDirectory(SftpOperations.java:872)
 at 
org.apache.camel.component.file.remote.AbstractSftpConsumer.pollDirectory(AbstractSftpConsumer.java:94)
 at 
org.apache.camel.component.file.GenericFileConsumer.poll(GenericFileConsumer.java:151)
 at 
org.apache.camel.component.file.GenericFilePollingConsumer.doReceive(GenericFilePollingConsumer.java:152)
 at 
org.apache.camel.component.file.GenericFilePollingConsumer.receive(GenericFilePollingConsumer.java:102)
 Caused by: 4: java.io.IOException: Pipe closed at 
com.jcraft.jsch.ChannelSftp.cd(ChannelSftp.java:344) {noformat}

In production (4.10.9) we also see the move to the archive folder fail 
({{Cannot rename file ... 2: No such file}}, {{Cannot create directory: 
.../archive}}) on the JMS reply thread, within a second of a poll failing on 
the route's own thread.

h3. Analysis so far

For file endpoints {{pollEnrich}} uses {{GenericFilePollingConsumer}}, which 
polls synchronously on the calling thread ({{doReceive}} -> 
{{GenericFileConsumer.poll()}}).
With JMS request-reply after the {{pollEnrich}}, the exchange (and the 
handed-over {{GenericFileOnCompletion}} that moves the file and removes the 
name from the in-progress repository) completes on the JMS reply-manager thread 
({{TemporaryQueueReplyManager}} or {{JmsReplyManagerOnTimeout}}).
So the commit of file N runs on the reply thread while the poll for file N+1 
runs on the route's thread, both through the endpoint's single 
{{SftpOperations}}. With {{disconnect=true}} one side closes the channel under 
the other ("Pipe closed"). With the default {{stepwise=true}} one side changes 
the current directory under the other ("Cannot change directory to: ..", "No 
such file" on the rename).
Some of these collisions leave the file's name in the in-progress repository. 
We have not pinned down the exact code path. In one run a stuck file was moved 
to the archive folder much later, so at least some entries are released 
eventually rather than lost. In other runs a renamed copy of a stuck file was 
not picked up within 2 minutes either.
h3. Reproducer

Attached: {{sftp-pollenrich-stuck-inprogress-reproducer.tar.gz}} (plain Java, 
JDK 21, about 270 lines). It starts an embedded sftp server (Apache MINA SSHD, 
40 ms added per request) and an embedded ActiveMQ broker, and runs:

a writer route that writes 5 files into the inbox every 60 s,
a timer route (every 10 s, runs never overlap) with the {{pollEnrich}} above, 
followed by 
{{to("activemq:queue:next-route?exchangePattern=InOut&requestTimeout=20000")}},
a second route on that queue that usually answers within 5 s and in one of five 
calls takes 15-30 s (so some requests time out).
It reads the endpoint's in-progress repository every second and, after the 
writer stops and the poller has drained the folder, reports files that were 
never processed and whose name has been in the repository for over 60 s.

{noformat} $ mvn compile exec:java -Dexec.args="current 600 600"

written=50 processed=40 left in inbox=14 stuck (never processed, in progress 
for over 60 s)=10 STUCK 
ProductChanges_20261119_180332_offset_0_20261119T180332237.xml ...

$ mvn compile exec:java -Dexec.args="fixed 600 600"

written=50 processed=50 left in inbox=0 stuck (never processed, in progress for 
over 60 s)=0 {noformat}

Use {{-Dcamel.version=4.23.0 -Dactivemq.version=6.3.2}} for the latest release 
(default 4.10.9 with 6.1.5).

||Camel||Options||Written||Stuck||SFTP errors||
|4.10.9|as above|50|12|34x Pipe closed, 6x Cannot change directory|
|4.23.0|as above|50|10|32x Pipe closed, 2x Cannot change directory to: ..|
|4.23.0|fixed|50|0|none (18 request timeouts)|
|4.10.9|as above, seda instead of JMS request-reply|50|0|none|

"fixed" adds {{readLockCheckInterval=1000&readLockTimeout=4000&stepwise=false}} 
and uses {{disconnect=false}}. We have not yet tested these options one by one.

h3. Expected

A file whose exchange has finished, failed or timed out should always be 
removed from the in-progress repository, as for the poll-error case in 
CAMEL-7407. If an entry can stay for a long time, a WARN log would make the 
silent skipping visible.

h3. Related

CAMEL-7407: FTP route considers files as in progress after error in poll (fixed 
in 2.12.4, 2.13.2, 2.14.0).

----
_This ticket was created using Claude (Anthropic). The reproducer and results 
were run and checked by the reporter._



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

Reply via email to