allthingssecurity opened a new pull request, #30460:
URL: https://github.com/apache/camel/pull/30460

   # Description
   
   [CAMEL-27214](https://issues.apache.org/jira/browse/CAMEL-27214)
   
   The commit of a consumed file (`delete=true`, `move`, `moveFailed`) runs 
when the exchange is done, on the connection of the consumer. When the route 
hands the exchange over to another thread, as `to("seda:...")` does for an 
`InOnly` exchange (`GenericFileOnCompletion` goes with the copy), the delete or 
rename runs on the seda thread while the ftp consumer goes on with the next 
file of the poll (RETR) or with the next poll (stepwise CWD, LIST). 
`FtpOperations` has no lock, and an FTP control connection runs one command at 
a time, so:
   
   - the commands and replies of the two threads mix: a thread reads the reply 
to the other thread's command, so a download, a listing or the delete fails, or 
a reply (such as the one to PASV) cannot be parsed;
   - with `stepwise=true` (the default) the delete reads the current directory 
(PWD) while the next poll is in the directory of the files, changes to the 
directory of the file from there, and fails: the file stays on the server, its 
in-progress entry is removed, and the next poll consumes it again (a 
duplicate), every time the commit and the walk overlap.
   
   `SftpOperations` holds a lock in every operation since CAMEL-8887 (the same 
report for sftp + seda); ftp and ftps never got it. `synchronous=true` does not 
help, as the seda producer hands the completions over anyway.
   
   Fix:
   
   - Each public operation of `FtpOperations` holds a `ReentrantLock` (an 
operation calls others, such as `deleteFile` -> `changeCurrentDirectory`), and 
so does the connect of `FtpsOperations` (PBSZ / PROT after the login).
   - `FtpConsumer.pollDirectory` holds the lock from the first change of 
directory to the change back, so a commit cannot run while the walk is in 
another directory. A lock per operation alone is not enough for that (see the 
model).
   - `disconnectOnStop` does not take the lock (`FtpOperations` overrides it 
with what the default did): when the stop interrupts a poll, the poll can still 
run an operation that the server does not answer, and closing the connection 
ends it instead of waiting for it (CAMEL-26999). `forceDisconnect` and 
`isConnected` do not take it either.
   - The Concurrency section of the ftp docs says that a commit on another 
thread waits for the operation of the consumer.
   
   Most of the diff in `FtpOperations` is the indentation of the method bodies 
inside `lock.lock(); try { ... } finally { lock.unlock(); }`; no operation 
changed otherwise.
   
   Model: a small TLA+ model (`FtpSharedConnection.tla`: one control connection 
that answers in order, the consumer walk + RETR, the seda delete PWD / CWD / 
DELE / CWD back) checks "each thread reads the reply to its own command" and 
"the delete does not fail because of the other thread". On main both are 
violated; with a lock per operation only (the naive fix) the replies no longer 
mix but the delete still fails during the walk; with the lock also over the 
walk both hold (59 distinct states).
   
   Tests: `FtpConsumerCommitOnOtherThreadTest` (camel-ftp, with the in-JVM 
`FakeFtpServer` of the module and an `FTPClient` subclass given with 
`ftpClient=#client` that records the commands another thread sends while the 
consumer is in a window): `delete=true` and `to("seda:work")`; the delete of 
the first file comes while the consumer downloads the second file, and while 
the next poll lists the directory (stepwise). Each file must be consumed once, 
deleted, and no command of the seda thread may come inside the window.
   
   On main both tests fail: the seda thread sends its commands during the 
download and during the listing, and in both tests a file is consumed twice 
(the delete failed). With the change they pass, and the camel-ftp unit tests 
pass (255 run, 0 failures; the ITs were not run locally, CI runs them). 
`RemoteFileConsumerSendNoopTest` mocks `FtpOperations`; its mock is now created 
with the constructor, so it has the lock that the consumer holds during the 
walk. The new class and `RemoteFileConsumerSendNoopTest` 20 times in one JVM: 
940 run, 0 failures.
   
   Not changed (sibling, follow-up JIRA): `SftpOperations` (and the mina-sftp 
operations) take their lock per operation only; the stepwise walk of 
`AbstractSftpConsumer.pollDirectory` (camel-ftp-common, used by sftp and 
mina-sftp) does not hold it, so a commit on another thread during the walk can 
still run from the directory of the walk there (the second case above, which 
the model shows a lock per operation does not cover). The lock of 
`SftpOperations` is private, so covering the walk needs a small API change in 
both components.
   
   Not changed: `streamDownload=true` with a route that hands the exchange over 
keeps the transfer open until the completion on the other thread; the next 
download cannot run on the same connection while that transfer is open, with or 
without this change.
   
   # Target
   
   - [x] I checked that the commit is targeting the correct branch (Camel 4 
uses the `main` branch)
   
   # Tracking
   - [x] If this is a large change, bug fix, or code improvement, I checked 
there is a [JIRA issue](https://issues.apache.org/jira/browse/CAMEL) filed for 
the change (usually before you start working on it): 
[CAMEL-27214](https://issues.apache.org/jira/browse/CAMEL-27214).
   
   # Apache Camel coding standards and style
   
   - [x] I checked that each commit in the pull request has a meaningful 
subject line and body.
   - [ ] I have run `mvn clean install -DskipTests` locally from root folder 
and I have committed all auto-generated changes.
     (I built the touched modules with the formatter and import-sort plugins; 
the docs change is in the component page and its catalog copy. I did not run 
the full root build.)
   
   # AI-assisted contributions
   
   - [x] If this PR includes AI-generated code, commits have proper 
co-authorship attribution (e.g., `Co-authored-by` trailers) and the PR 
description identifies the AI tool used.
     This PR was prepared with Claude Code (Claude Opus 5.5). The commit 
carries a `Co-Authored-By` trailer.
   
   _Claude Code on behalf of allthingssecurity_
   
   🤖 Generated with [Claude Code](https://claude.com/claude-code)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to