allthingssecurity opened a new pull request, #30460: URL: https://github.com/apache/camel/pull/30460
# Description [CAMEL-27214](https://issues.apache.org/jira/browse/CAMEL-27214) The commit of a consumed file (`delete=true`, `move`, `moveFailed`) runs when the exchange is done, on the connection of the consumer. When the route hands the exchange over to another thread, as `to("seda:...")` does for an `InOnly` exchange (`GenericFileOnCompletion` goes with the copy), the delete or rename runs on the seda thread while the ftp consumer goes on with the next file of the poll (RETR) or with the next poll (stepwise CWD, LIST). `FtpOperations` has no lock, and an FTP control connection runs one command at a time, so: - the commands and replies of the two threads mix: a thread reads the reply to the other thread's command, so a download, a listing or the delete fails, or a reply (such as the one to PASV) cannot be parsed; - with `stepwise=true` (the default) the delete reads the current directory (PWD) while the next poll is in the directory of the files, changes to the directory of the file from there, and fails: the file stays on the server, its in-progress entry is removed, and the next poll consumes it again (a duplicate), every time the commit and the walk overlap. `SftpOperations` holds a lock in every operation since CAMEL-8887 (the same report for sftp + seda); ftp and ftps never got it. `synchronous=true` does not help, as the seda producer hands the completions over anyway. Fix: - Each public operation of `FtpOperations` holds a `ReentrantLock` (an operation calls others, such as `deleteFile` -> `changeCurrentDirectory`), and so does the connect of `FtpsOperations` (PBSZ / PROT after the login). - `FtpConsumer.pollDirectory` holds the lock from the first change of directory to the change back, so a commit cannot run while the walk is in another directory. A lock per operation alone is not enough for that (see the model). - `disconnectOnStop` does not take the lock (`FtpOperations` overrides it with what the default did): when the stop interrupts a poll, the poll can still run an operation that the server does not answer, and closing the connection ends it instead of waiting for it (CAMEL-26999). `forceDisconnect` and `isConnected` do not take it either. - The Concurrency section of the ftp docs says that a commit on another thread waits for the operation of the consumer. Most of the diff in `FtpOperations` is the indentation of the method bodies inside `lock.lock(); try { ... } finally { lock.unlock(); }`; no operation changed otherwise. Model: a small TLA+ model (`FtpSharedConnection.tla`: one control connection that answers in order, the consumer walk + RETR, the seda delete PWD / CWD / DELE / CWD back) checks "each thread reads the reply to its own command" and "the delete does not fail because of the other thread". On main both are violated; with a lock per operation only (the naive fix) the replies no longer mix but the delete still fails during the walk; with the lock also over the walk both hold (59 distinct states). Tests: `FtpConsumerCommitOnOtherThreadTest` (camel-ftp, with the in-JVM `FakeFtpServer` of the module and an `FTPClient` subclass given with `ftpClient=#client` that records the commands another thread sends while the consumer is in a window): `delete=true` and `to("seda:work")`; the delete of the first file comes while the consumer downloads the second file, and while the next poll lists the directory (stepwise). Each file must be consumed once, deleted, and no command of the seda thread may come inside the window. On main both tests fail: the seda thread sends its commands during the download and during the listing, and in both tests a file is consumed twice (the delete failed). With the change they pass, and the camel-ftp unit tests pass (255 run, 0 failures; the ITs were not run locally, CI runs them). `RemoteFileConsumerSendNoopTest` mocks `FtpOperations`; its mock is now created with the constructor, so it has the lock that the consumer holds during the walk. The new class and `RemoteFileConsumerSendNoopTest` 20 times in one JVM: 940 run, 0 failures. Not changed (sibling, follow-up JIRA): `SftpOperations` (and the mina-sftp operations) take their lock per operation only; the stepwise walk of `AbstractSftpConsumer.pollDirectory` (camel-ftp-common, used by sftp and mina-sftp) does not hold it, so a commit on another thread during the walk can still run from the directory of the walk there (the second case above, which the model shows a lock per operation does not cover). The lock of `SftpOperations` is private, so covering the walk needs a small API change in both components. Not changed: `streamDownload=true` with a route that hands the exchange over keeps the transfer open until the completion on the other thread; the next download cannot run on the same connection while that transfer is open, with or without this change. # Target - [x] I checked that the commit is targeting the correct branch (Camel 4 uses the `main` branch) # Tracking - [x] If this is a large change, bug fix, or code improvement, I checked there is a [JIRA issue](https://issues.apache.org/jira/browse/CAMEL) filed for the change (usually before you start working on it): [CAMEL-27214](https://issues.apache.org/jira/browse/CAMEL-27214). # Apache Camel coding standards and style - [x] I checked that each commit in the pull request has a meaningful subject line and body. - [ ] I have run `mvn clean install -DskipTests` locally from root folder and I have committed all auto-generated changes. (I built the touched modules with the formatter and import-sort plugins; the docs change is in the component page and its catalog copy. I did not run the full root build.) # AI-assisted contributions - [x] If this PR includes AI-generated code, commits have proper co-authorship attribution (e.g., `Co-authored-by` trailers) and the PR description identifies the AI tool used. This PR was prepared with Claude Code (Claude Opus 5.5). The commit carries a `Co-Authored-By` trailer. _Claude Code on behalf of allthingssecurity_ 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
