allthingssecurity opened a new pull request, #28035: URL: https://github.com/apache/camel/pull/28035
# Description [CAMEL-25728](https://issues.apache.org/jira/browse/CAMEL-25728) With a done file of a fixed name (`doneFileName=ready`) the file consumer only picks up the files of a directory while the done file exists, and `GenericFileOnCompletion.handleDoneFile` deletes it when the last file of the batch (`CamelBatchComplete`, CAMEL-5883) is committed. The batch is the poll, not the directory. Files the poll did not take then stay without their done file, and are left until a new done file is written, without a log: - with `maxMessagesPerPoll`, the files beyond the limit; - a file whose processing could not start (read lock not acquired, begin or download failed). Change: the poll remembers the files it did not take (beyond the limit, or not started), per done file (`GenericFileDoneFileAdmission`, package-private), and the last file of the batch does not delete the done file while there are any. The next poll only consumes these files, so a file written to the directory meanwhile waits for a new done file as on main; the done file is then deleted after its last file as before, or by the next poll if the files left are gone. A file that was taken and failed is not left: it is handled as on main (waits for the next done file, or is retried under the same done file when it was the last file of the poll), so a file that always fails cannot hold the done file. With exactly `maxMessagesPerPoll` files nothing is left and nothing changes. Dynamic done files (`${file:name}.done`), `noop=true` and `recursive=true` (only the done file of the last file's directory is deleted per poll) behave as before. The FTP, SFTP, SMB and Azure Files consumers share these classes (`GenericFileConsumer`, `GenericFileOnCompletion`). Cost: to know the files beyond the limit, such a consumer checks every file of the directory also with `eagerMaxMessagesPerPoll=true` (the default), and then cuts the list to the same files the eager limit takes (no Exchange for the others). Only for a fixed `doneFileName` with `maxMessagesPerPoll`, and only while more than `maxMessagesPerPoll` files match; with `recursive=true` every sub-directory is listed too (one more listing each on a remote server). Measured locally with 20 000 files and `ready`, `maxMessagesPerPoll=10`: about 0.5 s per poll instead of 0.1 s (5 000 files: about 0.13 s per poll); the per-file check is the one `eagerMaxMessagesPerPoll=false` and a directory without its done file already pay on main. Limits (in memory, cleared on stop): after a restart a kept done file admits the files present at the first poll, as main does with any done file it finds. A producer must not rewrite a done file that is still there to announce new files. A file that can never be started (for example behind a stale lock file) keeps the done file until it is gone or can be started. Tests: - New `FileConsumerDoneFileNameFilesLeftTest` (11 tests). Without the change four fail: `maxMessagesPerPoll=2` with three files (`mock://limit Received message count. Expected: <3> but was: <2>`); the same with `eagerMaxMessagesPerPoll=false` (`mock://noteager ... Expected: <3> but was: <2>`); the read lock of b not acquired the first time (`mock://notstarted ... Expected: <3> but was: <2>`); `maxMessagesPerPoll=1` with x written while the first file is routed, x must wait for a new done file while a, b, c are consumed (`mock://gate ... Expected: <3> but was: <1>`). - The other seven pass on main as well and pin what must not change: exactly `maxMessagesPerPoll` files (done file deleted; a file written meanwhile not consumed); a file failing once waits for the next done file; a last file failing once keeps the done file and is retried with the file written meanwhile; a file that always fails does not block the next batch (d and `ready` again), also with `maxMessagesPerPoll`; `${file:name}.done` unchanged. Tests that expect a file not to be consumed check it once the done file is gone (synchronous routes), so they do not depend on timing. - With the change: camel-file 24, the file tests of camel-core (`org.apache.camel.component.file.**`) 471 run, 12 skipped, 0 failures; the camel-ftp done-file and maxMessagesPerPoll tests (embedded FTP and SFTP servers: 14 FTP done-file and maxMessagesPerPoll ITs, `RemoteFileIgnoreDoPollErrorTest`, `SftpPreSortTest`, `SftpSimpleConsumeRecursiveIT`; 29 tests), 0 failures. Formatter and import-sort checks pass. What is not changed: with asynchronous routing a poll can still run while the last file is completing (the same window as on main). What was not run: the full root build; the whole camel-ftp suite; the camel-smb and camel-azure-files tests. # Target - [x] I checked that the commit is targeting the correct branch (Camel 4 uses the `main` branch) # Tracking - [x] If this is a large change, bug fix, or code improvement, I checked there is a [JIRA issue](https://issues.apache.org/jira/browse/CAMEL) filed for the change (usually before you start working on it): [CAMEL-25728](https://issues.apache.org/jira/browse/CAMEL-25728). # Apache Camel coding standards and style - [x] I checked that each commit in the pull request has a meaningful subject line and body. - [ ] I have run `mvn clean install -DskipTests` locally from root folder and I have committed all auto-generated changes. (I built and tested camel-file, camel-core, camel-ftp-common and camel-ftp, including the formatter and import-sort plugins. I did not run the full root build.) # AI-assisted contributions - [x] If this PR includes AI-generated code, commits have proper co-authorship attribution (e.g., `Co-authored-by` trailers) and the PR description identifies the AI tool used. This PR was prepared with Claude Code (Claude Opus 5.5). The commit carries a `Co-Authored-By` trailer. _Claude Code on behalf of allthingssecurity_ 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
