[
https://issues.apache.org/jira/browse/CAMEL-25728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Claus Ibsen updated CAMEL-25728:
--------------------------------
Fix Version/s: 4.24.0
> camel-file - a done file with a fixed name is deleted by the last file of a
> poll although the poll left files of the directory (maxMessagesPerPoll, not
> started), so they are left until a new done file is written
> -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: CAMEL-25728
> URL: https://issues.apache.org/jira/browse/CAMEL-25728
> Project: Camel
> Issue Type: Bug
> Components: camel-file
> Reporter: shashank
> Assignee: shashank
> Priority: Major
> Fix For: 4.24.0
>
>
> With a done file of a fixed name ({{doneFileName=ready}}), the file consumer
> only picks up the files of a directory while the done file exists, and
> deletes the done file when the last file of the batch
> ({{CamelBatchComplete}}) is committed
> ({{GenericFileOnCompletion.handleDoneFile}}, main 55ed20f14e90 line 189;
> CAMEL-5883). The batch is the poll, not the directory. When the poll does not
> take some files of the directory, the done file is still deleted, and these
> files are left in the directory until a new done file is written, without any
> log:
> * {{maxMessagesPerPoll}}: the poll takes N files; the last of them deletes
> the done file; the other files stay;
> * a file whose processing cannot start (read lock not acquired, begin or
> download failed) is skipped by the poll, and a later file of the same poll
> deletes the done file.
> Trigger: {{file:in?doneFileName=ready&maxMessagesPerPoll=2}} with three files
> and {{ready}}: two files are routed, the third stays in {{in}} until the
> producer writes {{ready}} again. A done file with a fixed name is the
> documented way to say that "all files of this directory are ready" ("Camel
> will automatically delete the done file when it's done consuming the files"),
> and a producer usually writes the next done file only with the next batch.
> The same base classes are used by camel-ftp (ftp, ftps, sftp), camel-smb and
> camel-azure-files.
> A file that was processed and failed is not part of this issue: it is rolled
> back and waits for the next done file (or is retried under the same done file
> when it was the last file of the poll), and {{moveFailed}} or redelivery are
> the tools for it. The fix keeps that behaviour.
> h3. Reproduction
> New test {{FileConsumerDoneFileNameFilesLeftTest}} (11 tests). Without the
> fix four fail:
> * {{maxMessagesPerPoll=2}}, files a, b, c and {{ready}}: {{mock://limit
> Received message count. Expected: <3> but was: <2>}};
> * the same with {{eagerMaxMessagesPerPoll=false}}: {{mock://noteager ...
> Expected: <3> but was: <2>}};
> * the read lock of b not acquired the first time (a custom
> {{exclusiveReadLockStrategy}}): {{mock://notstarted ... Expected: <3> but
> was: <2>}};
> * {{maxMessagesPerPoll=1}}, files a, b, c and {{ready}}, a file x written
> while the first file is routed: {{mock://gate ... Expected: <3> but was:
> <1>}} (x must wait for a new done file, a, b and c must not).
> The other seven pass on main as well and pin the behaviour that must not
> change: exactly {{maxMessagesPerPoll}} files and {{ready}} deleted; the same
> with a file written meanwhile, not consumed; a file failing once waits for
> the next done file; a last file failing once keeps the done file and is
> retried with the file written meanwhile (as on main); a file that always
> fails does not hold the done file, and the next batch (d and {{ready}} again)
> is consumed, also with {{maxMessagesPerPoll}}; a dynamic
> {{$\{file:name\}.done}} unchanged. A test that expects a file not to be
> consumed checks it once the done file is gone (the routes are synchronous),
> so it does not depend on timing.
> A TLA+ model (one directory: files, done file, limit per poll, files failing
> once or always, files not started once, a producer that writes files of a
> next batch at any time and announces them with a new done file, consumer
> restart) checks with the fix: no file consumed unless a done file was created
> after it (except after a rolled back last file, where main does the same),
> every left file eventually consumed, no orphan done file, and every later
> file eventually consumed (no permanent block by a failing file). On main a
> left file is stranded. A previous version of this fix, which kept a failed
> file in the set, blocked the directory behind a file that always fails; the
> model shows that too. Two limits show up as expected (below): a restart while
> the done file is kept, and a producer that rewrites a done file that is still
> there. Not modelled: asynchronous routing, recursive directories, other
> consumers.
> h3. Proposed fix
> The poll remembers the files it did not take (beyond {{maxMessagesPerPoll}},
> or not started), per done file, and the last file of the batch does not
> delete the done file while there are any. The next poll only consumes these
> files, so a file written to the directory meanwhile waits for a new done file
> as on main; the done file is then deleted after its last file as before, or
> by the next poll if the files left are gone. A file that was taken and failed
> is not left (released), so it cannot hold the done file. With exactly
> {{maxMessagesPerPoll}} files nothing is left and nothing changes.
> To know the files beyond the limit, such a consumer checks every file of the
> directory also with {{eagerMaxMessagesPerPoll=true}} (the default), and then
> cuts the list to the same files the eager limit takes (no Exchange for the
> others). Only for a fixed {{doneFileName}} with {{maxMessagesPerPoll}}: each
> poll then costs what {{eagerMaxMessagesPerPoll=false}} costs for the listing
> (20 000 files and {{ready}}: about 0.5 s per poll instead of 0.1 s, as long
> as files are left; with {{recursive=true}} every sub-directory is listed too,
> one more listing each on a remote server). The state is per consumer, bounded
> by the files left, and cleared on stop. After a restart a kept done file
> admits the files present at the first poll, as main does with any done file
> it finds. A producer must not rewrite a done file that is still there to
> announce new files. Dynamic done files ({{$\{file:name\}.done}}),
> {{noop=true}} and {{recursive=true}} (only the done file of the last file's
> directory is deleted per poll) are unchanged.
> Not changed: with asynchronous routing a poll can still run while the last
> file is completing (the same window as on main). A file that can never be
> started (for example behind a stale lock file) keeps the done file until it
> is gone or can be started.
> Affected: main (tested); camel-4.23.x and camel-4.22.x have the same lines
> (169/189), camel-4.18.x and camel-4.14.x (173/193) (read, not run).
> Duplicate check (2026-10-17): JIRA text "done file" with "maxMessagesPerPoll"
> (0), with "rollback"/"retry" (CAMEL-6964, moveFailed only), "doneFileName"
> (16, none about files left); GitHub pull requests "doneFileName": none open
> or related.
> _Filed with Claude Code on behalf of allthingssecurity._
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
