shashank created CAMEL-25728:
--------------------------------
Summary: camel-file - a done file with a fixed name is deleted by
the last file of a poll although the poll left files of the directory
(maxMessagesPerPoll, not started), so they are left until a new done file is
written
Key: CAMEL-25728
URL: https://issues.apache.org/jira/browse/CAMEL-25728
Project: Camel
Issue Type: Bug
Components: camel-file
Reporter: shashank
Assignee: shashank
With a done file of a fixed name ({{doneFileName=ready}}), the file consumer
only picks up the files of a directory while the done file exists, and deletes
the done file when the last file of the batch ({{CamelBatchComplete}}) is
committed ({{GenericFileOnCompletion.handleDoneFile}}, main 55ed20f14e90 line
189; CAMEL-5883). The batch is the poll, not the directory. When the poll does
not take some files of the directory, the done file is still deleted, and these
files are left in the directory until a new done file is written, without any
log:
* {{maxMessagesPerPoll}}: the poll takes N files; the last of them deletes the
done file; the other files stay;
* a file whose processing cannot start (read lock not acquired, begin or
download failed) is skipped by the poll, and a later file of the same poll
deletes the done file.
Trigger: {{file:in?doneFileName=ready&maxMessagesPerPoll=2}} with three files
and {{ready}}: two files are routed, the third stays in {{in}} until the
producer writes {{ready}} again. A done file with a fixed name is the
documented way to say that "all files of this directory are ready" ("Camel will
automatically delete the done file when it's done consuming the files"), and a
producer usually writes the next done file only with the next batch. The same
base classes are used by camel-ftp (ftp, ftps, sftp), camel-smb and
camel-azure-files.
A file that was processed and failed is not part of this issue: it is rolled
back and waits for the next done file (or is retried under the same done file
when it was the last file of the poll), and {{moveFailed}} or redelivery are
the tools for it. The fix keeps that behaviour.
h3. Reproduction
New test {{FileConsumerDoneFileNameFilesLeftTest}} (11 tests). Without the fix
four fail:
* {{maxMessagesPerPoll=2}}, files a, b, c and {{ready}}: {{mock://limit
Received message count. Expected: <3> but was: <2>}};
* the same with {{eagerMaxMessagesPerPoll=false}}: {{mock://noteager ...
Expected: <3> but was: <2>}};
* the read lock of b not acquired the first time (a custom
{{exclusiveReadLockStrategy}}): {{mock://notstarted ... Expected: <3> but was:
<2>}};
* {{maxMessagesPerPoll=1}}, files a, b, c and {{ready}}, a file x written while
the first file is routed: {{mock://gate ... Expected: <3> but was: <1>}} (x
must wait for a new done file, a, b and c must not).
The other seven pass on main as well and pin the behaviour that must not
change: exactly {{maxMessagesPerPoll}} files and {{ready}} deleted; the same
with a file written meanwhile, not consumed; a file failing once waits for the
next done file; a last file failing once keeps the done file and is retried
with the file written meanwhile (as on main); a file that always fails does not
hold the done file, and the next batch (d and {{ready}} again) is consumed,
also with {{maxMessagesPerPoll}}; a dynamic {{$\{file:name\}.done}} unchanged.
A test that expects a file not to be consumed checks it once the done file is
gone (the routes are synchronous), so it does not depend on timing.
A TLA+ model (one directory: files, done file, limit per poll, files failing
once or always, files not started once, a producer that writes files of a next
batch at any time and announces them with a new done file, consumer restart)
checks with the fix: no file consumed unless a done file was created after it
(except after a rolled back last file, where main does the same), every left
file eventually consumed, no orphan done file, and every later file eventually
consumed (no permanent block by a failing file). On main a left file is
stranded. A previous version of this fix, which kept a failed file in the set,
blocked the directory behind a file that always fails; the model shows that
too. Two limits show up as expected (below): a restart while the done file is
kept, and a producer that rewrites a done file that is still there. Not
modelled: asynchronous routing, recursive directories, other consumers.
h3. Proposed fix
The poll remembers the files it did not take (beyond {{maxMessagesPerPoll}}, or
not started), per done file, and the last file of the batch does not delete the
done file while there are any. The next poll only consumes these files, so a
file written to the directory meanwhile waits for a new done file as on main;
the done file is then deleted after its last file as before, or by the next
poll if the files left are gone. A file that was taken and failed is not left
(released), so it cannot hold the done file. With exactly
{{maxMessagesPerPoll}} files nothing is left and nothing changes.
To know the files beyond the limit, such a consumer checks every file of the
directory also with {{eagerMaxMessagesPerPoll=true}} (the default), and then
cuts the list to the same files the eager limit takes (no Exchange for the
others). Only for a fixed {{doneFileName}} with {{maxMessagesPerPoll}}: each
poll then costs what {{eagerMaxMessagesPerPoll=false}} costs for the listing
(20 000 files and {{ready}}: about 0.5 s per poll instead of 0.1 s, as long as
files are left; with {{recursive=true}} every sub-directory is listed too, one
more listing each on a remote server). The state is per consumer, bounded by
the files left, and cleared on stop. After a restart a kept done file admits
the files present at the first poll, as main does with any done file it finds.
A producer must not rewrite a done file that is still there to announce new
files. Dynamic done files ({{$\{file:name\}.done}}), {{noop=true}} and
{{recursive=true}} (only the done file of the last file's directory is deleted
per poll) are unchanged.
Not changed: with asynchronous routing a poll can still run while the last file
is completing (the same window as on main). A file that can never be started
(for example behind a stale lock file) keeps the done file until it is gone or
can be started.
Affected: main (tested); camel-4.23.x and camel-4.22.x have the same lines
(169/189), camel-4.18.x and camel-4.14.x (173/193) (read, not run).
Duplicate check (2026-10-17): JIRA text "done file" with "maxMessagesPerPoll"
(0), with "rollback"/"retry" (CAMEL-6964, moveFailed only), "doneFileName" (16,
none about files left); GitHub pull requests "doneFileName": none open or
related.
_Filed with Claude Code on behalf of allthingssecurity._
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
