Federico Mariani created CAMEL-24090:
----------------------------------------
Summary: camel-file: eagerly-added idempotent keys are not drained
when pollDirectory throws mid-poll - scanned files are never consumed (List
drain overload missed by CAMEL-21830)
Key: CAMEL-24090
URL: https://issues.apache.org/jira/browse/CAMEL-24090
Project: Camel
Issue Type: Bug
Components: camel-ftp, camel-file
Affects Versions: 4.21.0
Reporter: Federico Mariani
Attachments:
FileConsumerIdempotentEagerKeyLeakOnPollErrorIssueTest.java
h3. Problem
With {{idempotent=true}} (implied by {{noop=true}}) and default
{{idempotentEager=true}}, keys are added to the idempotent repository during
directory scanning ({{isValidFile}} -> {{notUnique}}). If {{pollDirectory}}
then throws (FTP disconnect mid-scan, I/O error in a subdirectory, filter
expression error), the catch in {{GenericFileConsumer.poll()}}
(GenericFileConsumer.java:146-157) drains via the *List* overload of
{{removeExcessiveInProgressFiles}} (line 310-315), which only removes
in-progress keys - the eagerly-added idempotent keys stay behind:
{code:java}
protected void removeExcessiveInProgressFiles(List<GenericFile<T>> files) {
for (GenericFile file : files) {
String key = file.getAbsoluteFilePath();
endpoint.getInProgressRepository().remove(key);
// no removeExcessiveIdempotentFile(...), unlike the Deque overload
above it
}
}
{code}
Every subsequent poll sees {{add(key) == false}} -> the never-processed files
are treated as already consumed and silently skipped forever (until 1000-entry
LRU eviction/restart for the memory repo; permanently for persistent repos).
Affects file, FTP, FTPS and SFTP consumers - remote consumers are the likeliest
trigger (mid-scan disconnect).
h3. History
CAMEL-21830 (130bde60462b, Mar 2025) fixed exactly this leak in the *Deque*
overload used for batch draining, adding {{removeExcessiveIdempotentFile(file,
null)}} there - the List overload used on the poll-exception path was
overlooked. The asymmetry between the two drain methods shows this is an
oversight, not intent.
h3. Suggested fix
Mirror the Deque overload: in the List drain, when {{isIdempotentEager()}} and
an idempotent repository is configured, call
{{removeExcessiveIdempotentFile(file, null)}}. Consider extracting a shared
drain helper so a third incarnation cannot appear.
h3. Reproducer
Attached failing JUnit test (place in
{{core/camel-core/src/test/java/org/apache/camel/component/file/}}, where the
file component tests live). A filter throws on the second file during the first
poll only; the first file's eagerly-added key leaks and the file is never
consumed on subsequent polls.
----
_This issue was researched and filed by Claude Code on behalf of [~croway]
(GitHub: Croway), as part of a deep code review of camel-file and camel-ftp. A
failing JUnit reproducer is attached; it fails deterministically on current
main (4.22.0-SNAPSHOT)._
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
