[
https://issues.apache.org/jira/browse/CAMEL-24090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Federico Mariani updated CAMEL-24090:
-------------------------------------
Attachment: FileConsumerIdempotentEagerKeyLeakOnPollErrorIssueTest.java
> camel-file: eagerly-added idempotent keys are not drained when pollDirectory
> throws mid-poll - scanned files are never consumed (List drain overload
> missed by CAMEL-21830)
> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: CAMEL-24090
> URL: https://issues.apache.org/jira/browse/CAMEL-24090
> Project: Camel
> Issue Type: Bug
> Components: camel-file, camel-ftp
> Affects Versions: 4.21.0
> Reporter: Federico Mariani
> Priority: Major
> Labels: code-review
> Attachments:
> FileConsumerIdempotentEagerKeyLeakOnPollErrorIssueTest.java
>
>
> h3. Problem
> With {{idempotent=true}} (implied by {{noop=true}}) and default
> {{idempotentEager=true}}, keys are added to the idempotent repository during
> directory scanning ({{isValidFile}} -> {{notUnique}}). If {{pollDirectory}}
> then throws (FTP disconnect mid-scan, I/O error in a subdirectory, filter
> expression error), the catch in {{GenericFileConsumer.poll()}}
> (GenericFileConsumer.java:146-157) drains via the *List* overload of
> {{removeExcessiveInProgressFiles}} (line 310-315), which only removes
> in-progress keys - the eagerly-added idempotent keys stay behind:
> {code:java}
> protected void removeExcessiveInProgressFiles(List<GenericFile<T>> files) {
> for (GenericFile file : files) {
> String key = file.getAbsoluteFilePath();
> endpoint.getInProgressRepository().remove(key);
> // no removeExcessiveIdempotentFile(...), unlike the Deque overload
> above it
> }
> }
> {code}
> Every subsequent poll sees {{add(key) == false}} -> the never-processed files
> are treated as already consumed and silently skipped forever (until
> 1000-entry LRU eviction/restart for the memory repo; permanently for
> persistent repos). Affects file, FTP, FTPS and SFTP consumers - remote
> consumers are the likeliest trigger (mid-scan disconnect).
> h3. History
> CAMEL-21830 (130bde60462b, Mar 2025) fixed exactly this leak in the *Deque*
> overload used for batch draining, adding
> {{removeExcessiveIdempotentFile(file, null)}} there - the List overload used
> on the poll-exception path was overlooked. The asymmetry between the two
> drain methods shows this is an oversight, not intent.
> h3. Suggested fix
> Mirror the Deque overload: in the List drain, when {{isIdempotentEager()}}
> and an idempotent repository is configured, call
> {{removeExcessiveIdempotentFile(file, null)}}. Consider extracting a shared
> drain helper so a third incarnation cannot appear.
> h3. Reproducer
> Attached failing JUnit test (place in
> {{core/camel-core/src/test/java/org/apache/camel/component/file/}}, where the
> file component tests live). A filter throws on the second file during the
> first poll only; the first file's eagerly-added key leaks and the file is
> never consumed on subsequent polls.
> ----
> _This issue was researched and filed by Claude Code on behalf of [~croway]
> (GitHub: Croway), as part of a deep code review of camel-file and camel-ftp.
> A failing JUnit reproducer is attached; it fails deterministically on current
> main (4.22.0-SNAPSHOT)._
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
