shashank created CAMEL-25681:
--------------------------------
Summary: camel-support - FileIdempotentRepository.add keeps a key
in its cache when the file store cannot be read or written, so the retry of
that message is dropped as a duplicate
Key: CAMEL-25681
URL: https://issues.apache.org/jira/browse/CAMEL-25681
Project: Camel
Issue Type: Bug
Components: camel-core
Reporter: shashank
Assignee: shashank
{{FileIdempotentRepository.add(key)}} (main cd2ce899764e, lines 145-176) puts
the key into its 1st-level cache first, then checks the file store
({{containsStore}}) and appends the key to it ({{appendToStore}}). When the
store cannot be read or written (disk full, permissions, an I/O error of a
network file system) these methods throw a {{RuntimeCamelException}}, {{add}}
throws, and the message fails - but the key stays in the cache. Every later
{{add}} of the same key returns {{false}} from the cache, until the application
is restarted (a key that was never stored is then gone from the cache too).
Consequence: the redelivery of the failed message is taken for a duplicate and
skipped, also after the disk has space again. For example, an
{{idempotentConsumer}} with a {{FileIdempotentRepository}} after a JMS
consumer: during a "disk full" period every message fails (logged), the broker
redelivers them, and the redeliveries are filtered as duplicates and
acknowledged - lost without a further trace. (The file consumer with
{{idempotentRepository}} calls {{add}} while it polls; what it does with the
file after such a failure was not checked.)
{{KafkaIdempotentRepository.add}} (camel-kafka, lines 352-363 on main) already
removes the key from its cache again when storing it (broadcasting it on the
topic) fails. The other repositories of the project have one store only.
h3. Reproduction
New {{FileIdempotentRepositoryFailedStoreTest}} (camel-support):
* {{appendToStore}} fails once ("No space left on device"), then works: {{[the
retry of a key that was not stored] Expecting value to be true but was false}};
* the store cannot be read (a directory took its place), then it can: the same
failure.
* A third test (passes on main too) pins that a key found in the store is still
put into the cache.
h3. Proposed fix
Put the key into the cache only after it was found in the store or appended to
it. The order of the store accesses does not change, and a key found in the
store is still cached (CAMEL-12148). No option or behaviour change for a store
that works, so no upgrade guide entry.
Tests with the fix: camel-support 136 run, 0 failures; camel-core tests
matching {{*Idempotent*}}: 77 run, 0 failures. Not run: the full camel-core
suite.
Not changed: when {{trunkStore}} (the maximum file size is reached) fails after
the key was appended, {{add}} still throws although the key is stored, so the
retry is a duplicate; and {{remove}} removes the key from the cache before the
store, so a failing store keeps it in the file (but {{contains}} reads the file
and still answers {{true}}).
How it was found: a review of the repositories that keep a cache in front of a
store (backlog item of an earlier sweep); the test confirmed it.
Affected: main (cd2ce899764e; tested). camel-4.23.x, camel-4.22.x, camel-4.18.x
and camel-4.14.x put the key into the cache before the store accesses the same
way (read, not run).
Duplicate check (2026-10-15): JIRA text "FileIdempotentRepository" (17 issues,
the latest CAMEL-24953 / CAMEL-24954 about the idempotent consumer, none about
a failing store); GitHub pull requests "FileIdempotentRepository" (#22114,
camel-atom; nothing related).
_Filed with Claude Code on behalf of allthingssecurity._
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
