shashank created CAMEL-25710:
--------------------------------
Summary: camel-mail - with an idempotentRepository, mails of a
poll that are not routed (consumer stopped mid-batch, or the poll failed on a
later mail) keep their key and are skipped as duplicates by every later poll
Key: CAMEL-25710
URL: https://issues.apache.org/jira/browse/CAMEL-25710
Project: Camel
Issue Type: Bug
Components: camel-mail
Reporter: shashank
Assignee: shashank
With an {{idempotentRepository}} (the documented way to let several consumers
share one mailbox), {{MailConsumer.retrieveMessages}} adds the key of every
mail it retrieves to the repository ({{isValidMessage}}, main 71633593ebdc
{{MailConsumer.java}} 369-381, {{retrieveMessages}} 327), before any of them is
routed. The key is removed (default
{{idempotentRepositoryRemoveOnCommit=true}}) or confirmed when the exchange of
the mail completes, and removed on rollback. Mails that the poll retrieved but
did not route keep their key, and every later poll skips them as duplicates,
also on the other consumers sharing the repository. They stay unread on the
server and are never routed, without any log line; with a persistent repository
this survives restarts.
Two paths leave keys behind:
* *Consumer stopped in the middle of a batch.* {{processBatch}} (224-270) stops
at {{isBatchAllowed()}} when the route or the CamelContext is stopped
gracefully (default {{ShutdownRunningTask.CompleteCurrentTaskOnly}}): the rest
of the batch (up to {{maxMessagesPerPoll}} / {{fetchSize}}, unlimited by
default) is neither routed nor given back. The exchanges are not released
either.
* *Poll failed while retrieving.* When creating the exchange of a mail throws
(for example its attachments cannot be read, default {{mapMailMessage=true}},
{{skipFailedMessage=false}}), the poll fails as documented and no mail of it is
routed; but the keys of the mails retrieved before it, and its own key, stay in
the repository. After the bad mail is removed, those mails are still skipped.
h3. Reproduction
New {{MailIdempotentRepositoryNotRoutedTest}} (camel-mail, GreenMail IMAP,
{{MemoryIdempotentRepository}}):
* 5 mails; the route is stopped while it routes the first one. On main 4 keys
stay in the repository ({{expected: <0> but was: <4>}}), so a later poll skips
these 4 mails. With the fix they are routed after the route is started again.
* 3 mails and a multipart mail without its boundary; the poll fails on the last
one. On main the 4 keys stay ({{expected: <0> but was: <4>}}). With the fix the
3 good mails are routed once the bad mail is deleted.
h3. Proposed fix
{{processBatch}} removes the keys of the exchanges it did not route and
releases them, as the aws2-s3 consumer does with its in-progress keys since
CAMEL-25521; {{retrieveMessages}} does the same (and removes the key of the
failing mail) when creating an exchange fails, then rethrows. No option or
default changes.
Behaviour change (upgrade guide note added): with an {{idempotentRepository}}
and the defaults {{skipFailedMessage=false}} / {{handleFailedMessage=false}}, a
mail whose exchange cannot be created now fails every poll until it is removed,
as it already does without an {{idempotentRepository}} and as the option
documentation describes ("the consumer throws an exception and no mails from
the batch would be able to be routed"). Before, it failed only the first poll
and was then skipped, together with the mails retrieved before it.
{{skipFailedMessage}} or {{handleFailedMessage}} skip such a mail.
Not changed: a mail skipped by {{skipFailedMessage}} / {{handleFailedMessage}}
(or flagged DELETED) keeps its key, as before.
How it was found: a sweep of the file-like consumers that commit or give back
messages on the wrong outcome, after CAMEL-25521 / CAMEL-25551 fixed the
in-progress keys of the object store consumers; the mail consumer's idempotent
repository plays the same role and has no give-back path. The test confirmed it.
Affected: main (71633593ebdc; tested). camel-4.23.x, camel-4.22.x, camel-4.18.x
and camel-4.14.x have the same {{processBatch}} loop and {{isValidMessage}} add
(read, not run).
Duplicate check (2026-10-16): JIRA text "mail idempotentRepository",
"MailConsumer", component camel-mail since 2025-06 (CAMEL-23046, CAMEL-24432,
CAMEL-24434, CAMEL-24403, CAMEL-24567: none about keys of mails that were not
routed); GitHub open pull requests "camel-mail" (none touching MailConsumer).
_Filed with Claude Code on behalf of allthingssecurity._
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
