Federico Mariani created CAMEL-27008:
----------------------------------------
Summary: camel-core - Aggregate EIP: when a recoverable repository
throws while confirming an exchange that the recover task moved to the dead
letter channel, it is moved there again
Key: CAMEL-27008
URL: https://issues.apache.org/jira/browse/CAMEL-27008
Project: Camel
Issue Type: Bug
Components: camel-core
Reporter: Federico Mariani
Fix For: 4.24.0
With a {{RecoverableAggregationRepository}} that has a {{deadLetterUri}} and
{{maximumRedeliveries}}, the recover task moves an aggregated exchange that
keeps failing to the dead letter channel, and then confirms it on the
repository, so that it is not recovered again. That confirm is not guarded
({{core/camel-core-processor/src/main/java/org/apache/camel/processor/aggregate/AggregateProcessor.java}},
lines on main at {{21b8ed8e9613}}; the code of this path is the same in 4.22.x
and 4.23.0):
* {{RecoverTask.run}} sends the exhausted exchange to the dead letter channel
(line 1907) and then calls {{recoverable.confirm(camelContext, exchangeId)}}
(line 1921). When that confirm throws, the exception leaves the scan loop and
is logged by the outer catch ({{Error during recover task. This exception is
ignored.}}, line 1965).
* The id is not added to {{unconfirmedCompleteExchanges}}, and its redelivery
state still says it is exhausted. On the next pass the scan returns it again,
it is neither in progress nor unconfirmed (line 1874), so it is recovered (line
1883) and moved to the dead letter channel *again*. This repeats for as long as
the confirm fails.
* The ids after it in the scan are neither recovered nor confirmed in that
pass, and not at all while that confirm keeps failing and comes first in the
scan.
This is the dead letter channel counterpart of CAMEL-26995, which made a
confirm that throws in {{AggregateOnCompletion.onComplete}} and
{{RecoverTask.retryConfirm}} be handled like a confirm that returns {{false}}
(CAMEL-17613: "retry it in the RecoverTask, but do not redeliver the
exchange"). The dead letter path also calls {{confirm}} where the javadoc of
{{AggregationRepository.confirm}} says that {{confirmWithResult}} is invoked
instead for a recoverable repository, so a {{false}} result is ignored there,
and the exchange is moved again if the repository still returns it.
{{JdbcAggregationRepository.confirm}} runs a {{DELETE}} in a
{{TransactionTemplate}} and throws when the database or the connection is not
available, as fault-injection tests against PostgreSQL showed for the confirm
after processing (CAMEL-26995). The other recoverable repositories throw
whatever their store throws.
h3. Steps to reproduce
An in-memory {{RecoverableAggregationRepository}} that moves a completed group
to a completed store (like {{JdbcAggregationRepository}}), with a recovery
interval of 100 ms, {{maximumRedeliveries=2}} and {{deadLetterUri=mock:dead}}:
{code:java}
from("direct:start")
.aggregate(header("id"),
strategy).aggregationRepository(repository).completionSize(2)
.to("mock:result")
.throwException(new IllegalArgumentException("fails every time"));
// send A and B with header id=0; the confirm of A+B after it was moved to
mock:dead throws once
{code}
h3. Actual (main)
* The confirm throws once: {{mock:dead}} receives {{A+B}} twice.
* The confirm returns {{false}} once: {{mock:dead}} receives {{A+B}} twice.
* Two failing groups A+B and C+D, the confirm of A+B keeps throwing:
{{mock:dead}} receives {{A+B}} at every pass (about 100 times in 10 seconds),
with {{Error during recover task. This exception is ignored.}} each time, and
C+D is never moved to the dead letter channel nor confirmed.
On 4.18.x, whose {{RecoverTask.run}} does not have the catch of CAMEL-24141,
the exception of that confirm ends the scheduled recover task instead: no
exchange is recovered any more until the route is restarted.
h3. Expected
Each exchange is moved to the dead letter channel once. A confirm that throws
or returns {{false}} is retried by the recover task as a confirm, as in
{{onComplete}}, and the pass goes on with the other ids.
h3. Suggested fix
After the exchange was moved to the dead letter channel, call
{{confirmWithResult}}; when it throws (except a {{VirtualMachineError}}, as
elsewhere in the class since CAMEL-26881) or returns {{false}}, log and add the
id to {{unconfirmedCompleteExchanges}}, so the next passes call
{{retryConfirm}} for it. Remove its redelivery state, as {{onComplete}} does.
h3. Related
* CAMEL-26995: the same for the confirm in {{onComplete}} and {{retryConfirm}}.
* CAMEL-17613: a confirm that returns {{false}} is retried, not redelivered.
_Claude Code on behalf of Croway_
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)