Federico Mariani created CAMEL-26652:
----------------------------------------
Summary: camel-git - the commit, tag and branch consumers mark a
failed exchange as consumed and never deliver it again
Key: CAMEL-26652
URL: https://issues.apache.org/jira/browse/CAMEL-26652
Project: Camel
Issue Type: Bug
Components: camel-git
Affects Versions: 4.14.9, 4.18.5, 4.23.0, 4.22.2
Reporter: Federico Mariani
h3. Problem
The three git consumers ({{type=commit}}, {{type=tag}}, {{type=branch}})
remember an item as consumed after routing it, whether the route succeeded or
not. A commit, tag or branch whose exchange failed is never offered again: with
the default error handler the message is lost, and only the error handler's log
line records it.
*
{{components/camel-git/src/main/java/org/apache/camel/component/git/consumer/AbstractGitConsumer.java}}
lines 87-89 (main, commit 7978db3d42dd) call {{onPreProcessed}}, then
{{getProcessor().process(exchange)}}, then {{onProcessed(exchange, value)}}
unconditionally.
* A route failure does not make {{process()}} throw: the exception stays on the
exchange. {{processBatch}} never reads {{exchange.getException()}} and never
calls the consumer's {{ExceptionHandler}}.
* {{onProcessed}} adds the item to the consumed list:
{{GitCommitConsumer.java}} line 82, {{GitTagConsumer.java}} line 63,
{{GitBranchConsumer.java}} line 64.
{code:java}
Object value = onPreProcessed(exchange);
getProcessor().process(exchange);
onProcessed(exchange, value); // also when the exchange failed
{code}
This predates the batch consumer. Up to 4.8 the commit consumer did
{{getProcessor().process(e); commitsConsumed.add(commit.getId());}}. The same
code is in camel-4.14.9, 4.18.5, 4.22.2, 4.23.0 and main (checked in the
source).
h3. How to reproduce
Measured on main with a JUnit test:
* A repository created with JGit in a temporary directory, with 3 commits on
{{master}}.
* The route {{from("git:<dir>?type=commit&branchName=master")}}, whose
processor counts the attempts per {{CamelGitCommitId}} and then throws
{{IllegalStateException}}. The default error handler is used.
* The test waits until the 3 commits were attempted, then watches for 3 more
seconds, about 6 polls.
* Control: a 4th commit added afterwards is picked up, so the consumer is still
polling.
Outcome:
* Each of the 4 commits was attempted exactly once.
* The log shows {{DefaultErrorHandler - Failed delivery for ... Exhausted after
delivery attempt: 1}} once per commit.
* No commit was offered again.
h3. Proposed change
* After {{process()}}, read the outcome. If the exchange failed (an exception
that is not handled), do not call {{onProcessed}}, and report the failure
through {{getExceptionHandler().handleException(...)}}. This is what
camel-mongodb (CAMEL-25024) and camel-couchbase (CAMEL-25171) do now.
* Read the outcome before the exchange is given back. The exchanges are created
with {{createExchange(true)}}, so with the pooled exchange factory they can be
reset when their unit of work is done. Create them with autoRelease=false and
release them after the outcome has been read, as the camel-mongodb fix does.
* Risk: a commit that always fails is then routed again on every poll. The poll
interval cannot be raised today (see CAMEL-26651), so that one should be fixed
first or together. A route that wants to drop a failing commit can still do so
with a dead letter channel or a handled {{onException}}.
* Alternative, if a retry on the next poll is not wanted: keep marking the item
consumed, but report the failure through the {{ExceptionHandler}}. The failure
is then visible, and {{bridgeErrorHandler}} can route it.
h3. Impact / upgrade
* This is a behaviour change: a failed commit, tag or branch is delivered again
on the next poll instead of being skipped for good. It needs an entry in the
4.24 upgrade guide.
* Routes with a dead letter channel or a handled {{onException}} are not
affected, because their exchanges complete without an exception.
* All three consumer types go through {{AbstractGitConsumer.processBatch}}.
h3. Related
* See also CAMEL-26651: the scheduler and error handler options of the git
consumer are ignored.
* CAMEL-19665 (commits 51f270663f86 and 83ab5f1536c5) turned the git consumers
into batch consumers and kept these semantics.
* CAMEL-25024 (camel-mongodb) and CAMEL-25171 (camel-couchbase) fixed the same
kind of bug.
* CAMEL-26192 changed {{CamelGitCommitId}} to the sha, which is the value
{{onPreProcessed}} keeps.
_Claude Code on behalf of Federico Mariani_
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
