shashank created CAMEL-26967:
--------------------------------
Summary: camel-kafka - breakOnFirstError with synchronous manual
commits: a commitSync timeout in the break skips the failed record and the rest
of the poll
Key: CAMEL-26967
URL: https://issues.apache.org/jira/browse/CAMEL-26967
Project: Camel
Issue Type: Bug
Components: camel-kafka
Reporter: shashank
Assignee: shashank
Found by the Kafka DST world of CAMEL-26947 (PR #29992, branch
{{camel-kafka-dst-world}}): with {{-Ddst.kafka.syncBreakOnFirstError=true}},
seed 95 and 5 more of 1000 seeds fail the invariant "a committed offset never
passes a record the route has not finished with". Reproduced outside the DST
harness with a plain unit test (below).
h3. What happens
With {{breakOnFirstError=true}} and synchronous manual commits
({{allowManualCommit=true}},
{{kafkaManualCommitFactory=DefaultKafkaManualCommitFactory}}, so the
{{SyncCommitManager}}), a record that fails makes
{{KafkaRecordStreamingProcessor.processException}} commit the records routed
before it ({{commitManager.commit(partition)}}, a {{commitSync(offsets,
commitTimeoutMs)}}) and return the break, so {{KafkaFetchRecords}} reconnects
and the new Kafka consumer polls again from the committed offset.
{{commitSync}} can throw: a {{TimeoutException}} when the timeout expires
before the commit completed (the caller does not know whether it was applied),
or {{CommitFailedException}} / {{RebalanceInProgressException}}. The exception
leaves {{processPolledRecords}} and reaches the catch of
{{KafkaFetchRecords.startPolling}}, which hands it to the poll exception
strategy. With {{pollOnError}} {{ERROR_HANDLER}} (the default), {{DISCARD}} or
{{RETRY}} the fetcher goes on polling with the same Kafka consumer, whose
position is already after the whole batch: the failed record and the records
after it in that poll are never routed, and the next manual commit (of a later
record) commits past them. Only {{RECONNECT}} (and {{STOP}}) avoid it.
h3. Expected
The option documentation of {{breakOnFirstError}}: "Using either the
SyncCommitManager or the AsyncCommitManager, the consumer will seek back to the
offset of the message that caused a failure, and then re-attempt to process
this message." {{KafkaConsumer#commitSync(Map, Duration)}} documents the
{{TimeoutException}} ("if the timeout expires before successful completion of
the offset commit") as a normal outcome. A failed commit in the break must not
turn the break into a skip: the consumer should still reconnect and poll from
the committed offset. If the commit was applied, that is the failed record; if
not, an earlier offset, and records are routed again (at-least-once), never
skipped.
h3. Reproduction
New {{KafkaConsumerBreakOnFirstErrorCommitTimeoutTest}} (a {{MockConsumer}} on
a log, a new Kafka consumer starts from the committed offset, as in
{{KafkaConsumerBreakOnFirstErrorPausedTest}}): records a..e, the route commits
each record manually, c fails once, the {{commitSync}} of the break throws
{{TimeoutException}} (not applied), and f is stored after it. On main with
{{pollOnError}} ERROR_HANDLER, DISCARD and RETRY the route gets {{[a, b, f]}}
and the group commits offset 6, so c, d and e are lost; RECONNECT (control)
gets all six.
h3. Fix
In {{processException}}, catch a {{KafkaException}} of the commit (not
{{WakeupException}} / {{InterruptException}}), report it to the exception
handler of the consumer, and return the break as before, so the consumer
reconnects and polls again from the committed offset. So that commit failure no
longer goes through {{pollOnError}} (it reached the route's error handler
through ERROR_HANDLER, and STOP stopped the consumer). Branch
{{camel-kafka-sync-commit-timeout-break-on-first-error}}.
_Filed with Claude Code on behalf of allthingssecurity._
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)