shashank created CAMEL-26967:
--------------------------------

             Summary: camel-kafka - breakOnFirstError with synchronous manual 
commits: a commitSync timeout in the break skips the failed record and the rest 
of the poll
                 Key: CAMEL-26967
                 URL: https://issues.apache.org/jira/browse/CAMEL-26967
             Project: Camel
          Issue Type: Bug
          Components: camel-kafka
            Reporter: shashank
            Assignee: shashank


Found by the Kafka DST world of CAMEL-26947 (PR #29992, branch 
{{camel-kafka-dst-world}}): with {{-Ddst.kafka.syncBreakOnFirstError=true}}, 
seed 95 and 5 more of 1000 seeds fail the invariant "a committed offset never 
passes a record the route has not finished with". Reproduced outside the DST 
harness with a plain unit test (below).

h3. What happens

With {{breakOnFirstError=true}} and synchronous manual commits 
({{allowManualCommit=true}}, 
{{kafkaManualCommitFactory=DefaultKafkaManualCommitFactory}}, so the 
{{SyncCommitManager}}), a record that fails makes 
{{KafkaRecordStreamingProcessor.processException}} commit the records routed 
before it ({{commitManager.commit(partition)}}, a {{commitSync(offsets, 
commitTimeoutMs)}}) and return the break, so {{KafkaFetchRecords}} reconnects 
and the new Kafka consumer polls again from the committed offset.

{{commitSync}} can throw: a {{TimeoutException}} when the timeout expires 
before the commit completed (the caller does not know whether it was applied), 
or {{CommitFailedException}} / {{RebalanceInProgressException}}. The exception 
leaves {{processPolledRecords}} and reaches the catch of 
{{KafkaFetchRecords.startPolling}}, which hands it to the poll exception 
strategy. With {{pollOnError}} {{ERROR_HANDLER}} (the default), {{DISCARD}} or 
{{RETRY}} the fetcher goes on polling with the same Kafka consumer, whose 
position is already after the whole batch: the failed record and the records 
after it in that poll are never routed, and the next manual commit (of a later 
record) commits past them. Only {{RECONNECT}} (and {{STOP}}) avoid it.

h3. Expected

The option documentation of {{breakOnFirstError}}: "Using either the 
SyncCommitManager or the AsyncCommitManager, the consumer will seek back to the 
offset of the message that caused a failure, and then re-attempt to process 
this message." {{KafkaConsumer#commitSync(Map, Duration)}} documents the 
{{TimeoutException}} ("if the timeout expires before successful completion of 
the offset commit") as a normal outcome. A failed commit in the break must not 
turn the break into a skip: the consumer should still reconnect and poll from 
the committed offset. If the commit was applied, that is the failed record; if 
not, an earlier offset, and records are routed again (at-least-once), never 
skipped.

h3. Reproduction

New {{KafkaConsumerBreakOnFirstErrorCommitTimeoutTest}} (a {{MockConsumer}} on 
a log, a new Kafka consumer starts from the committed offset, as in 
{{KafkaConsumerBreakOnFirstErrorPausedTest}}): records a..e, the route commits 
each record manually, c fails once, the {{commitSync}} of the break throws 
{{TimeoutException}} (not applied), and f is stored after it. On main with 
{{pollOnError}} ERROR_HANDLER, DISCARD and RETRY the route gets {{[a, b, f]}} 
and the group commits offset 6, so c, d and e are lost; RECONNECT (control) 
gets all six.

h3. Fix

In {{processException}}, catch a {{KafkaException}} of the commit (not 
{{WakeupException}} / {{InterruptException}}), report it to the exception 
handler of the consumer, and return the break as before, so the consumer 
reconnects and polls again from the committed offset. So that commit failure no 
longer goes through {{pollOnError}} (it reached the route's error handler 
through ERROR_HANDLER, and STOP stopped the consumer). Branch 
{{camel-kafka-sync-commit-timeout-break-on-first-error}}.

_Filed with Claude Code on behalf of allthingssecurity._



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

Reply via email to