allthingssecurity opened a new pull request, #30132:
URL: https://github.com/apache/camel/pull/30132

   # Description
   
   [CAMEL-26967](https://issues.apache.org/jira/browse/CAMEL-26967)
   
   Found by the Kafka DST world of #29992 (CAMEL-26947), which leaves this case 
out of its default set: with `-Ddst.kafka.syncBreakOnFirstError=true`, seed 95 
and 5 more of 1000 seeds commit an offset past a record the route never 
finished. Reproduced here with a plain unit test, outside the DST harness.
   
   With `breakOnFirstError=true` and synchronous manual commits 
(`allowManualCommit=true` with `DefaultKafkaManualCommitFactory`, so the 
`SyncCommitManager`), a failed record makes 
`KafkaRecordStreamingProcessor.processException` commit the records routed 
before it and return the break; `KafkaFetchRecords` then reconnects, and the 
new Kafka consumer polls again from the committed offset. That commit is a 
`commitSync` with `commitTimeoutMs`, and it can throw: `TimeoutException` when 
the timeout expires first (the caller does not know whether the commit was 
applied), or `CommitFailedException` / `RebalanceInProgressException`. The 
exception left the batch and reached the catch of 
`KafkaFetchRecords.startPolling`, and with `pollOnError` `ERROR_HANDLER` (the 
default), `DISCARD` or `RETRY` the fetcher went on polling with the same Kafka 
consumer. Its position was already after the whole poll, so the failed record 
and the records after it in that poll were never routed, and the next manual
  commit of a later record committed past them.
   
   The option documentation says that with the sync or async commit manager 
"the consumer will seek back to the offset of the message that caused a 
failure, and then re-attempt to process this message", and 
`KafkaConsumer#commitSync(Map, Duration)` documents the `TimeoutException` as 
one of its outcomes.
   
   Change, in `KafkaRecordStreamingProcessor.processException`: a 
`KafkaException` of that commit is reported to the exception handler of the 
consumer and the break is returned as before, so the consumer reconnects and 
polls from the committed offset. If the commit was applied, that is the failed 
record; if not, an earlier offset, and some records are routed again 
(at-least-once) instead of skipped. `WakeupException` and `InterruptException` 
still go to the poll loop as before.
   
   Visible change: that commit failure no longer goes through `pollOnError`. It 
reaches the exception handler of the consumer (logged by default, routed with 
`bridgeErrorHandler=true`) instead of the error handler of the route through 
the `ERROR_HANDLER` strategy, and with `pollOnError=STOP` the consumer 
reconnects (as the break does) instead of stopping. Both only for a failed 
commit in the break.
   
   Not changed: the commit of `KafkaConsumerListener.afterProcess` (the 
resumable consumer with a resume strategy) can fail the same way and still goes 
through `pollOnError`; I can open a follow-up JIRA. The commit at the end of a 
partition runs only without manual commits, where the commit manager makes no 
Kafka commit (no-op, or the offset repository).
   
   The batching consumer is not affected: its break calls 
`commitManager.commit()`, which does nothing with manual commits, and with 
auto-commit the commit runs in an on-completion.
   
   Test, new `KafkaConsumerBreakOnFirstErrorCommitTimeoutTest` (a 
`MockConsumer` on a log, a new Kafka consumer starts from the committed offset 
of the group, as in `KafkaConsumerBreakOnFirstErrorPausedTest`): records a to 
e, the route commits each record manually, c fails once, the `commitSync` of 
the break times out (not applied), and f is stored after it. For `pollOnError` 
`ERROR_HANDLER`, `DISCARD` and `RETRY` the route must get a to f in order; 
`RECONNECT` is a control that passes without the change too.
   
   `testFailedCommitOfTheBreakIsReportedToTheExceptionHandler` checks the 
report (with `RECONNECT`, which polls the record again on main too).
   
   Without the change the three cases and the report test fail (the route gets 
`[a, b, f]` and the group commits offset 6; nothing reaches the exception 
handler); with it all five pass, and the camel-kafka unit tests pass (340 run, 
0 failures, 1 skipped). With the change merged locally into the world of 
#29992, `-Ddst.kafka.syncBreakOnFirstError=true` passes 1000 of 1000 seeds (6 
fail without it), and the default set passes 1000 of 1000. The ITs need a Kafka 
container and were not run locally (CI runs them). The local offline build used 
the cached kafka-clients 4.3.1.
   
   No documentation or upgrade guide change: the consumer now does what the 
option already describes.
   
   Backport: camel-4.18.x has the same `processException`; the change applies 
there as is (records are lost, so it may be worth it).
   
   # Target
   
   - [x] I checked that the commit is targeting the correct branch (Camel 4 
uses the `main` branch)
   
   # Tracking
   - [x] If this is a large change, bug fix, or code improvement, I checked 
there is a [JIRA issue](https://issues.apache.org/jira/browse/CAMEL) filed for 
the change (usually before you start working on it): 
[CAMEL-26967](https://issues.apache.org/jira/browse/CAMEL-26967).
   
   # Apache Camel coding standards and style
   
   - [x] I checked that each commit in the pull request has a meaningful 
subject line and body.
   - [ ] I have run `mvn clean install -DskipTests` locally from root folder 
and I have committed all auto-generated changes.
     (I built camel-kafka and the core modules with the formatter and 
import-sort plugins and ran the camel-kafka unit tests. I did not run the full 
root build.)
   
   # AI-assisted contributions
   
   - [x] If this PR includes AI-generated code, commits have proper 
co-authorship attribution (e.g., `Co-authored-by` trailers) and the PR 
description identifies the AI tool used.
     This PR was prepared with Claude Code (Claude Opus 5.5). The commit 
carries a `Co-Authored-By` trailer.
   
   _Claude Code on behalf of allthingssecurity_
   
   🤖 Generated with [Claude Code](https://claude.com/claude-code)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to