Federico Mariani created CAMEL-26993:
----------------------------------------
Summary: camel-kafka - with manual commits, a rebalance or a stop
commits records the route did not commit
Key: CAMEL-26993
URL: https://issues.apache.org/jira/browse/CAMEL-26993
Project: Camel
Issue Type: Bug
Components: camel-kafka
Reporter: Federico Mariani
With {{allowManualCommit=true}} and the {{DefaultKafkaManualCommitFactory}} or
{{DefaultKafkaManualAsyncCommitFactory}},
{{KafkaRecordStreamingProcessor.processExchange}} records the offset of every
processed record in the {{OffsetCache}} of the commit manager
({{SyncCommitManager}}, {{AsyncCommitManager}}), whether the route called
{{KafkaManualCommit.commit()}} or not. When a partition is revoked
({{ClassicRebalanceListener.onPartitionsRevoked}}, also on stop through the
unsubscribe), {{commit(partition)}} commits the cached offset: the last
processed record, not the last one the route committed.
So records that the route still holds, for example in an aggregation group that
is not complete, are committed on a rebalance or a stop although the route
never committed them. If the held exchanges are then lost (the application
stops or fails before the group completes, with an in-memory aggregation
repository), these records are not consumed again. Since CAMEL-23997 the commit
after each poll is skipped with manual commits, but the revoke and stop paths
still use the cache.
Changing this is a design decision: {{breakOnFirstError}} relies on the same
cache ({{KafkaRecordStreamingProcessor.processException}} calls
{{commit(partition)}} to commit the last record processed before the failed
one, so that the failed one is retried), so caching only the offsets given to
{{commit()}} changes that behaviour too.
_Claude Code on behalf of Federico Mariani_
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)