Federico Mariani created CAMEL-26993:
----------------------------------------

             Summary: camel-kafka - with manual commits, a rebalance or a stop 
commits records the route did not commit
                 Key: CAMEL-26993
                 URL: https://issues.apache.org/jira/browse/CAMEL-26993
             Project: Camel
          Issue Type: Bug
          Components: camel-kafka
            Reporter: Federico Mariani


With {{allowManualCommit=true}} and the {{DefaultKafkaManualCommitFactory}} or 
{{DefaultKafkaManualAsyncCommitFactory}}, 
{{KafkaRecordStreamingProcessor.processExchange}} records the offset of every 
processed record in the {{OffsetCache}} of the commit manager 
({{SyncCommitManager}}, {{AsyncCommitManager}}), whether the route called 
{{KafkaManualCommit.commit()}} or not. When a partition is revoked 
({{ClassicRebalanceListener.onPartitionsRevoked}}, also on stop through the 
unsubscribe), {{commit(partition)}} commits the cached offset: the last 
processed record, not the last one the route committed.

So records that the route still holds, for example in an aggregation group that 
is not complete, are committed on a rebalance or a stop although the route 
never committed them. If the held exchanges are then lost (the application 
stops or fails before the group completes, with an in-memory aggregation 
repository), these records are not consumed again. Since CAMEL-23997 the commit 
after each poll is skipped with manual commits, but the revoke and stop paths 
still use the cache.

Changing this is a design decision: {{breakOnFirstError}} relies on the same 
cache ({{KafkaRecordStreamingProcessor.processException}} calls 
{{commit(partition)}} to commit the last record processed before the failed 
one, so that the failed one is retried), so caching only the offsets given to 
{{commit()}} changes that behaviour too.

_Claude Code on behalf of Federico Mariani_



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

Reply via email to