[ 
https://issues.apache.org/jira/browse/CAMEL-27317?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on CAMEL-27317 started by shashank.
----------------------------------------
> camel-kafka - an offset kept in the OffsetCache after a partition was lost, 
> or after its commit failed on a revoke, is committed later and moves back the 
> committed offset
> --------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: CAMEL-27317
>                 URL: https://issues.apache.org/jira/browse/CAMEL-27317
>             Project: Camel
>          Issue Type: Bug
>          Components: camel-kafka
>            Reporter: shashank
>            Assignee: shashank
>            Priority: Major
>
> h3. What happens
> The Sync, Async and CommitToOffset commit managers keep the last processed 
> offset of a partition in an {{OffsetCache}} until it is committed.
> * {{ClassicRebalanceListener}} and {{ResumeRebalanceListener}} do not 
> override {{onPartitionsLost}}, so Kafka's default calls 
> {{onPartitionsRevoked}}, which commits the cached offset of a partition that 
> another member owns already. The commit fails (stale generation) and the 
> entry stays; with an {{offsetRepository}} the write to the repository is not 
> refused, and the repository goes back.
> * A revoke whose commit fails (timeout, failed async commit) keeps the entry 
> too.
> * {{OffsetCache.removeCommittedEntry}} removes the entry on the callback of 
> any successful commit, also an older one whose callback runs after a newer 
> offset was recorded (async callbacks run in a later {{poll()}} or 
> {{commitAsync()}}).
> When the partition comes back after another member committed further, and is 
> revoked again (or {{breakOnFirstError}} commits it) before a record of it 
> updated the entry, the stale offset is committed: the committed offset goes 
> back and the other member's records are read again. In the third case a 
> failed commit of the newer offset leaves nothing to commit on the revoke, and 
> its records are read again.
> This is not the question of CAMEL-26993 (what is recorded and committed for a 
> partition the consumer still owns), only of when an entry is removed.
> h3. How to reproduce
> {{CommitManagerRebalanceTest}} (camel-kafka, {{MockConsumer}} with group 
> offsets, refused commits and deferred async callbacks, the real 
> {{ClassicRebalanceListener}}): the partition is lost or its revoke commit 
> fails, another member commits 10, the partition comes back and is revoked 
> again: the committed offset goes back to 5.
> Found with a TLA+ model of the commit managers.
> h3. Fix
> New default method {{CommitManager.discardRecordedOffset}}. The rebalance 
> listeners override {{onPartitionsLost}} to forget the offsets without 
> committing, and forget the offsets of revoked partitions after the commit 
> whether it succeeded or not. {{OffsetCache}} removes an entry after a commit 
> only if the committed offset covers it.
> _Filed with Claude Code on behalf of allthingssecurity._



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

Reply via email to