[
https://issues.apache.org/jira/browse/CAMEL-27317?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on CAMEL-27317 started by shashank.
----------------------------------------
> camel-kafka - an offset kept in the OffsetCache after a partition was lost,
> or after its commit failed on a revoke, is committed later and moves back the
> committed offset
> --------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: CAMEL-27317
> URL: https://issues.apache.org/jira/browse/CAMEL-27317
> Project: Camel
> Issue Type: Bug
> Components: camel-kafka
> Reporter: shashank
> Assignee: shashank
> Priority: Major
>
> h3. What happens
> The Sync, Async and CommitToOffset commit managers keep the last processed
> offset of a partition in an {{OffsetCache}} until it is committed.
> * {{ClassicRebalanceListener}} and {{ResumeRebalanceListener}} do not
> override {{onPartitionsLost}}, so Kafka's default calls
> {{onPartitionsRevoked}}, which commits the cached offset of a partition that
> another member owns already. The commit fails (stale generation) and the
> entry stays; with an {{offsetRepository}} the write to the repository is not
> refused, and the repository goes back.
> * A revoke whose commit fails (timeout, failed async commit) keeps the entry
> too.
> * {{OffsetCache.removeCommittedEntry}} removes the entry on the callback of
> any successful commit, also an older one whose callback runs after a newer
> offset was recorded (async callbacks run in a later {{poll()}} or
> {{commitAsync()}}).
> When the partition comes back after another member committed further, and is
> revoked again (or {{breakOnFirstError}} commits it) before a record of it
> updated the entry, the stale offset is committed: the committed offset goes
> back and the other member's records are read again. In the third case a
> failed commit of the newer offset leaves nothing to commit on the revoke, and
> its records are read again.
> This is not the question of CAMEL-26993 (what is recorded and committed for a
> partition the consumer still owns), only of when an entry is removed.
> h3. How to reproduce
> {{CommitManagerRebalanceTest}} (camel-kafka, {{MockConsumer}} with group
> offsets, refused commits and deferred async callbacks, the real
> {{ClassicRebalanceListener}}): the partition is lost or its revoke commit
> fails, another member commits 10, the partition comes back and is revoked
> again: the committed offset goes back to 5.
> Found with a TLA+ model of the commit managers.
> h3. Fix
> New default method {{CommitManager.discardRecordedOffset}}. The rebalance
> listeners override {{onPartitionsLost}} to forget the offsets without
> committing, and forget the offsets of revoked partitions after the commit
> whether it succeeded or not. {{OffsetCache}} removes an entry after a commit
> only if the committed offset covers it.
> _Filed with Claude Code on behalf of allthingssecurity._
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)