shashank created CAMEL-27317:
--------------------------------
Summary: camel-kafka - an offset kept in the OffsetCache after a
partition was lost, or after its commit failed on a revoke, is committed later
and moves back the committed offset
Key: CAMEL-27317
URL: https://issues.apache.org/jira/browse/CAMEL-27317
Project: Camel
Issue Type: Bug
Components: camel-kafka
Reporter: shashank
Assignee: shashank
h3. What happens
The Sync, Async and CommitToOffset commit managers keep the last processed
offset of a partition in an {{OffsetCache}} until it is committed.
* {{ClassicRebalanceListener}} and {{ResumeRebalanceListener}} do not override
{{onPartitionsLost}}, so Kafka's default calls {{onPartitionsRevoked}}, which
commits the cached offset of a partition that another member owns already. The
commit fails (stale generation) and the entry stays; with an
{{offsetRepository}} the write to the repository is not refused, and the
repository goes back.
* A revoke whose commit fails (timeout, failed async commit) keeps the entry
too.
* {{OffsetCache.removeCommittedEntry}} removes the entry on the callback of any
successful commit, also an older one whose callback runs after a newer offset
was recorded (async callbacks run in a later {{poll()}} or {{commitAsync()}}).
When the partition comes back after another member committed further, and is
revoked again (or {{breakOnFirstError}} commits it) before a record of it
updated the entry, the stale offset is committed: the committed offset goes
back and the other member's records are read again. In the third case a failed
commit of the newer offset leaves nothing to commit on the revoke, and its
records are read again.
This is not the question of CAMEL-26993 (what is recorded and committed for a
partition the consumer still owns), only of when an entry is removed.
h3. How to reproduce
{{CommitManagerRebalanceTest}} (camel-kafka, {{MockConsumer}} with group
offsets, refused commits and deferred async callbacks, the real
{{ClassicRebalanceListener}}): the partition is lost or its revoke commit
fails, another member commits 10, the partition comes back and is revoked
again: the committed offset goes back to 5.
Found with a TLA+ model of the commit managers.
h3. Fix
New default method {{CommitManager.discardRecordedOffset}}. The rebalance
listeners override {{onPartitionsLost}} to forget the offsets without
committing, and forget the offsets of revoked partitions after the commit
whether it succeeded or not. {{OffsetCache}} removes an entry after a commit
only if the committed offset covers it.
_Filed with Claude Code on behalf of allthingssecurity._
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)