shashank created CAMEL-27282:
--------------------------------
Summary: camel-kafka - with offsetRepository a consumer that gets
a partition back writes its old offset when the partition is revoked again
Key: CAMEL-27282
URL: https://issues.apache.org/jira/browse/CAMEL-27282
Project: Camel
Issue Type: Bug
Components: camel-kafka
Reporter: shashank
Assignee: shashank
h3. What happens
With {{offsetRepository}} (and no manual commits) each fetcher uses a
{{CommitToOffsetManager}}. Its {{commit(partition)}} saves the last offset the
fetcher processed in that partition but never removes it from its
{{OffsetCache}} (the Sync and Async commit managers remove it after a commit).
{{ClassicRebalanceListener.onPartitionsRevoked}} (and
{{ResumeRebalanceListener}}) commits every revoked partition. The repository is
shared by all the consumers of the endpoint ({{consumersCount}}) and can be
shared by several applications.
# Consumer A processes partition P up to offset 100 and saves it; a rebalance
moves P to consumer B.
# B resumes at 101, processes up to 500 and saves 500.
# P comes back to A (B leaves, reconnects, scale down; several rebalances in a
row are common during a rolling restart): {{OffsetPartitionAssignmentAdapter}}
seeks to 501.
# Another rebalance revokes P before A polled a record of it: the revoke saves
A's cached 100. The repository goes back from 500 to 100, and the next owner
processes 101..500 again.
No message is lost, but every record processed since A last owned P is
processed again.
h3. How to reproduce
{{CommitToOffsetManagerRevokeTest}}: a {{MockConsumer}}, a
{{MemoryStateRepository}}, a {{CommitToOffsetManager}} and the
{{ClassicRebalanceListener}}: save 100, revoke, set the repository to 500 (the
other consumer), assign (position 501), revoke: the repository holds 100.
h3. Fix
{{CommitToOffsetManager.commit(partition)}} removes the offset from its cache
after saving it, as the other commit managers do.
_Filed with Claude Code on behalf of allthingssecurity._
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)