shashank created CAMEL-27282:
--------------------------------

             Summary: camel-kafka - with offsetRepository a consumer that gets 
a partition back writes its old offset when the partition is revoked again
                 Key: CAMEL-27282
                 URL: https://issues.apache.org/jira/browse/CAMEL-27282
             Project: Camel
          Issue Type: Bug
          Components: camel-kafka
            Reporter: shashank
            Assignee: shashank


h3. What happens

With {{offsetRepository}} (and no manual commits) each fetcher uses a 
{{CommitToOffsetManager}}. Its {{commit(partition)}} saves the last offset the 
fetcher processed in that partition but never removes it from its 
{{OffsetCache}} (the Sync and Async commit managers remove it after a commit). 
{{ClassicRebalanceListener.onPartitionsRevoked}} (and 
{{ResumeRebalanceListener}}) commits every revoked partition. The repository is 
shared by all the consumers of the endpoint ({{consumersCount}}) and can be 
shared by several applications.

# Consumer A processes partition P up to offset 100 and saves it; a rebalance 
moves P to consumer B.
# B resumes at 101, processes up to 500 and saves 500.
# P comes back to A (B leaves, reconnects, scale down; several rebalances in a 
row are common during a rolling restart): {{OffsetPartitionAssignmentAdapter}} 
seeks to 501.
# Another rebalance revokes P before A polled a record of it: the revoke saves 
A's cached 100. The repository goes back from 500 to 100, and the next owner 
processes 101..500 again.

No message is lost, but every record processed since A last owned P is 
processed again.

h3. How to reproduce

{{CommitToOffsetManagerRevokeTest}}: a {{MockConsumer}}, a 
{{MemoryStateRepository}}, a {{CommitToOffsetManager}} and the 
{{ClassicRebalanceListener}}: save 100, revoke, set the repository to 500 (the 
other consumer), assign (position 501), revoke: the repository holds 100.

h3. Fix

{{CommitToOffsetManager.commit(partition)}} removes the offset from its cache 
after saving it, as the other commit managers do.

_Filed with Claude Code on behalf of allthingssecurity._



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

Reply via email to