[
https://issues.apache.org/jira/browse/CAMEL-14935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17119740#comment-17119740
]
Darius Cooper commented on CAMEL-14935:
---------------------------------------
[~Chris McCarthy] Very clear, thank you.
If you can review [this PR on my personal
fork|[https://github.com/DariusX/camel/pull/1/commits/9df63769d4ed9ffcc0f2d7aa6bd04955ccf15e2f]],
and find matches what you were thinking, that'll be helpful.
As a next step, I'll work on sending it back into the Camel base repo.
Thx.
> KafkaConsumer commits old offset values in a failure scenario causing message
> replays and offset reset error
> ------------------------------------------------------------------------------------------------------------
>
> Key: CAMEL-14935
> URL: https://issues.apache.org/jira/browse/CAMEL-14935
> Project: Camel
> Issue Type: Bug
> Components: camel-kafka
> Affects Versions: 2.24.0
> Reporter: Chris McCarthy
> Priority: Major
> Fix For: 3.x
>
>
> We are experiencing unexpected offset reset errors occasionally, as well as
> occasional replay of messages (without an offset reset error).
> The cause seems to be a failed commit on rebalance, leaving an old value in
> the hashMap used to store the latest processed offset for a partition. This
> old value is then re-read and re-committed across rebalances in certain
> situations.
> Our relevant configuration details are:
> autoCommitEnable=false
> allowManualCommit=true
> autoOffsetReset=earliest
> It seems when the KafkaConsumer experiences an Exception committing the
> offset (CommitFailedException) upon a rebalance, this leaves the old offset
> value in the lastProcessedOffset hashMap.
> A subsequent rebalance that assigns the same partition to the same consumer,
> that then thereafter experiences another rebalance (before any messages have
> been processed successfully as this will over write the invalid value and
> self correct the problem) will commit this old offset again. This offset may
> be very old if there have been many rebalances in between the original
> rebalance that failed to commit its offset.
> If the old offset is beyond the retention period and the message is no longer
> available the outcome is an offset reset error. If the offset is within the
> retention period all messages are replayed from that offset without an error
> being thrown.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
