[
https://issues.apache.org/jira/browse/CAMEL-14935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris McCarthy updated CAMEL-14935:
-----------------------------------
Summary: KafkaConsumer commits old offset values in a failure scenario
causing message replays and offset reset error (was: KafkaConsumer commits old
offset values in certain failure case causing message replays and offset reset
error)
> KafkaConsumer commits old offset values in a failure scenario causing message
> replays and offset reset error
> ------------------------------------------------------------------------------------------------------------
>
> Key: CAMEL-14935
> URL: https://issues.apache.org/jira/browse/CAMEL-14935
> Project: Camel
> Issue Type: Bug
> Components: camel-kafka
> Affects Versions: 2.24.0
> Reporter: Chris McCarthy
> Priority: Major
>
> We are getting unexpected offset reset errors occasionally.
> The cause seems to be a failed commit on rebalance, leaving an old value in
> the hashMap that is then re-read and re-committed across rebalances in
> certain situations.
> Our relevant configuration details are:
> autoCommitEnable=false
> allowManualCommit=true
> autoOffsetReset=earliest
> It seems when the KafkaConsumer experiences an Exception committing the
> offset upon a rebalance, this leaves the old offset value in the
> lastProcessedOffset hashMap.
> A subsequent rebalance that assigns the same partition to the same consumer,
> that then thereafter experiences another rebalance (before any messages have
> been processed successfully as this will over write the invalid value and
> self correct the problem) will commit this old offset again. This offset may
> be very old if there have been many rebalances in between the original
> rebalance that failed to commit its offset.
> If the old offset is beyond the retention period and the message is no longer
> available the outcome is an offset reset error. If the offset is within the
> retention period all messages are replayed from that offset without an error
> being thrown.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
