[
https://issues.apache.org/jira/browse/KAFKA-6121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16285975#comment-16285975
]
ASF GitHub Bot commented on KAFKA-6121:
---------------------------------------
Github user asfgit closed the pull request at:
https://github.com/apache/kafka/pull/4215
> Restore and global consumer should not use auto.offset.reset
> ------------------------------------------------------------
>
> Key: KAFKA-6121
> URL: https://issues.apache.org/jira/browse/KAFKA-6121
> Project: Kafka
> Issue Type: Bug
> Components: streams
> Affects Versions: 1.0.0
> Reporter: Matthias J. Sax
> Assignee: Matthias J. Sax
> Labels: streams-exception-handling
> Fix For: 1.1.0, 1.0.1
>
>
> Streams uses three different consumers internally. The main consumer, as well
> as one consumer for state restore (restore consumer, also used by
> StandbyTasks) and a consumer for global state (used by GlobalThreadThread).
> While main consumer handles InvalidOffsetException correctly, restore and
> global consumer don't. Currently, they rely on auto.offset.reset with default
> value "latest" -- thus, if there is an InvalidOffsetException we just jump to
> the end of the changelog topic instead of proper handler this case.
> An InvalidOffsetException can occur for two cases:
> # An Kafka Streams application is offline for some time and on restart it
> reads it local offset file. This offset file might contain offsets that are
> not valid anymore as the log got compacted in between.
> # Even if we have valid offset and we do a seek, log compaction can actually
> tick an in the background at any point and could make our offset invalid --
> this is a rather rare race conditions but we need to handle it anyway
> For both cases, we can apply the same strategy: wipe out the local RocksDB,
> seekToBeginning, and recreate the store from scratch. Thus, we need to set
> auto.offset.reset to "none" for each consumer and handle
> InvalidOffsetException that might be throw by poll() or position().
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
