[
https://issues.apache.org/jira/browse/KAFKA-10249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
John Roesler updated KAFKA-10249:
---------------------------------
Fix Version/s: 2.6.0
> In-memory stores are skipped when checkpointing but not skipped when reading
> the checkpoint
> -------------------------------------------------------------------------------------------
>
> Key: KAFKA-10249
> URL: https://issues.apache.org/jira/browse/KAFKA-10249
> Project: Kafka
> Issue Type: Bug
> Components: streams
> Affects Versions: 2.6.0
> Reporter: Sophie Blee-Goldman
> Assignee: Sophie Blee-Goldman
> Priority: Major
> Fix For: 2.6.0
>
>
> As the title suggests, offsets for in-memory stores (including the
> suppression buffer) are not written to the checkpoint file. However, when
> reading from the checkpoint file during task initialization, we do not check
> StateStore#persistent. We attempt to look up the offsets for in-memory stores
> in the checkpoint file, and obviously do not find them.
> With eos we have to conclude that the existing state is dirty and thus throw
> a TaskCorruptedException. So pretty much any task with in-memory state will
> always hit this exception when reinitializing from the checkpoint, forcing it
> to clear the entire state directory and build up all of its state again from
> scratch (both persistent and in-memory).
> This is especially unfortunate for KIP-441, as we will hit this any time a
> task is moved from one thread to another.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
