[
https://issues.apache.org/jira/browse/KAFKA-3184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17096718#comment-17096718
]
Nikolay Izhikov edited comment on KAFKA-3184 at 4/30/20, 4:32 PM:
------------------------------------------------------------------
Hello, [~guozhang]
I've prepared PR with the simplest implementation of an in-memory state store
checkpointer.
If persistent mode is enabled then:
* checkpoint thread started on KeyValueStore#init
* every InMemoryKeyValueStore#COUNT_FLUSH_TO_STORE flush execution copy of
the InMemoryKeyValueStore#map passed to checkpoint thread.
* checkpoint thread persists data every time it sees a new instance of
InMemoryKeyValueStore#map.
* persisted data are loaded on KeyValueStore#init.
Can you, please, take a look.
https://github.com/apache/kafka/pull/8592
was (Author: nizhikov):
Hello, [~guozhang]
I've prepared PR with the simplest implementation of an in-memory state store
checkpointer.
If persistent mode is enabled then:
* checkpoint thread started on KeyValueStore#init
* every InMemoryKeyValueStore#COUNT_FLUSH_TO_STORE flush execution copy of
the InMemoryKeyValueStore#map passed to checkpoint thread.
* checkpoint thread persists data every time it sees a new instance of
InMemoryKeyValueStore#map.
* persisted data are loaded on KeyValueStore#init.
Can you, please, take a look.
> Add Checkpoint for In-memory State Store
> ----------------------------------------
>
> Key: KAFKA-3184
> URL: https://issues.apache.org/jira/browse/KAFKA-3184
> Project: Kafka
> Issue Type: Improvement
> Components: streams
> Reporter: Guozhang Wang
> Assignee: Nikolay Izhikov
> Priority: Major
> Labels: user-experience
>
> Currently Kafka Streams does not make a checkpoint of the persistent state
> store upon committing, which would be expensive since it is "stopping the
> world" and write on disks: for example, RocksDB would require you to copy the
> file directory to make a copy naively.
> However, for in-memory stores checkpointing maybe doable in an asynchronous
> manner hence it can be done quickly. And the benefit of having intermediate
> checkpoint is to avoid restoring from scratch if standby tasks are not
> present.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
