[
https://issues.apache.org/jira/browse/KAFKA-15625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17813099#comment-17813099
]
Ayoub Omari commented on KAFKA-15625:
-------------------------------------
Hi [~cadonna], is this improvement ready for development ? I am interested in
taking it.
One question, do you have any idea about the delta we will be considering for
global state stores ?
I see that it is set to 10.000 records for task state stores. My understanding
is that flushing will only help when restoring the store from the same
instance, so that it reads less records from the source topic. So, I am
thinking it's a fair value ?
> Do not flush global state store at each commit
> ----------------------------------------------
>
> Key: KAFKA-15625
> URL: https://issues.apache.org/jira/browse/KAFKA-15625
> Project: Kafka
> Issue Type: Improvement
> Components: streams
> Reporter: Bruno Cadonna
> Priority: Major
>
> Global state stores are flushed at each commit. While that is not a big issue
> with at-least-once processing mode since the commit interval is by default
> 30s, it might become an issue with EOS where the commit interval is 200ms by
> default.
> One option would be to flush and checkpoint global state stores when the
> delta of the content exceeds a given threshold as we do for other stores. See
> https://github.com/apache/kafka/blob/a1f3c6d16061566a4f53c72a95e2679b8ee229e0/streams/src/main/java/org/apache/kafka/streams/processor/internals/AbstractTask.java#L97
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
