[
https://issues.apache.org/jira/browse/FLINK-27155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17562118#comment-17562118
]
Feifan Wang edited comment on FLINK-27155 at 7/4/22 10:01 AM:
--------------------------------------------------------------
Hi [~roman] and [~ym] , I submit a
[PR|https://github.com/apache/flink/pull/20152] . In that implement I set delay
time same with PERIODIC_MATERIALIZATION_INTERVAL , I'm not sure should we add a
configuration for that, WDYT ?
cc [~masteryhx]
was (Author: feifan wang):
Hi [~roman] and [~ym] , I submit a
[PR|https://github.com/apache/flink/pull/20152] . In that implement I set delay
time same with PERIODIC_MATERIALIZATION_INTERVAL , I'm not sure should we add a
configuration for that, WDYT ?
> Reduce multiple reads to the same Changelog file in the same taskmanager
> during restore
> ---------------------------------------------------------------------------------------
>
> Key: FLINK-27155
> URL: https://issues.apache.org/jira/browse/FLINK-27155
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / State Backends
> Reporter: Feifan Wang
> Assignee: Feifan Wang
> Priority: Major
> Labels: pull-request-available
> Fix For: 1.16.0
>
>
> h3. Background
> In the current implementation, State changes of different operators in the
> same taskmanager may be written to the same changelog file, which effectively
> reduces the number of files and requests to DFS.
> But on the other hand, the current implementation also reads the same
> changelog file multiple times on recovery. More specifically, the number of
> times the same changelog file is accessed is related to the number of
> ChangeSets contained in it. And since each read needs to skip the preceding
> bytes, this network traffic is also wasted.
> The result is a lot of unnecessary request to DFS when there are multiple
> slots and keyed state in the same taskmanager.
> h3. Proposal
> We can reduce multiple reads to the same changelog file in the same
> taskmanager during restore.
> One possible approach is to read the changelog file all at once and cache it
> in memory or local file for a period of time when reading the changelog file.
> I think this could be a subtask of [v2 FLIP-158: Generalized incremental
> checkpoints|https://issues.apache.org/jira/browse/FLINK-25842] .
> Hi [~ym] , [~roman] how do you think about ?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
