[
https://issues.apache.org/jira/browse/FLINK-26803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17552631#comment-17552631
]
fanrui commented on FLINK-26803:
--------------------------------
Hi everyone, I've done a POC for merging the channel state files and organized
the documentation. Hope the masters can participate in the discussion in the
doc or JIRA.
[https://docs.google.com/document/d/1uGp1gJxxcJMUCh5DxzPklGc2SicMN-Dk-YOce2XMaAg/edit?usp=sharing]
POC branch:
[https://github.com/1996fanrui/flink/commits/26803/merge-channel-file-POC]
The latest commit:
[https://github.com/1996fanrui/flink/commit/48780677683dad43cfb2211781493a234bab087c]
> Merge small ChannelState file for Unaligned Checkpoint
> ------------------------------------------------------
>
> Key: FLINK-26803
> URL: https://issues.apache.org/jira/browse/FLINK-26803
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Checkpointing, Runtime / Network
> Reporter: fanrui
> Priority: Major
> Attachments: image-2022-05-05-12-36-09-969.png
>
>
> When making an unaligned checkpoint, the number of ChannelState files is
> TaskNumber * subtaskNumber. For high parallelism job, it writes too many
> small files. It causes high load for hdfs NN.
>
> In our production, a job writes more than 50K small files for each Unaligned
> Checkpoint. Could we merge these files before write FileSystem? We can
> configure the maximum number of files each TM can write in a single Unaligned
> Checkpoint.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
