[
https://issues.apache.org/jira/browse/FLINK-26803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17538819#comment-17538819
]
Piotr Nowojski commented on FLINK-26803:
----------------------------------------
Sorry for late response [~fanrui]. Thanks for the explanation, I now understand
the motivation. Indeed for mostly state less jobs this can be an issue.
However I'm still not sure if this is the right thing to do, or at least the
right way to solve this problem.
{{quote}}
For large parallelism flink jobs, this size is usually more than 1M.
{{quote}}
1M is not that small file. I think most of the stateful jobs have actually
smaller state files.
Maybe there should be some more generic mechanism for aggregating small state
handles, that would solve this problem and at the same time state files? CC
[~ym] But maybe that's not feasible, as we probably can not merge two RocksDB's
SST files from different operators/tasks into a single state handle the same
way we could do it for the in-flight data?
> Merge small ChannelState file for Unaligned Checkpoint
> ------------------------------------------------------
>
> Key: FLINK-26803
> URL: https://issues.apache.org/jira/browse/FLINK-26803
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Checkpointing, Runtime / Network
> Reporter: fanrui
> Priority: Major
> Attachments: image-2022-05-05-12-36-09-969.png
>
>
> When making an unaligned checkpoint, the number of ChannelState files is
> TaskNumber * subtaskNumber. For high parallelism job, it writes too many
> small files. It causes high load for hdfs NN.
>
> In our production, a job writes more than 50K small files for each Unaligned
> Checkpoint. Could we merge these files before write FileSystem? We can
> configure the maximum number of files each TM can write in a single Unaligned
> Checkpoint.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
