fanrui created FLINK-26803:
------------------------------
Summary: Merge small ChannelState file for Unaligned Checkpoint
Key: FLINK-26803
URL: https://issues.apache.org/jira/browse/FLINK-26803
Project: Flink
Issue Type: Improvement
Components: Runtime / Checkpointing, Runtime / Network
Reporter: fanrui
When making an unaligned checkpoint, the number of ChannelState files is
TaskNumber * subtaskNumber. For high parallelism job, it writes too many small
files. It causes high load for hdfs NN.
In our production, a job writes more than 50K small files for each Unaligned
Checkpoint. Could we merge these files before write FileSystem? We can
configure the maximum number of files each TM can write in a single Unaligned
Checkpoint.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
