Yun Tang created FLINK-11869:
--------------------------------
Summary: [checkpoint] Make buffer size in checkpoint stream
factory configurable
Key: FLINK-11869
URL: https://issues.apache.org/jira/browse/FLINK-11869
Project: Flink
Issue Type: Improvement
Components: Runtime / Checkpointing
Reporter: Yun Tang
Assignee: Yun Tang
Fix For: 1.9.0
Currently, the default buffer size for {{FsCheckpointStateOutputStream}} is
only 4KB. This would case a lot of IOPS if stream is large. Unfortunately, when
user want to checkpoint on a totally disaggregated file system which has no
data node manager running in local machine, they might have a IOPS limit or
cannot serve too many IOPS at a time. This would cause the checkpoint duration
really large and might expire often.
If we want to increase this buffer size, we have to increase the
{{fileStateThreshold}} to indirectly increase the buffer size. However, as we
all know, too many not-so-small {{ByteStreamStateHandle}} returned to
checkpoint coordinator would easily cause job manager OOM and checkpoint meta
file large.
We should also make the buffer size configurable.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
