Yun Tang created FLINK-11696:
--------------------------------
Summary: [checkpoint] Avoid to send mkdir requests to DFS from
task side
Key: FLINK-11696
URL: https://issues.apache.org/jira/browse/FLINK-11696
Project: Flink
Issue Type: Improvement
Components: State Backends, Checkpointing
Reporter: Yun Tang
Assignee: Yun Tang
Currently, when we create checkpoint directory in the distributed file system.
Not only {{CheckpointCoordinator}} but also {{FsCheckpointStorage}} in
{{StreamTask}} would create the {{checkpointsDirectory}},
{{sharedStateDirectory}} and {{taskOwnedStateDirectory}}. These many {{mkdir}}
RPC requests would cause a very high pressure on the distributed file system,
especially when the parallelism is large or jobs continue to failover again and
again.
We could avoid these {{mkdir}} requests from task side if writing to a
distributed file system.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
