[
https://issues.apache.org/jira/browse/FLINK-11696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yun Tang updated FLINK-11696:
-----------------------------
Summary: Avoid to send mkdir requests to DFS from task side (was:
[checkpoint] Avoid to send mkdir requests to DFS from task side)
> Avoid to send mkdir requests to DFS from task side
> --------------------------------------------------
>
> Key: FLINK-11696
> URL: https://issues.apache.org/jira/browse/FLINK-11696
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Checkpointing
> Reporter: Yun Tang
> Assignee: Yun Tang
> Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Currently, when we create checkpoint directory in the distributed file
> system. Not only {{CheckpointCoordinator}} but also {{FsCheckpointStorage}}
> in {{StreamTask}} would create the {{checkpointsDirectory}},
> {{sharedStateDirectory}} and {{taskOwnedStateDirectory}}. These many
> {{mkdir}} RPC requests would cause a very high pressure on the distributed
> file system, especially when the parallelism is large or jobs continue to
> failover again and again.
> We could avoid these {{mkdir}} requests from task side if writing to a
> distributed file system.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
