[
https://issues.apache.org/jira/browse/FLINK-12343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16831489#comment-16831489
]
Till Rohrmann commented on FLINK-12343:
---------------------------------------
[~ZhenqiuHuang] I think the {{ResourceManager}} only stores the
{{TaskExecutors}} configuration file to HDFS when setting up the
{{TaskExecutorContext}}. Ideally we treat this situation similarly to the
upload of the other Flink cluster artifacts. But this should not be hard to do.
I guess we have agreed on the overall approach and can now start with the
implementation. I think in order to test it, we either need a
{{HdfsMiniCluster}} or the {{YarnMiniCluster}} to deploy and check the
replication factor.
> Allow set file.replication in Yarn Configuration
> ------------------------------------------------
>
> Key: FLINK-12343
> URL: https://issues.apache.org/jira/browse/FLINK-12343
> Project: Flink
> Issue Type: Improvement
> Components: Command Line Client, Deployment / YARN
> Affects Versions: 1.6.4, 1.7.2, 1.8.0
> Reporter: Zhenqiu Huang
> Assignee: Zhenqiu Huang
> Priority: Minor
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Currently, FlinkYarnSessionCli upload jars into hdfs with default 3
> replications. From our production experience, we find that 3 replications
> will block big job (256 containers) to launch, when the HDFS is slow due to
> big workload for batch pipelines. Thus, we want to make the factor
> customizable from FlinkYarnSessionCli by adding an option.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
