[
https://issues.apache.org/jira/browse/FLINK-12343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16827997#comment-16827997
]
Zhenqiu Huang commented on FLINK-12343:
---------------------------------------
It can be set by in both way. Here is to set the bigger number for yarn
applications with container requirement more than 128 containers to accelerate
the bootstrap of container initialization. If changing the replication factor
for a directory will only affect the existing files and the new files under the
directory will get created with the default replication factor (dfs.replication
from hdfs-site.xml) of the cluster. As we don't want to apply bigger
replication number for any of other files in hdfs, I think we have to set from
client side. How do you think?
> Allow set hdfs.replication in Yarn Configuration
> ------------------------------------------------
>
> Key: FLINK-12343
> URL: https://issues.apache.org/jira/browse/FLINK-12343
> Project: Flink
> Issue Type: Improvement
> Components: Command Line Client, Deployment / YARN
> Affects Versions: 1.6.4, 1.7.2, 1.8.0
> Reporter: Zhenqiu Huang
> Assignee: Zhenqiu Huang
> Priority: Minor
>
> Currently, FlinkYarnSessionCli upload jars into hdfs with default 3
> replications. From our production experience, we find that 3 replications
> will block big job (256 containers) to launch, when the HDFS is slow due to
> big workload for batch pipelines. Thus, we want to make the factor
> customizable from FlinkYarnSessionCli by adding an option.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
