Giambattista Bloisi created HADOOP-20084:
--------------------------------------------
Summary: Allow the HDFS client to set the default replication per
cluster with dfs.replication.<cluster>
Key: HADOOP-20084
URL: https://issues.apache.org/jira/browse/HADOOP-20084
Project: Hadoop Common
Issue Type: New Feature
Components: hdfs-client
Affects Versions: 3.5.0
Reporter: Giambattista Bloisi
When one application or Spark job uses several HDFS clusters, the client
applies the same default replication to every file it creates on any of them.
That default comes from {_}dfs.replication{_}, and the HDFS client has only
that one key; it has no per-nameservice form.
This is a problem when the clusters need different replication. For example, a
job may write its output to a production cluster with 3 replicas and its
intermediate data to a scratch cluster with 1 replica for speed. Most clients,
including the S3A committers, Spark and many libraries, create files without
setting a replication. So every file the job creates on the scratch cluster
asks for 3 replicas. If that cluster sets {_}dfs.replication.max=1{_}, the
NameNode rejects the create:
{noformat}
java.io.IOException: Requested replication factor of 3 exceeds maximum of 1 for
/tmp-staging/.../task_..._m_000041.pendingset{noformat}
The workarounds are all poor:
* Setting _dfs.replication_ for the job affects every cluster the job writes
to, so production data loses replicas.
* Running _setrep_ on a directory changes only the files already in it, not
new ones.
* Patching every writer to pass a replication is not possible for code the
user doesn't control.
{*}Proposal{*}: when the client creates a file, use _dfs.replication.<cluster>_
as a per-cluster override of the global _dfs.replication:_
{code:java}
<property>
<name>dfs.replication</name>
<value>3</value></property>
<property>
<name>dfs.replication.scratch</name>
<value>1</value>
</property>{code}
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
