Rémy SAISSY created HDFS-11300:
----------------------------------
Summary: HDFS Nameservices introduces DNS latency
inter-dependencies between namenodes
Key: HDFS-11300
URL: https://issues.apache.org/jira/browse/HDFS-11300
Project: Hadoop HDFS
Issue Type: Improvement
Components: hdfs, hdfs-client
Affects Versions: 2.6.0
Reporter: Rémy SAISSY
When using HDFS Nameservices, a DNS outage or strong latency on one of
nameservices will impact any DFSClient or other component which has this
nameservice in its hdfs-site.xml. Even though it doesn't use it.
The issue is due to all nameservices being listed in hdfs-site.xml, and hadoop
code trying to resolve all of them upfront.
To ilustrate the issue, here is the use case we have:
* the 'prod' cluster with its dedicated DNS
* the 'preprod' cluster with its dedicated DNS
* both 'prod' and 'preprod' hdfs-site.xml files have each other nameservices
so that they can use HDFS as follows:
(from prod) hdfs dfs -ls hdfs://preprod/user/j.doe
A DNS outage on the preprod cluster slow down production jobs being scheduled
because of the upfront resolution of the hdfs-site.xml nameservices entries
even though a specific job doesn't use it.
A solution to fix this issue is to fix the upfront resolution and do it in a
lazy way.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
