[
https://issues.apache.org/jira/browse/HDFS-11300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15806162#comment-15806162
]
Allen Wittenauer commented on HDFS-11300:
-----------------------------------------
bq. A solution to fix this issue is to fix the upfront resolution and do it in
a lazy way.
Or, what most large sites do, deploy DNS caching servers on every node. This
greatly improves performance in general.
> HDFS Nameservices introduces DNS latency inter-dependencies between namenodes
> -----------------------------------------------------------------------------
>
> Key: HDFS-11300
> URL: https://issues.apache.org/jira/browse/HDFS-11300
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs, hdfs-client
> Affects Versions: 2.6.0
> Reporter: Rémy SAISSY
>
> When using HDFS Nameservices, a DNS outage or strong latency on one of
> nameservices will impact any DFSClient or other component which has this
> nameservice in its hdfs-site.xml. Even though it doesn't use it.
> The issue is due to all nameservices being listed in hdfs-site.xml, and
> hadoop code trying to resolve all of them upfront.
> To ilustrate the issue, here is the use case we have:
> * the 'prod' cluster with its dedicated DNS
> * the 'preprod' cluster with its dedicated DNS
> * both 'prod' and 'preprod' hdfs-site.xml files have each other nameservices
> so that they can use HDFS as follows:
> (from prod) hdfs dfs -ls hdfs://preprod/user/j.doe
> A DNS outage on the preprod cluster slow down production jobs being scheduled
> because of the upfront resolution of the hdfs-site.xml nameservices entries
> even though a specific job doesn't use it.
> A solution to fix this issue is to fix the upfront resolution and do it in a
> lazy way.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
