[
https://issues.apache.org/jira/browse/HDFS-11300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15806232#comment-15806232
]
Rémy SAISSY commented on HDFS-11300:
------------------------------------
I agree, but it does not fix the dependency issue.
It gives more time to fix a DNS outage as long as the local caches have not
expired.
But the issue might still arise if you have let's say, the outage starting a
few moment before the cache local cache expires on some jobs submission gateway
machines.
Unless you set your cache to never expire and decide to flush them manually
when you change your namenodes which doesn't occur every day indeed.
> HDFS Nameservices introduces DNS latency inter-dependencies between namenodes
> -----------------------------------------------------------------------------
>
> Key: HDFS-11300
> URL: https://issues.apache.org/jira/browse/HDFS-11300
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs, hdfs-client
> Affects Versions: 2.6.0
> Reporter: Rémy SAISSY
>
> When using HDFS Nameservices, a DNS outage or strong latency on one of
> nameservices will impact any DFSClient or other component which has this
> nameservice in its hdfs-site.xml. Even though it doesn't use it.
> The issue is due to all nameservices being listed in hdfs-site.xml, and
> hadoop code trying to resolve all of them upfront.
> To ilustrate the issue, here is the use case we have:
> * the 'prod' cluster with its dedicated DNS
> * the 'preprod' cluster with its dedicated DNS
> * both 'prod' and 'preprod' hdfs-site.xml files have each other nameservices
> so that they can use HDFS as follows:
> (from prod) hdfs dfs -ls hdfs://preprod/user/j.doe
> A DNS outage on the preprod cluster slow down production jobs being scheduled
> because of the upfront resolution of the hdfs-site.xml nameservices entries
> even though a specific job doesn't use it.
> A solution to fix this issue is to fix the upfront resolution and do it in a
> lazy way.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
