[
https://issues.apache.org/jira/browse/HDFS-17333?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jian Zhang updated HDFS-17333:
------------------------------
Description:
Currently, when dfsclient is started, it will resolve all hosts of all
namservices:
at DFSUtilClient#getAddresses(conf, null, addressKey)
at AbstractNNFailoverProxyProvider#getProxyAddresses(URI uri, String
addressKey)
If the current environment where the dfsClient is located causes resolution of
host->ip to be very slow, the existing logic will undoubtedly take a long time
when there are too many nameservices.
Now, each dfsclient only needs the IPs of all namenodes of a certain
nameservice at most. A better situation is that if the namenode selected by
dfsclient for the first time can provide the required services normally, then
the client only needs to know the IP of this namenode. Therefore, it is not
necessary to resolve all namenodes of all nameservices in the configuration
file, when dfsclient is started.
This patch supports lazy resolution of host->ip, which will only be resolved
when the host needs to be accessed.
was:
Currently, when dfsclient is started, it will parse all hosts of all ns:
at DFSUtilClient#getAddresses(conf, null, addressKey)
at AbstractNNFailoverProxyProvider#getProxyAddresses(URI uri, String
addressKey)
If the current environment where the client is located causes resolution of
host->ip to be very slow, the existing logic will undoubtedly take a long time
when there are too many NSs.
This patch supports lazy resolution of host->ip, which will only be resolved
when the host needs to be accessed.
> Support lazy resolve host->ip.
> ------------------------------
>
> Key: HDFS-17333
> URL: https://issues.apache.org/jira/browse/HDFS-17333
> Project: Hadoop HDFS
> Issue Type: Improvement
> Reporter: Jian Zhang
> Assignee: Jian Zhang
> Priority: Major
> Labels: pull-request-available
> Attachments: HDFS-17333.001.patch
>
>
> Currently, when dfsclient is started, it will resolve all hosts of all
> namservices:
> at DFSUtilClient#getAddresses(conf, null, addressKey)
> at AbstractNNFailoverProxyProvider#getProxyAddresses(URI uri,
> String addressKey)
> If the current environment where the dfsClient is located causes resolution
> of host->ip to be very slow, the existing logic will undoubtedly take a long
> time when there are too many nameservices.
> Now, each dfsclient only needs the IPs of all namenodes of a certain
> nameservice at most. A better situation is that if the namenode selected by
> dfsclient for the first time can provide the required services normally, then
> the client only needs to know the IP of this namenode. Therefore, it is not
> necessary to resolve all namenodes of all nameservices in the configuration
> file, when dfsclient is started.
> This patch supports lazy resolution of host->ip, which will only be resolved
> when the host needs to be accessed.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
