[
https://issues.apache.org/jira/browse/HDFS-6648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15706258#comment-15706258
]
Inigo Goiri commented on HDFS-6648:
-----------------------------------
I'm working on Router-based HDFS federation (HDFS-10467) and I've been hitting
issues related to this JIRA.
As I have multiple routers that can server requests, I'm setting the
ConfiguredFailoverProxyProvider to point to all of them.
The problem is that all the clients point to the same Router defeating one of
the advantages of Router-based federation.
In addition, the comment that [~ravwojdyla] mentioned is pretty missleading as
the order is just the hash order not the client one.
My proposal would be to fix the comment and instead of always starting by the
NN in index 0, we would start by a random index.
Another proposal would be to fully randomize the list of NNs.
In this way, the clients would evenly distribute the load across the Routers.
> Order of namenodes in ConfiguredFailoverProxyProvider is not defined by order
> in hdfs-site.xml
> ----------------------------------------------------------------------------------------------
>
> Key: HDFS-6648
> URL: https://issues.apache.org/jira/browse/HDFS-6648
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: ha, hdfs-client
> Affects Versions: 2.2.0
> Reporter: Rafal Wojdyla
>
> In org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider,
> in the constructor, there's a map <nameservice : < service-id :
> service-rpc-address > > (DFSUtil.getHaNnRpcAddresses). It's a LinkedHashMap
> of HashMaps. The order is kept for _nameservices_. Then to find active
> namenode, for nameservice, we get HashMap of <service-id :
> service-rpc-address > for requested nameservice (taken from URI request), And
> for this HashMap we get values - order of this collection is not strictly
> defined! In the code:
> {code}
> Collection<InetSocketAddress> addressesOfNns = addressesInNN.values();
> {code}
> And then we put these values (in not defined order) into ArrayList of
> proxies, and then in getProxy we start from first proxy in the list and
> failover to next if needed.
> It would make sense for ConfiguredFailoverProxyProvider to keep order of
> proxies/namenodes defined in hdfs-site.xml.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
