Hi Daniel,
“””
HDFS does not support two active NNs in a single cluster. Each DN
belongs to a single NN, so a federated cluster is really multiple
clusters that are stitched together at the NNs.
“””
I don’t think this information is entirely correct as of 2.7. According to the
documentation about Federated Namenodes[1], Datanodes can manage blocks from
multiple block pools – working with multiple Namenodes. This is indeed
reflected in the directory structure of the hdfs/data directory (described
here: [2]).
To configure a Datanode to talk to multiple Namenodes, one configures the
dfs.namenode.rpc-address to add namespaces, as follows:
<property>
<name>dfs.nameservices</name>
<value>ns1,ns2</value>
</property>
<property>
<name>dfs.namenode.rpc-address.ns1</name>
<value>nn1-host:port</value>
</property>
<property>
<name>dfs.namenode.rpc-address.ns2</name>
<value>nn2-host:rpc-port</value>
</property>
Yours,
Ewan
[1]
https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/Federation.html
[2] http://hortonworks.com/blog/hdfs-metadata-directories-explained/
On 22/07/16 21:58, "Daniel Templeton" <[email protected]> wrote:
On 7/22/16 12:23 PM, Kun Ren wrote:
> Thanks a lot for your reply, Daniel, very helpful.
>
> About (1) : I will consider this way, thanks. Also beside multiple
> clusters, are there any other options to do so? Thanks.
HDFS does not support two active NNs in a single cluster. Each DN
belongs to a single NN, so a federated cluster is really multiple
clusters that are stitched together at the NNs.
> About (2), if I understand correctly, HDFS used the quorum journal
> manager(QJM) for HA, and the client still only communicates with the
> active namenode, not both node, am I understanding right?
I might be confusing HDFS with YARN, but I thought the way the client
found the active NN was by just trying them all until one responds.
Daniel
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Western Digital Corporation (and its subsidiaries) E-mail Confidentiality
Notice & Disclaimer:
This e-mail and any files transmitted with it may contain confidential or
legally privileged information of WDC and/or its affiliates, and are intended
solely for the use of the individual or entity to which they are addressed. If
you are not the intended recipient, any disclosure, copying, distribution or
any action taken or omitted to be taken in reliance on it, is prohibited. If
you have received this e-mail in error, please notify the sender immediately
and delete the e-mail in its entirety from your system.
