[
https://issues.apache.org/jira/browse/HDFS-6469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14120723#comment-14120723
]
Konstantin Shvachko commented on HDFS-6469:
-------------------------------------------
Thanks for the thoughts, Sanjay. I truly appreciate your commitment to support
CNode plugins. Will create a jira to track that.
Here is one of the user stories I heard recently. To set up NN HA on a big
cluster people used
* 2 nodes running NN and SBN
* 5 QJM nodes for the shared journal, and
* 5 ZK nodes for automatic fail over.
Total of 12 nodes, but with only one being actively used by clients. And
because of that even this setup did not prevent them from Yarn jobs failures or
HBase service interruption, when NN hit full GC causing its fail-over to SBN.
With active-active you can simply have 5 ConsensusNodes. This still tolerates
two node failures, and all five could be used by clients interchangeably
* preventing from somebody going into full GC or being overwhelmed by many RPCs
* allowing load-balancing of read workloads,
* all this with half of the resources.
* I also consider CNode as a prerequisite for geographically distributed HDFS,
which allows to scale file system across data centers.
Setting aside rather technical issues, like [CNode
complexity|https://issues.apache.org/jira/browse/HDFS-6469?focusedCommentId=14021017&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14021017],
[usability of
ZooKeeper|https://issues.apache.org/jira/browse/HDFS-6469?focusedCommentId=14069598&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14069598],
[read-enabled SBN
alternative|https://issues.apache.org/jira/browse/HDFS-6469?focusedCommentId=14027179&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14027179],
which I addressed in the linked comments, _is it not a sufficient
justification for the costs of adding CNode for you?_
NN HA was built about two years ago, which essentially paved its road into the
enterprise world. I see CNode as a step in HDFS evolution, as it is utilizing
and built on top of all the innovations introduced by federation and HA.
> Coordinated replication of the namespace using ConsensusNode
> ------------------------------------------------------------
>
> Key: HDFS-6469
> URL: https://issues.apache.org/jira/browse/HDFS-6469
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: namenode
> Affects Versions: 3.0.0
> Reporter: Konstantin Shvachko
> Assignee: Konstantin Shvachko
> Attachments: CNodeDesign.pdf
>
>
> This is a proposal to introduce ConsensusNode - an evolution of the NameNode,
> which enables replication of the namespace on multiple nodes of an HDFS
> cluster by means of a Coordination Engine.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
