[
https://issues.apache.org/jira/browse/HDFS-6469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14062559#comment-14062559
]
Sanjay Radia commented on HDFS-6469:
------------------------------------
Todd said:
bq. a fully usable solution would be available to the community at large,
whereas the design you're proposing seems like it will only be usably
implemented by a proprietary extension (I don't consider the ZK "reference
implementation" likely to actually work in a usable fashion).
Konstanine I had mentioned exactly the above point to you at the Hadoop summit
Europe. ZK is a coordination service and for this to be practical it needs to
be an inline Paxos protocol. We had also discussed 2 potential paxos libraries
that could come into open source: I believe Facebook has one that they may
contribute and CMU has one called E-Paxos; if either of these become available
then it addresses this particular issue. I have no objections to a customer
going to Wandisco for the enterprise supported version, but if the community
is going to maintain such an extension then there needs to a practical,
in-production-usable free solution; sending offline messages to a coordinator
service for each transaction is not usable. Lets discuss the performance part
in a separate comment. Let me comment on your comparisons to the topology and
windows examples that the community supported in the past:
* Topology - these changes allowed Hadoop to be used on containers such as VMs.
** Both KVM and VirtualBox offer free VM solutions - the customer does not need
to buy ESX.
** The topology solution would will also help with a Docker container
deployment which is freely available and offers even better performance than
VMs.
** Hadoop is commonly used in cloud environment (e.g. AWS, or Azure, or
Altiscale) which all use VMs or containers
** Further, it was recognized that while, in the past, we had considered racks
to be a failure zone, that there could be other failure zones: nodes (for the
case of VMs or containers on a host) and also groups of machines.
* Windows - this was done for platform support which is very different than
what we are talking about here; many open source solutions support multiple
platforms to enable the widest adoption. BTW Hadoop supported windows via
cygwin but we made it first class since the initial support via cygwin was
messy.
> Coordinated replication of the namespace using ConsensusNode
> ------------------------------------------------------------
>
> Key: HDFS-6469
> URL: https://issues.apache.org/jira/browse/HDFS-6469
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: namenode
> Affects Versions: 3.0.0
> Reporter: Konstantin Shvachko
> Assignee: Konstantin Shvachko
> Attachments: CNodeDesign.pdf
>
>
> This is a proposal to introduce ConsensusNode - an evolution of the NameNode,
> which enables replication of the namespace on multiple nodes of an HDFS
> cluster by means of a Coordination Engine.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
