[
https://issues.apache.org/jira/browse/HDFS-3077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13229634#comment-13229634
]
Todd Lipcon commented on HDFS-3077:
-----------------------------------
Hi Flavio. The philosophy of having building blocks is a great one. I'm an
ardent supporter of using ZooKeeper, and other building blocks which fit our
needs. I think BookKeeper, though, is like using a chainsaw to get a haircut.
It does lots of stuff we don't need, and will take a ton of work to move it to
fit the other requirements we do have.
bq. Also, if these aspects are important for you, why don't you want to
contribute them to the project?
Moving BK over to Hadoop-based configuration, IPC, security, local edit log
implementation, quorum based commit, etc, would leave almost nothing left of
BK, except for (a) features we don't need, like striping and interleaved logs
and (b) the core protocol, which you have described yourself as relatively
simple.
bq. Finally, we have focused on the implementation of core protocols like zab
and the quorum consensus of BookKeeper. Why not leverage this experience and
focus?
I absolutely am leveraging that experience. I spent some 10 hours this weekend
studying your papers and presentations on BK and ZAB. BK is clearly a very
similar system and I'm sure I will reference its design while working on this
system.
Regardless, as I said above, I intend to continue down this avenue. Please
continue working on the BookKeeper one if you think it is a better avenue. If
my proposed solution turns into a disaster because I screw up the quorum
implementations, I'm sure no one will use it, and I'll be glad that you
continued to work on an alternate. The joy of open source and pluggable
implementations is that we can both prove our ideas in code and let the
community vote with their feet.
I don't think this argument is likely to be fruitful if we continue. So let's
just agree to disagree and each of us can get to work on proving the other
wrong with actual code.
> Quorum-based protocol for reading and writing edit logs
> -------------------------------------------------------
>
> Key: HDFS-3077
> URL: https://issues.apache.org/jira/browse/HDFS-3077
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: ha, name-node
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
>
> Currently, one of the weak points of the HA design is that it relies on
> shared storage such as an NFS filer for the shared edit log. One alternative
> that has been proposed is to depend on BookKeeper, a ZooKeeper subproject
> which provides a highly available replicated edit log on commodity hardware.
> This JIRA is to implement another alternative, based on a quorum commit
> protocol, integrated more tightly in HDFS and with the requirements driven
> only by HDFS's needs rather than more generic use cases. More details to
> follow.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
