[
https://issues.apache.org/jira/browse/HBASE-2098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12798157#action_12798157
]
Berk D. Demir commented on HBASE-2098:
--------------------------------------
Until HDFS comes up with a solution to eliminate NN SPoF, old fashioned HA
measures are required to keep NameNode available.
So far, best and seemingly reliable bet on Linux is to have network replicated
block device, a heart beat providing messaging connection between HA nodes and
a cluster resource manager software to keep track of infrastructural resource
dependencies and moving them between machines in the requirement order.
All in all, HBase's tolerance window for NN unavailability mostly depends on
particular load at the time of failover and RSs requirements to create new
files.
Failing over to another node where a healthy replica of NN store exists and
starting an NN instance will cause the NN to collect block information from
every "new" and "unknown" DataNode for the first time. Additionally, default
value for extension of safe mode after threshold reach is 30 seconds.
(property: {{dfs.namenode.safemode.extension}}). This prolonged unavailability
window can/will have bad effects on RSs. (jdcryans will comment his
observations).
We implemented a NameNode HA cluster with open source tools like OpenAIS,
Pacemaker, Heartbeat and DRBD.
* NameNode disk storage is replicated between two machines _(adding a 3rd
machine is possible with new DRBD)_.
* OpenAIS provides intra-cluster messaging and heart beat availability layer.
* Pacemaker is used to manage Cluster Resources. _(DRBD disks, filesystem
mount, NN service IP, NN daemon)_
* An OCF script to start, stop, validate and monitor _(periodic calls)_ the
subsystem _(NN, JT, SNN)_.
At the end of the day, this is applicable to not only NameNode but also to
JobTracker and SecondaryNameNode.
For a starting point, ClusterLabs _(creators of Pacemaker)_ offer an e-book on
creating clusters with DRBD, Pacemaker, OpenAIS called [Clusters From
Scratch|http://www.clusterlabs.org/mediawiki/images/9/9d/Clusters_from_Scratch_-_Apache_on_Fedora11.pdf]
^(PDF Link)^
> [EC2] Build a HA cluster
> ------------------------
>
> Key: HBASE-2098
> URL: https://issues.apache.org/jira/browse/HBASE-2098
> Project: Hadoop HBase
> Issue Type: Sub-task
> Reporter: Andrew Purtell
> Assignee: Andrew Purtell
> Priority: Minor
> Fix For: 0.21.0
>
>
> The Hadoop NameNode is a single point of failure. If the master instance
> fails, HDFS is down; therefore, HBase as well. So we do not try to deploy
> HBase in a multimaster configuration for that reason. Instead we colocate the
> HDFS NameNode and HBase HMaster on the same instance and run with its failure
> as a known risk. As these EC2 scripts are starter scripts which can (and
> should) be customized, this is ok, but we can do better. We should deploy a
> fully fault tolerant Hadoop+HBase cluster as a worked example of how to do
> it.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
