Stack, Ted, Thanks a lot for the pointers. I'm amazed at the quick responses that come in this mailing lists from this community!
I had changed the directory from /tmp to some other directory for hbase.rootdir long back only, but had missed doing the same for Zookeeper in the conf/hbase-site.xml, since I assumed that setting the HBASE_MANAGES_ZK=true flag in the hbase-env.sh causes zookeeper to be managed internally by HBase and as such control over the zookeeper data directory is not possible. After reading the pointers you gave, I changed it to some other directory other than /tmp for zookeeper. So when the /tmp file system on the master node in our cluster went corrupt somehow, HQuorumPeer daemon went kaput forcing shut down of the HMaster daemon. Thanks once again. Best, - Bharat -- B On Sat, Nov 23, 2013 at 10:52 AM, Stack <[email protected]> wrote: > On Fri, Nov 22, 2013 at 6:11 PM, Bharat Shetty <[email protected] > >wrote: > > > Yeah, error is related to HDFS. > > > > I'm using HBase 0.96.0 with Hadoop 2.1.0-beta. > > > > On more further digging, it appears that the /tmp directory on the master > > node had some problems. Zookeeper for deployment is being managed > > internally by the HBase, appears to store data in /tmp directory > > (/tmp/hbase-iouser/zookeeper/). Since this region of the filesystem was > > unstable owing filesystem problems in the /tmp folder of the master node, > > zookeeper communication btw Master and slaves failed and as such HMaster > > seems to have shut down. I was able to get up the master running on other > > node with same configurations used previously. > > > > There was no changes to the lib directory of the HBase during deployment. > > The setup was working fine and I was able to run map-reduce programs for > > importing, exporting and filtering on millions of records in HBase prior > to > > HMaster failure. > > > > In a production level scenario, which is ideal ? Zookeeper managed > > ourselves as opposed to zookeeper managed internally by the HBase ? > > > > Also, are there any documentation or links anywhere for production level > > configurations for HBase running on top of the HDFS (Hadoop) ? > > > > See the first section in the refguide, the quickstart section: > http://hbase.apache.org/book.html#quickstart The first thing it tells you > do is edit hbase-site.xml and change where zk is writing its data. "By > default, hbase.rootdir is set to /tmp/hbase-${user.name} and similarly so > for the default ZooKeeper data location which means you'll lose all your > data whenever your server reboots unless you change it (Most operating > systems clear /tmp on restart)." > > St.Ack >
