Yes, the hbase-hadoop-regionserver-hofvmecln21.ecom.sears.com.log is empty.
-----Original Message----- From: Joey Echeverria [mailto:[email protected]] Sent: Tuesday, June 07, 2011 10:04 AM To: [email protected] Subject: Re: Hbase startup error: NoNode for /hbase/master after running out of space Does your master log really only have those lines in it? Is there a hbase-hadoop-master-XXX.out file? -Joey On Tue, Jun 7, 2011 at 10:37 AM, Zhong, Sheng <[email protected]> wrote: > > Joey, > > Yeah, the master did not start properly, even though I could see the > process and listener port: 60020, but could not have access. Thank you > for looking into this. Thanks, - Andy Zhong > > hbase-hadoop-master-XXX.log: > Mon Jun 6 21:53:14 CDT 2011 Starting master on > hofvmecln21.ecom.sears.com ulimit -n 1024 Mon Jun 6 22:08:04 CDT 2011 > Starting master on hofvmecln21.ecom.sears.com ulimit -n 1024 > > Hbase.log: > 2011-06-07 09:33:16,996 WARN regionserver.HRegionServer - Unable to read > master address from ZooKeeper. Retrying. Error was: > java.io.IOException: > org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode > = NoNode for /hbase/master > at > org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readAddressOrThrow( > ZooKeeperWrapper.java:394) > at > org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readMasterAddressOr > Throw(ZooKeeperWrapper.java:298) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.getMaster(HRegionSe > rver.java:1282) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegi > onServer.java:1314) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.j > ava:430) > at java.lang.Thread.run(Thread.java:619) > Caused by: org.apache.zookeeper.KeeperException$NoNodeException: > KeeperErrorCode = NoNode for /hbase/master > at > org.apache.zookeeper.KeeperException.create(KeeperException.java:102) > at > org.apache.zookeeper.KeeperException.create(KeeperException.java:42) > at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:892) > at > org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readAddressOrThrow( > ZooKeeperWrapper.java:390) > ... 5 more > 2011-06-07 09:33:17,999 WARN regionserver.HRegionServer - Unable to read > master address from ZooKeeper. Retrying. Error was: > java.io.IOException: > org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode > = NoNode for /hbase/master > at > org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readAddressOrThrow( > ZooKeeperWrapper.java:394) > at > org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readMasterAddressOr > Throw(ZooKeeperWrapper.java:298) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.getMaster(HRegionSe > rver.java:1282) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegi > onServer.java:1314) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.j > ava:430) > at java.lang.Thread.run(Thread.java:619) > Caused by: org.apache.zookeeper.KeeperException$NoNodeException: > KeeperErrorCode = NoNode for /hbase/master > at > org.apache.zookeeper.KeeperException.create(KeeperException.java:102) > at > org.apache.zookeeper.KeeperException.create(KeeperException.java:42) > at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:892) > at > org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readAddressOrThrow( > ZooKeeperWrapper.java:390) > ... 5 more > > > > > > -----Original Message----- > From: Joey Echeverria [mailto:[email protected]] > Sent: Tuesday, June 07, 2011 9:30 AM > To: [email protected] > Subject: Re: Hbase startup error: NoNode for /hbase/master after > running out of space > >> 2011-06-06 18:40:11,833 WARN regionserver.HRegionServer - Unable to >> read master address from ZooKeeper. Retrying. Error was: > > This means that your master didn't start. Can you post the log for the master? > > -Joey > > On Tue, Jun 7, 2011 at 9:47 AM, Zhong, Sheng <[email protected]> wrote: >> >> Hey, >> >> The system works perfect until receiving device out of space and more >> space is added. Hope anyone here could provide help on the hbase >> issue we got after our of space. I could bring up HDFS via one of >> post found by manually modifying namenode's /var/hdfs/current/edits >> file: $ printf "\xff\xff\xff\xee\xff" > edits. However, the issue >> came to hbase startup. We could see HMaster and HRegion processes >> running, but could not access Hbase. The errors is described below: >> >> I am appreciated by any help and suggestion. We are even okay to lose >> recent unprocessed data. P.S: we're using apache hadoop 0.20.2 and >> hbase 0.20.3, and zookeeper is running via zookeeper-3.2.2 (not >> managed by Hbase). >> >> Hbase.log: >> 2011-06-06 18:40:11,833 WARN regionserver.HRegionServer - Unable to >> read master address from ZooKeeper. Retrying. Error was: >> java.io.IOException: >> org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode >> = NoNode for /hbase/master >> at >> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readAddressOrThrow >> ( >> Zo >> oKeeperWrapper.java:394) >> at >> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readMasterAddressO >> r >> Th >> row(ZooKeeperWrapper.java:298) >> at >> org.apache.hadoop.hbase.regionserver.HRegionServer.getMaster(HRegionS >> e >> rv >> er.java:1282) >> at >> org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HReg >> i >> on >> Server.java:1314) >> at >> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer. >> j >> av >> a:430) >> at java.lang.Thread.run(Thread.java:619) >> Caused by: org.apache.zookeeper.KeeperException$NoNodeException: >> KeeperErrorCode = NoNode for /hbase/master >> at >> org.apache.zookeeper.KeeperException.create(KeeperException.java:102) >> at >> org.apache.zookeeper.KeeperException.create(KeeperException.java:42) >> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:892) >> at >> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readAddressOrThrow >> ( >> Zo >> oKeeperWrapper.java:390) >> ... 5 more >> >> Zookeeper.log: >> 2011-06-06 21:48:36,174 - WARN >> [ProcessThread:-1:PrepRequestProcessor@402] - Got exception when >> processing sessionid:0x23067fc51970002 type:create cxid:0x19a9 >> zxid:0xfffffffffffffffe txntype:unknown n/a >> org.apache.zookeeper.KeeperException$NodeExistsException: >> KeeperErrorCode = NodeExists >> at >> org.apache.zookeeper.server.PrepRequestProcessor.pRequest(PrepRequest >> P >> ro >> cessor.java:245) >> at >> org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProce >> s >> so >> r.java:114) >> >> >> Regards, >> Andy (Sheng) Zhong >> >> >> This message, including any attachments, is the property of Sears Holdings >> Corporation and/or one of its subsidiaries. It is confidential and may >> contain proprietary or legally privileged information. If you are not the >> intended recipient, please delete it without reading the contents. Thank you. >> > > > > -- > Joseph Echeverria > Cloudera, Inc. > 443.305.9434 > -- Joseph Echeverria Cloudera, Inc. 443.305.9434
