Does your master log really only have those lines in it? Is there a hbase-hadoop-master-XXX.out file?
-Joey On Tue, Jun 7, 2011 at 10:37 AM, Zhong, Sheng <[email protected]> wrote: > > Joey, > > Yeah, the master did not start properly, even though I could see the process > and listener port: 60020, but could not have access. Thank you for looking > into this. Thanks, - Andy Zhong > > hbase-hadoop-master-XXX.log: > Mon Jun 6 21:53:14 CDT 2011 Starting master on hofvmecln21.ecom.sears.com > ulimit -n 1024 > Mon Jun 6 22:08:04 CDT 2011 Starting master on hofvmecln21.ecom.sears.com > ulimit -n 1024 > > Hbase.log: > 2011-06-07 09:33:16,996 WARN regionserver.HRegionServer - Unable to read > master address from ZooKeeper. Retrying. Error was: > java.io.IOException: org.apache.zookeeper.KeeperException$NoNodeException: > KeeperErrorCode = NoNode for /hbase/master > at > org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readAddressOrThrow(ZooKeeperWrapper.java:394) > at > org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readMasterAddressOrThrow(ZooKeeperWrapper.java:298) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.getMaster(HRegionServer.java:1282) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:1314) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:430) > at java.lang.Thread.run(Thread.java:619) > Caused by: org.apache.zookeeper.KeeperException$NoNodeException: > KeeperErrorCode = NoNode for /hbase/master > at > org.apache.zookeeper.KeeperException.create(KeeperException.java:102) > at org.apache.zookeeper.KeeperException.create(KeeperException.java:42) > at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:892) > at > org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readAddressOrThrow(ZooKeeperWrapper.java:390) > ... 5 more > 2011-06-07 09:33:17,999 WARN regionserver.HRegionServer - Unable to read > master address from ZooKeeper. Retrying. Error was: > java.io.IOException: org.apache.zookeeper.KeeperException$NoNodeException: > KeeperErrorCode = NoNode for /hbase/master > at > org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readAddressOrThrow(ZooKeeperWrapper.java:394) > at > org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readMasterAddressOrThrow(ZooKeeperWrapper.java:298) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.getMaster(HRegionServer.java:1282) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:1314) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:430) > at java.lang.Thread.run(Thread.java:619) > Caused by: org.apache.zookeeper.KeeperException$NoNodeException: > KeeperErrorCode = NoNode for /hbase/master > at > org.apache.zookeeper.KeeperException.create(KeeperException.java:102) > at org.apache.zookeeper.KeeperException.create(KeeperException.java:42) > at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:892) > at > org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readAddressOrThrow(ZooKeeperWrapper.java:390) > ... 5 more > > > > > > -----Original Message----- > From: Joey Echeverria [mailto:[email protected]] > Sent: Tuesday, June 07, 2011 9:30 AM > To: [email protected] > Subject: Re: Hbase startup error: NoNode for /hbase/master after running out > of space > >> 2011-06-06 18:40:11,833 WARN regionserver.HRegionServer - Unable to >> read master address from ZooKeeper. Retrying. Error was: > > This means that your master didn't start. Can you post the log for the master? > > -Joey > > On Tue, Jun 7, 2011 at 9:47 AM, Zhong, Sheng <[email protected]> wrote: >> >> Hey, >> >> The system works perfect until receiving device out of space and more >> space is added. Hope anyone here could provide help on the hbase issue >> we got after our of space. I could bring up HDFS via one of post found >> by manually modifying namenode's /var/hdfs/current/edits file: $ >> printf "\xff\xff\xff\xee\xff" > edits. However, the issue came to >> hbase startup. We could see HMaster and HRegion processes running, but >> could not access Hbase. The errors is described below: >> >> I am appreciated by any help and suggestion. We are even okay to lose >> recent unprocessed data. P.S: we're using apache hadoop 0.20.2 and >> hbase 0.20.3, and zookeeper is running via zookeeper-3.2.2 (not >> managed by Hbase). >> >> Hbase.log: >> 2011-06-06 18:40:11,833 WARN regionserver.HRegionServer - Unable to >> read master address from ZooKeeper. Retrying. Error was: >> java.io.IOException: >> org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode >> = NoNode for /hbase/master >> at >> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readAddressOrThrow( >> Zo >> oKeeperWrapper.java:394) >> at >> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readMasterAddressOr >> Th >> row(ZooKeeperWrapper.java:298) >> at >> org.apache.hadoop.hbase.regionserver.HRegionServer.getMaster(HRegionSe >> rv >> er.java:1282) >> at >> org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegi >> on >> Server.java:1314) >> at >> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.j >> av >> a:430) >> at java.lang.Thread.run(Thread.java:619) >> Caused by: org.apache.zookeeper.KeeperException$NoNodeException: >> KeeperErrorCode = NoNode for /hbase/master >> at >> org.apache.zookeeper.KeeperException.create(KeeperException.java:102) >> at >> org.apache.zookeeper.KeeperException.create(KeeperException.java:42) >> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:892) >> at >> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readAddressOrThrow( >> Zo >> oKeeperWrapper.java:390) >> ... 5 more >> >> Zookeeper.log: >> 2011-06-06 21:48:36,174 - WARN >> [ProcessThread:-1:PrepRequestProcessor@402] - Got exception when >> processing sessionid:0x23067fc51970002 type:create cxid:0x19a9 >> zxid:0xfffffffffffffffe txntype:unknown n/a >> org.apache.zookeeper.KeeperException$NodeExistsException: >> KeeperErrorCode = NodeExists >> at >> org.apache.zookeeper.server.PrepRequestProcessor.pRequest(PrepRequestP >> ro >> cessor.java:245) >> at >> org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProces >> so >> r.java:114) >> >> >> Regards, >> Andy (Sheng) Zhong >> >> >> This message, including any attachments, is the property of Sears Holdings >> Corporation and/or one of its subsidiaries. It is confidential and may >> contain proprietary or legally privileged information. If you are not the >> intended recipient, please delete it without reading the contents. Thank you. >> > > > > -- > Joseph Echeverria > Cloudera, Inc. > 443.305.9434 > -- Joseph Echeverria Cloudera, Inc. 443.305.9434
