Can we see more of the master log, from before the zk complaints? It looks like the regions are not deploying. St.Ack
On Tue, Jun 7, 2011 at 8:07 AM, Zhong, Sheng <[email protected]> wrote: > Yes, the hbase-hadoop-regionserver-hofvmecln21.ecom.sears.com.log is empty. > > -----Original Message----- > From: Joey Echeverria [mailto:[email protected]] > Sent: Tuesday, June 07, 2011 10:04 AM > To: [email protected] > Subject: Re: Hbase startup error: NoNode for /hbase/master after running out > of space > > Does your master log really only have those lines in it? Is there a > hbase-hadoop-master-XXX.out file? > > -Joey > > On Tue, Jun 7, 2011 at 10:37 AM, Zhong, Sheng <[email protected]> wrote: >> >> Joey, >> >> Yeah, the master did not start properly, even though I could see the >> process and listener port: 60020, but could not have access. Thank you >> for looking into this. Thanks, - Andy Zhong >> >> hbase-hadoop-master-XXX.log: >> Mon Jun 6 21:53:14 CDT 2011 Starting master on >> hofvmecln21.ecom.sears.com ulimit -n 1024 Mon Jun 6 22:08:04 CDT 2011 >> Starting master on hofvmecln21.ecom.sears.com ulimit -n 1024 >> >> Hbase.log: >> 2011-06-07 09:33:16,996 WARN regionserver.HRegionServer - Unable to read >> master address from ZooKeeper. Retrying. Error was: >> java.io.IOException: >> org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode >> = NoNode for /hbase/master >> at >> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readAddressOrThrow( >> ZooKeeperWrapper.java:394) >> at >> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readMasterAddressOr >> Throw(ZooKeeperWrapper.java:298) >> at >> org.apache.hadoop.hbase.regionserver.HRegionServer.getMaster(HRegionSe >> rver.java:1282) >> at >> org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegi >> onServer.java:1314) >> at >> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.j >> ava:430) >> at java.lang.Thread.run(Thread.java:619) >> Caused by: org.apache.zookeeper.KeeperException$NoNodeException: >> KeeperErrorCode = NoNode for /hbase/master >> at >> org.apache.zookeeper.KeeperException.create(KeeperException.java:102) >> at >> org.apache.zookeeper.KeeperException.create(KeeperException.java:42) >> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:892) >> at >> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readAddressOrThrow( >> ZooKeeperWrapper.java:390) >> ... 5 more >> 2011-06-07 09:33:17,999 WARN regionserver.HRegionServer - Unable to read >> master address from ZooKeeper. Retrying. Error was: >> java.io.IOException: >> org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode >> = NoNode for /hbase/master >> at >> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readAddressOrThrow( >> ZooKeeperWrapper.java:394) >> at >> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readMasterAddressOr >> Throw(ZooKeeperWrapper.java:298) >> at >> org.apache.hadoop.hbase.regionserver.HRegionServer.getMaster(HRegionSe >> rver.java:1282) >> at >> org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegi >> onServer.java:1314) >> at >> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.j >> ava:430) >> at java.lang.Thread.run(Thread.java:619) >> Caused by: org.apache.zookeeper.KeeperException$NoNodeException: >> KeeperErrorCode = NoNode for /hbase/master >> at >> org.apache.zookeeper.KeeperException.create(KeeperException.java:102) >> at >> org.apache.zookeeper.KeeperException.create(KeeperException.java:42) >> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:892) >> at >> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readAddressOrThrow( >> ZooKeeperWrapper.java:390) >> ... 5 more >> >> >> >> >> >> -----Original Message----- >> From: Joey Echeverria [mailto:[email protected]] >> Sent: Tuesday, June 07, 2011 9:30 AM >> To: [email protected] >> Subject: Re: Hbase startup error: NoNode for /hbase/master after >> running out of space >> >>> 2011-06-06 18:40:11,833 WARN regionserver.HRegionServer - Unable to >>> read master address from ZooKeeper. Retrying. Error was: >> >> This means that your master didn't start. Can you post the log for the >> master? >> >> -Joey >> >> On Tue, Jun 7, 2011 at 9:47 AM, Zhong, Sheng <[email protected]> wrote: >>> >>> Hey, >>> >>> The system works perfect until receiving device out of space and more >>> space is added. Hope anyone here could provide help on the hbase >>> issue we got after our of space. I could bring up HDFS via one of >>> post found by manually modifying namenode's /var/hdfs/current/edits >>> file: $ printf "\xff\xff\xff\xee\xff" > edits. However, the issue >>> came to hbase startup. We could see HMaster and HRegion processes >>> running, but could not access Hbase. The errors is described below: >>> >>> I am appreciated by any help and suggestion. We are even okay to lose >>> recent unprocessed data. P.S: we're using apache hadoop 0.20.2 and >>> hbase 0.20.3, and zookeeper is running via zookeeper-3.2.2 (not >>> managed by Hbase). >>> >>> Hbase.log: >>> 2011-06-06 18:40:11,833 WARN regionserver.HRegionServer - Unable to >>> read master address from ZooKeeper. Retrying. Error was: >>> java.io.IOException: >>> org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode >>> = NoNode for /hbase/master >>> at >>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readAddressOrThrow >>> ( >>> Zo >>> oKeeperWrapper.java:394) >>> at >>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readMasterAddressO >>> r >>> Th >>> row(ZooKeeperWrapper.java:298) >>> at >>> org.apache.hadoop.hbase.regionserver.HRegionServer.getMaster(HRegionS >>> e >>> rv >>> er.java:1282) >>> at >>> org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HReg >>> i >>> on >>> Server.java:1314) >>> at >>> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer. >>> j >>> av >>> a:430) >>> at java.lang.Thread.run(Thread.java:619) >>> Caused by: org.apache.zookeeper.KeeperException$NoNodeException: >>> KeeperErrorCode = NoNode for /hbase/master >>> at >>> org.apache.zookeeper.KeeperException.create(KeeperException.java:102) >>> at >>> org.apache.zookeeper.KeeperException.create(KeeperException.java:42) >>> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:892) >>> at >>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readAddressOrThrow >>> ( >>> Zo >>> oKeeperWrapper.java:390) >>> ... 5 more >>> >>> Zookeeper.log: >>> 2011-06-06 21:48:36,174 - WARN >>> [ProcessThread:-1:PrepRequestProcessor@402] - Got exception when >>> processing sessionid:0x23067fc51970002 type:create cxid:0x19a9 >>> zxid:0xfffffffffffffffe txntype:unknown n/a >>> org.apache.zookeeper.KeeperException$NodeExistsException: >>> KeeperErrorCode = NodeExists >>> at >>> org.apache.zookeeper.server.PrepRequestProcessor.pRequest(PrepRequest >>> P >>> ro >>> cessor.java:245) >>> at >>> org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProce >>> s >>> so >>> r.java:114) >>> >>> >>> Regards, >>> Andy (Sheng) Zhong >>> >>> >>> This message, including any attachments, is the property of Sears Holdings >>> Corporation and/or one of its subsidiaries. It is confidential and may >>> contain proprietary or legally privileged information. If you are not the >>> intended recipient, please delete it without reading the contents. Thank >>> you. >>> >> >> >> >> -- >> Joseph Echeverria >> Cloudera, Inc. >> 443.305.9434 >> > > > > -- > Joseph Echeverria > Cloudera, Inc. > 443.305.9434 >
