Joey,
Yeah, the master did not start properly, even though I could see the process
and listener port: 60020, but could not have access. Thank you for looking into
this. Thanks, - Andy Zhong
hbase-hadoop-master-XXX.log:
Mon Jun 6 21:53:14 CDT 2011 Starting master on hofvmecln21.ecom.sears.com
ulimit -n 1024
Mon Jun 6 22:08:04 CDT 2011 Starting master on hofvmecln21.ecom.sears.com
ulimit -n 1024
Hbase.log:
2011-06-07 09:33:16,996 WARN regionserver.HRegionServer - Unable to read
master address from ZooKeeper. Retrying. Error was:
java.io.IOException: org.apache.zookeeper.KeeperException$NoNodeException:
KeeperErrorCode = NoNode for /hbase/master
at
org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readAddressOrThrow(ZooKeeperWrapper.java:394)
at
org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readMasterAddressOrThrow(ZooKeeperWrapper.java:298)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.getMaster(HRegionServer.java:1282)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:1314)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:430)
at java.lang.Thread.run(Thread.java:619)
Caused by: org.apache.zookeeper.KeeperException$NoNodeException:
KeeperErrorCode = NoNode for /hbase/master
at org.apache.zookeeper.KeeperException.create(KeeperException.java:102)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:892)
at
org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readAddressOrThrow(ZooKeeperWrapper.java:390)
... 5 more
2011-06-07 09:33:17,999 WARN regionserver.HRegionServer - Unable to read
master address from ZooKeeper. Retrying. Error was:
java.io.IOException: org.apache.zookeeper.KeeperException$NoNodeException:
KeeperErrorCode = NoNode for /hbase/master
at
org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readAddressOrThrow(ZooKeeperWrapper.java:394)
at
org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readMasterAddressOrThrow(ZooKeeperWrapper.java:298)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.getMaster(HRegionServer.java:1282)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:1314)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:430)
at java.lang.Thread.run(Thread.java:619)
Caused by: org.apache.zookeeper.KeeperException$NoNodeException:
KeeperErrorCode = NoNode for /hbase/master
at org.apache.zookeeper.KeeperException.create(KeeperException.java:102)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:892)
at
org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readAddressOrThrow(ZooKeeperWrapper.java:390)
... 5 more
-----Original Message-----
From: Joey Echeverria [mailto:[email protected]]
Sent: Tuesday, June 07, 2011 9:30 AM
To: [email protected]
Subject: Re: Hbase startup error: NoNode for /hbase/master after running out of
space
> 2011-06-06 18:40:11,833 WARN regionserver.HRegionServer - Unable to
> read master address from ZooKeeper. Retrying. Error was:
This means that your master didn't start. Can you post the log for the master?
-Joey
On Tue, Jun 7, 2011 at 9:47 AM, Zhong, Sheng <[email protected]> wrote:
>
> Hey,
>
> The system works perfect until receiving device out of space and more
> space is added. Hope anyone here could provide help on the hbase issue
> we got after our of space. I could bring up HDFS via one of post found
> by manually modifying namenode's /var/hdfs/current/edits file: $
> printf "\xff\xff\xff\xee\xff" > edits. However, the issue came to
> hbase startup. We could see HMaster and HRegion processes running, but
> could not access Hbase. The errors is described below:
>
> I am appreciated by any help and suggestion. We are even okay to lose
> recent unprocessed data. P.S: we're using apache hadoop 0.20.2 and
> hbase 0.20.3, and zookeeper is running via zookeeper-3.2.2 (not
> managed by Hbase).
>
> Hbase.log:
> 2011-06-06 18:40:11,833 WARN regionserver.HRegionServer - Unable to
> read master address from ZooKeeper. Retrying. Error was:
> java.io.IOException:
> org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode
> = NoNode for /hbase/master
> at
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readAddressOrThrow(
> Zo
> oKeeperWrapper.java:394)
> at
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readMasterAddressOr
> Th
> row(ZooKeeperWrapper.java:298)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.getMaster(HRegionSe
> rv
> er.java:1282)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegi
> on
> Server.java:1314)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.j
> av
> a:430)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: org.apache.zookeeper.KeeperException$NoNodeException:
> KeeperErrorCode = NoNode for /hbase/master
> at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:102)
> at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:892)
> at
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readAddressOrThrow(
> Zo
> oKeeperWrapper.java:390)
> ... 5 more
>
> Zookeeper.log:
> 2011-06-06 21:48:36,174 - WARN
> [ProcessThread:-1:PrepRequestProcessor@402] - Got exception when
> processing sessionid:0x23067fc51970002 type:create cxid:0x19a9
> zxid:0xfffffffffffffffe txntype:unknown n/a
> org.apache.zookeeper.KeeperException$NodeExistsException:
> KeeperErrorCode = NodeExists
> at
> org.apache.zookeeper.server.PrepRequestProcessor.pRequest(PrepRequestP
> ro
> cessor.java:245)
> at
> org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProces
> so
> r.java:114)
>
>
> Regards,
> Andy (Sheng) Zhong
>
>
> This message, including any attachments, is the property of Sears Holdings
> Corporation and/or one of its subsidiaries. It is confidential and may
> contain proprietary or legally privileged information. If you are not the
> intended recipient, please delete it without reading the contents. Thank you.
>
--
Joseph Echeverria
Cloudera, Inc.
443.305.9434
