[Changed the title of the mail, probably not a REST server issue.]
This is the HBase client library embedded in the REST server warning that the
znode in ZooKeeper corresponding to the root regionserver went away. It waits
and waits for it to come back, but it never does.
Nothing in the master or regionserver logs around this time?
A restart of the REST server clears the problem? If so it seems a client side
or server side issue with ZooKeeper or the ZK wrapper.
On the server side, I have heard one thing that could cause this is if you have
a ZK ensemble in which the ensemble peers are running in stand-alone mode.
Perhaps the ZK server logs may shed some light.
You might also consider modifying hbase/conf/log4j.properties:
log4j.logger.org.apache.hadoop.hbase.zookeeper.ZKUtil=DEBUG
to get debug level ZK logging on the client side.
Best regards,
- Andy
--- On Thu, 10/21/10, Jack Levin <[email protected]> wrote:
> From: Jack Levin <[email protected]>
> Subject: REST server disconnects
[...]
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers:
> Sleeping 1000ms, waiting for root region.
> 2010-10-21 10:15:10,846 DEBUG
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers:
> locateRegionInMeta attempt 0 of 10 failed; retrying after
> sleep of
> 1000 because: Timed out trying to locate root region
> 2010-10-21 10:15:10,847 DEBUG
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper:
> <rdaf1,rdaf2,rdaf3,mtab1,mtab2:/hbase,org.apache.hadoop.hbase.client.HConnectionManager>Trying
> to read /hbase/root-region-server
> 2010-10-21 10:15:10,848 DEBUG
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper:
> <rdaf1,rdaf2,rdaf3,mtab1,mtab2:/hbase,org.apache.hadoop.hbase.client.HConnectionManager>Failed
> to read org.apache.zookeeper.KeeperException$NoNodeExcep
> tion: KeeperErrorCode = NoNode for
> /hbase/root-region-server
> 2010-10-21 10:15:10,849 DEBUG
> org.apache.hadoop.hbase.client.
> ....
> 2010-10-21 11:39:17,569 DEBUG
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers:
> Sleeping 1000ms, waiting for root region.
> 2010-10-21 11:39:18,569 DEBUG
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper:
> <rdaf1,rdaf2,rdaf3,mtab1,mtab2:/hbase,org.apache.hadoop.hbase.client.HConnectionManager>Trying
> to read /hbase/root-region-server
> 2010-10-21 11:39:18,570 DEBUG
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper:
> <rdaf1,rdaf2,rdaf3,mtab1,mtab2:/hbase,org.apache.hadoop.hbase.client.HConnectionManager>Failed
> to read
> org.apache.zookeeper.KeeperException$NoNodeException:
> KeeperErrorCode = NoNode for /hbase/root-region-server
