Hi all,
Recently our hbase region server crashed. But after we restart the hbase and
hdfs, three of my region servers crashed immediately.
And the following error found in hdfs log:
13/12/19 11:06:09 INFO zookeeper.ZooKeeper: Client
environment:java.io.tmpdir=/tmp
13/12/19 11:06:09 INFO zookeeper.ZooKeeper: Client
environment:java.compiler=<NA>
13/12/19 11:06:09 INFO zookeeper.ZooKeeper: Client environment:os.name=Linux
13/12/19 11:06:09 INFO zookeeper.ZooKeeper: Client environment:os.arch=amd64
13/12/19 11:06:09 INFO zookeeper.ZooKeeper: Client
environment:os.version=2.6.32-358.el6.x86_64
13/12/19 11:06:09 INFO zookeeper.ZooKeeper: Client environment:user.name=hbase
13/12/19 11:06:09 INFO zookeeper.ZooKeeper: Client
environment:user.home=/var/run/hbase
13/12/19 11:06:09 INFO zookeeper.ZooKeeper: Client environment:user.dir=/root
13/12/19 11:06:09 INFO zookeeper.ZooKeeper: Initiating client connection,
connectString=slave02:2181,slave03:2181,slave04:2181,slave05:2181,slave01:2181
sessionTimeout=90000 watcher=hconnection
13/12/19 11:06:09 INFO zookeeper.RecoverableZooKeeper: The identifier of this
process is 18750@master
13/12/19 11:06:09 INFO zookeeper.ClientCnxn: Opening socket connection to
server slave05/192.168.1.205:2181. Will not attempt to authenticate using SASL
(Unable to locate a login configuration)
13/12/19 11:06:09 INFO zookeeper.ClientCnxn: Socket connection established to
slave05/192.168.1.205:2181, initiating session
13/12/19 11:06:09 INFO zookeeper.ClientCnxn: Session establishment complete on
server slave05/192.168.1.205:2181, sessionid = 0x143088237e90076, negotiated
timeout = 90000
13/12/19 11:06:09 WARN conf.Configuration: hadoop.native.lib is deprecated.
Instead, use io.native.lib.available
Exception in thread "main"
org.apache.hadoop.hbase.client.NoServerForRegionException: Unable to find
region for after 10 tries.
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:980)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:885)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:846)
at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:271)
at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:211)
at org.apache.hadoop.hbase.util.HBaseFsck.connect(HBaseFsck.java:281)
at org.apache.hadoop.hbase.util.HBaseFsck.exec(HBaseFsck.java:3645)
at org.apache.hadoop.hbase.util.HBaseFsck.run(HBaseFsck.java:3486)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
at org.apache.hadoop.hbase.util.HBaseFsck.main(HBaseFsck.java:3480)
no error logs found in hbase log files.
Currently the problem is I can list hbase tables . But I can not scan any
tables in hbase.
Thanks
Brad
