Which hbase version is used ? Can you pastebin region server log for the crashed servers ?
Thanks On Thu, Dec 19, 2013 at 6:14 PM, GuoWei <[email protected]> wrote: > Hi all, > > Recently our hbase region server crashed. But after we restart the hbase > and hdfs, three of my region servers crashed immediately. > And the following error found in hdfs log: > > 13/12/19 11:06:09 INFO zookeeper.ZooKeeper: Client > environment:java.io.tmpdir=/tmp > 13/12/19 11:06:09 INFO zookeeper.ZooKeeper: Client > environment:java.compiler=<NA> > 13/12/19 11:06:09 INFO zookeeper.ZooKeeper: Client environment:os.name > =Linux > 13/12/19 11:06:09 INFO zookeeper.ZooKeeper: Client > environment:os.arch=amd64 > 13/12/19 11:06:09 INFO zookeeper.ZooKeeper: Client > environment:os.version=2.6.32-358.el6.x86_64 > 13/12/19 11:06:09 INFO zookeeper.ZooKeeper: Client environment:user.name > =hbase > 13/12/19 11:06:09 INFO zookeeper.ZooKeeper: Client > environment:user.home=/var/run/hbase > 13/12/19 11:06:09 INFO zookeeper.ZooKeeper: Client > environment:user.dir=/root > 13/12/19 11:06:09 INFO zookeeper.ZooKeeper: Initiating client connection, > connectString=slave02:2181,slave03:2181,slave04:2181,slave05:2181,slave01:2181 > sessionTimeout=90000 watcher=hconnection > 13/12/19 11:06:09 INFO zookeeper.RecoverableZooKeeper: The identifier of > this process is 18750@master > 13/12/19 11:06:09 INFO zookeeper.ClientCnxn: Opening socket connection to > server slave05/192.168.1.205:2181. Will not attempt to authenticate using > SASL (Unable to locate a login configuration) > 13/12/19 11:06:09 INFO zookeeper.ClientCnxn: Socket connection established > to slave05/192.168.1.205:2181, initiating session > 13/12/19 11:06:09 INFO zookeeper.ClientCnxn: Session establishment > complete on server slave05/192.168.1.205:2181, sessionid = > 0x143088237e90076, negotiated timeout = 90000 > 13/12/19 11:06:09 WARN conf.Configuration: hadoop.native.lib is > deprecated. Instead, use io.native.lib.available > > Exception in thread "main" > org.apache.hadoop.hbase.client.NoServerForRegionException: Unable to find > region for after 10 tries. > at > org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:980) > at > org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:885) > at > org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:846) > at > org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:271) > at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:211) > at > org.apache.hadoop.hbase.util.HBaseFsck.connect(HBaseFsck.java:281) > at org.apache.hadoop.hbase.util.HBaseFsck.exec(HBaseFsck.java:3645) > at org.apache.hadoop.hbase.util.HBaseFsck.run(HBaseFsck.java:3486) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84) > at org.apache.hadoop.hbase.util.HBaseFsck.main(HBaseFsck.java:3480) > > > no error logs found in hbase log files. > > Currently the problem is I can list hbase tables . But I can not scan any > tables in hbase. > > > Thanks > Brad
