Its hard to read logs out of an email and the snippet below is too short to tell for sure whats going on. I see this on the end 'Session Expired', which would seem to indicate that hbase lost its lease with zookeeper. Are these nodes working hard?
Tell us more about your cluster. How many nodes, what versions of hbase. St.Ack On Thu, May 27, 2010 at 8:15 PM, 梁景明 <[email protected]> wrote: > hi , my hbase crashed , here is the log ,and i wanna recover my region > server ,after trying > "hbase/bin/hbase-daemon.sh start regionserver" > > it failed to start, what 's the problem to shutdown the region server? > and how can i recover my region server , i dont want to reinstall all region > server > recover hbase,then import my backup; > > does region server to be recovered? > > thanks for any help; > > 2010-05-28 07:57:19,646 DEBUG > org.apache.hadoop.hbase.io.hfile.LruBlockCache: Cache Stats: Sizes: > Total=0.9473572MB (993376), Free=198.41515MB (208053360), Max=199.3625MB > (209046736), Counts: Blocks=4, Access=16584, Hit=16578, Miss=6, Evictions=0, > Evicted=0, Ratios: Hit Ratio=99.96381998062134%, Miss > Ratio=0.03617944894358516%, Evicted/Run=NaN > 2010-05-28 07:57:19,651 INFO > org.apache.hadoop.hbase.regionserver.HRegionServer: Scanner > 3212049130314122049 lease expired > 2010-05-28 07:57:19,729 INFO > org.apache.hadoop.hbase.regionserver.HRegionServer: Scanner > 3787094744445092555 lease expired > 2010-05-28 07:57:19,729 ERROR > org.apache.hadoop.hbase.regionserver.HRegionServer: > org.apache.hadoop.hbase.LeaseException: lease '3212049130314122049' does not > exist or has already expired > at org.apache.hadoop.hbase.Leases.renewLease(Leases.java:190) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1899) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1877) > at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:657) > at > org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915) > 2010-05-28 07:57:19,853 ERROR > org.apache.hadoop.hbase.regionserver.HRegionServer: > org.apache.hadoop.hbase.UnknownScannerException: Name: 3787094744445092555 > at > org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1889) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1877) > at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:657) > at > org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915) > 2010-05-28 07:57:19,901 WARN org.apache.zookeeper.ClientCnxn: Exception > closing session 0x128d263e1450001 to sun.nio.ch.selectionkeyi...@1878144 > java.io.IOException: Read error rc = -1 java.nio.DirectByteBuffer[pos=0 > lim=4 cap=4] > at org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:701) > at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:945) > 2010-05-28 07:57:20,244 INFO > org.apache.hadoop.hbase.regionserver.HRegionServer: Got ZooKeeper event, > state: Disconnected, type: None, path: null > 2010-05-28 07:57:20,433 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server > handler 17 on 60020, call next(3212049130314122049) from 192.168.1.115:49741: > error: org.apache.hadoop.hbase.LeaseException: lease '3212049130314122049' > does not exist or has already expired > org.apache.hadoop.hbase.LeaseException: lease '3212049130314122049' does not > exist or has already expired > at org.apache.hadoop.hbase.Leases.renewLease(Leases.java:190) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1899) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1877) > at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:657) > at > org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915) > 2010-05-28 07:57:20,539 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server > handler 21 on 60020, call next(3787094744445092555) from 192.168.1.115:49741: > error: org.apache.hadoop.hbase.UnknownScannerException: Name: > 3787094744445092555 > org.apache.hadoop.hbase.UnknownScannerException: Name: 3787094744445092555 > at > org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1889) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1877) > at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:657) > at > org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915) > 2010-05-28 07:57:21,627 INFO org.apache.zookeeper.ClientCnxn: Attempting > connection to server test2/192.168.1.115:2181 > 2010-05-28 07:57:22,881 INFO org.apache.zookeeper.ClientCnxn: Priming > connection to java.nio.channels.SocketChannel[connected local=/ > 192.168.1.115:37147 remote=test2/192.168.1.115:2181] > 2010-05-28 07:57:23,405 INFO org.apache.zookeeper.ClientCnxn: Server > connection successful > 2010-05-28 07:57:23,664 WARN org.apache.zookeeper.ClientCnxn: Exception > closing session 0x128d263e1450001 to sun.nio.ch.selectionkeyi...@192343b > java.io.IOException: Session Expired > at > org.apache.zookeeper.ClientCnxn$SendThread.readConnectResult(ClientCnxn.java:589) > at org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:709) > at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:945) > ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- > > 2010-05-28 09:29:41,286 INFO > org.apache.hadoop.hbase.regionserver.HRegionServer: HRegionServer started > at: 192.168.1.115:60020 > 2010-05-28 09:29:41,314 DEBUG > org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Read ZNode > /hbase/root-region-server got 192.168.1.115:60020 > 2010-05-28 09:29:41,318 INFO org.apache.hadoop.hbase.regionserver.StoreFile: > Allocating LruBlockCache with maximum size 199.4m > 2010-05-28 09:29:41,660 ERROR > org.apache.hadoop.hbase.regionserver.HRegionServer: Failed openScanner > org.apache.hadoop.hbase.NotServingRegionException: -ROOT-,,0 > at > org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2278) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1857) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:657) > at > org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915) >
