hi ,hadoop 0.20.1 and hbase 0.20.4 two nodes . yes, they work hard, they also severed the web. they runs ok for one month . is there a way to rescue the regionserver?
2010/5/30 Stack <[email protected]> > Its hard to read logs out of an email and the snippet below is too > short to tell for sure whats going on. I see this on the end 'Session > Expired', which would seem to indicate that hbase lost its lease with > zookeeper. Are these nodes working hard? > > Tell us more about your cluster. How many nodes, what versions of hbase. > > St.Ack > > On Thu, May 27, 2010 at 8:15 PM, 梁景明 <[email protected]> wrote: > > hi , my hbase crashed , here is the log ,and i wanna recover my region > > server ,after trying > > "hbase/bin/hbase-daemon.sh start regionserver" > > > > it failed to start, what 's the problem to shutdown the region server? > > and how can i recover my region server , i dont want to reinstall all > region > > server > > recover hbase,then import my backup; > > > > does region server to be recovered? > > > > thanks for any help; > > > > 2010-05-28 07:57:19,646 DEBUG > > org.apache.hadoop.hbase.io.hfile.LruBlockCache: Cache Stats: Sizes: > > Total=0.9473572MB (993376), Free=198.41515MB (208053360), Max=199.3625MB > > (209046736), Counts: Blocks=4, Access=16584, Hit=16578, Miss=6, > Evictions=0, > > Evicted=0, Ratios: Hit Ratio=99.96381998062134%, Miss > > Ratio=0.03617944894358516%, Evicted/Run=NaN > > 2010-05-28 07:57:19,651 INFO > > org.apache.hadoop.hbase.regionserver.HRegionServer: Scanner > > 3212049130314122049 lease expired > > 2010-05-28 07:57:19,729 INFO > > org.apache.hadoop.hbase.regionserver.HRegionServer: Scanner > > 3787094744445092555 lease expired > > 2010-05-28 07:57:19,729 ERROR > > org.apache.hadoop.hbase.regionserver.HRegionServer: > > org.apache.hadoop.hbase.LeaseException: lease '3212049130314122049' does > not > > exist or has already expired > > at org.apache.hadoop.hbase.Leases.renewLease(Leases.java:190) > > at > > > org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1899) > > at > > > org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1877) > > at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source) > > at > > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > > at java.lang.reflect.Method.invoke(Method.java:597) > > at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:657) > > at > > org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915) > > 2010-05-28 07:57:19,853 ERROR > > org.apache.hadoop.hbase.regionserver.HRegionServer: > > org.apache.hadoop.hbase.UnknownScannerException: Name: > 3787094744445092555 > > at > > > org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1889) > > at > > > org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1877) > > at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source) > > at > > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > > at java.lang.reflect.Method.invoke(Method.java:597) > > at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:657) > > at > > org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915) > > 2010-05-28 07:57:19,901 WARN org.apache.zookeeper.ClientCnxn: Exception > > closing session 0x128d263e1450001 to sun.nio.ch.selectionkeyi...@1878144 > > java.io.IOException: Read error rc = -1 java.nio.DirectByteBuffer[pos=0 > > lim=4 cap=4] > > at > org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:701) > > at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:945) > > 2010-05-28 07:57:20,244 INFO > > org.apache.hadoop.hbase.regionserver.HRegionServer: Got ZooKeeper event, > > state: Disconnected, type: None, path: null > > 2010-05-28 07:57:20,433 INFO org.apache.hadoop.ipc.HBaseServer: IPC > Server > > handler 17 on 60020, call next(3212049130314122049) from > 192.168.1.115:49741: > > error: org.apache.hadoop.hbase.LeaseException: lease > '3212049130314122049' > > does not exist or has already expired > > org.apache.hadoop.hbase.LeaseException: lease '3212049130314122049' does > not > > exist or has already expired > > at org.apache.hadoop.hbase.Leases.renewLease(Leases.java:190) > > at > > > org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1899) > > at > > > org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1877) > > at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source) > > at > > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > > at java.lang.reflect.Method.invoke(Method.java:597) > > at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:657) > > at > > org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915) > > 2010-05-28 07:57:20,539 INFO org.apache.hadoop.ipc.HBaseServer: IPC > Server > > handler 21 on 60020, call next(3787094744445092555) from > 192.168.1.115:49741: > > error: org.apache.hadoop.hbase.UnknownScannerException: Name: > > 3787094744445092555 > > org.apache.hadoop.hbase.UnknownScannerException: Name: > 3787094744445092555 > > at > > > org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1889) > > at > > > org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1877) > > at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source) > > at > > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > > at java.lang.reflect.Method.invoke(Method.java:597) > > at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:657) > > at > > org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915) > > 2010-05-28 07:57:21,627 INFO org.apache.zookeeper.ClientCnxn: Attempting > > connection to server test2/192.168.1.115:2181 > > 2010-05-28 07:57:22,881 INFO org.apache.zookeeper.ClientCnxn: Priming > > connection to java.nio.channels.SocketChannel[connected local=/ > > 192.168.1.115:37147 remote=test2/192.168.1.115:2181] > > 2010-05-28 07:57:23,405 INFO org.apache.zookeeper.ClientCnxn: Server > > connection successful > > 2010-05-28 07:57:23,664 WARN org.apache.zookeeper.ClientCnxn: Exception > > closing session 0x128d263e1450001 to sun.nio.ch.selectionkeyi...@192343b > > java.io.IOException: Session Expired > > at > > > org.apache.zookeeper.ClientCnxn$SendThread.readConnectResult(ClientCnxn.java:589) > > at > org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:709) > > at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:945) > > > ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- > > > > 2010-05-28 09:29:41,286 INFO > > org.apache.hadoop.hbase.regionserver.HRegionServer: HRegionServer started > > at: 192.168.1.115:60020 > > 2010-05-28 09:29:41,314 DEBUG > > org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Read ZNode > > /hbase/root-region-server got 192.168.1.115:60020 > > 2010-05-28 09:29:41,318 INFO > org.apache.hadoop.hbase.regionserver.StoreFile: > > Allocating LruBlockCache with maximum size 199.4m > > 2010-05-28 09:29:41,660 ERROR > > org.apache.hadoop.hbase.regionserver.HRegionServer: Failed openScanner > > org.apache.hadoop.hbase.NotServingRegionException: -ROOT-,,0 > > at > > > org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2278) > > at > > > org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1857) > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > at > > > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > > at > > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > > at java.lang.reflect.Method.invoke(Method.java:597) > > at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:657) > > at > > org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915) > > >
