I've been testing a nuch on HBase and I'm running into an issue with
it when it's been running for a while.
I'm running hbase 0.19 on a single local host pseudo distributed
hadoop 0.19 cluster.
After some time I get the following in my logs:
2009-02-24 09:31:18,664 INFO
org.apache.hadoop.hbase.master.BaseScanner: All 1 .META. region(s)
scanned
2009-02-24 09:31:18,664 INFO
org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner
scanning meta region {regionname: .META.,,1, startKey: <>, server:
127.0.0.1:40076}
2009-02-24 09:31:24,293 INFO
org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner
scan of 1 row(s) of meta region {regionname: -ROOT-,,0, startKey: <>,
server: 127.0.0.1:40
076} complete
2009-02-24 09:31:24,293 INFO
org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner
scanning meta region {regionname: -ROOT-,,0, startKey: <>, server:
127.0.0.1:40076}
2009-02-24 09:31:27,217 INFO
org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner
scan of 2 row(s) of meta region {regionname: .META.,,1, startKey: <>,
server: 127.0.0.1:40
076} complete
2009-02-24 09:31:27,217 INFO
org.apache.hadoop.hbase.master.BaseScanner: All 1 .META. region(s)
scanned
2009-02-24 09:31:55,533 INFO
org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner
scan of 1 row(s) of meta region {regionname: -ROOT-,,0, startKey: <>,
server: 127.0.0.1:40
076} complete
2009-02-24 09:32:18,665 INFO
org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner
scanning meta region {regionname: .META.,,1, startKey: <>, server:
127.0.0.1:40076}
2009-02-24 09:32:35,749 INFO
org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner
scanning meta region {regionname: -ROOT-,,0, startKey: <>, server:
127.0.0.1:40076}
2009-02-24 09:32:38,957 INFO
org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner
scan of 2 row(s) of meta region {regionname: .META.,,1, startKey: <>,
server: 127.0.0.1:40
076} complete
2009-02-24 09:32:38,957 INFO
org.apache.hadoop.hbase.master.BaseScanner: All 1 .META. region(s)
scanned
2009-02-24 09:33:18,771 INFO
org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner
scanning meta region {regionname: .META.,,1, startKey: <>, server:
127.0.0.1:40076}
2009-02-24 09:33:54,889 INFO
org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner
scan of 1 row(s) of meta region {regionname: -ROOT-,,0, startKey: <>,
server: 127.0.0.1:40
076} complete
2009-02-24 09:33:57,387 INFO
org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner
scanning meta region {regionname: -ROOT-,,0, startKey: <>, server:
127.0.0.1:40076}
2009-02-24 09:34:41,579 INFO
org.apache.hadoop.hbase.master.ServerManager: 127.0.0.1:40076 lease
expired
2009-02-24 09:34:53,440 INFO
org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner
scan of 2 row(s) of meta region {regionname: .META.,,1, startKey: <>,
server: 127.0.0.1:40
076} complete
2009-02-24 09:34:53,441 INFO
org.apache.hadoop.hbase.master.BaseScanner: All 1 .META. region(s)
scanned
2009-02-24 09:34:53,441 INFO
org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner
scanning meta region {regionname: .META.,,1, startKey: <>, server:
127.0.0.1:40076}
2009-02-24 09:35:38,720 INFO
org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner
scan of 2 row(s) of meta region {regionname: .META.,,1, startKey: <>,
server: 127.0.0.1:40
076} complete
2009-02-24 09:35:38,720 INFO
org.apache.hadoop.hbase.master.BaseScanner: All 1 .META. region(s)
scanned
2009-02-24 09:35:56,951 INFO
org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner
scanning meta region {regionname: .META.,,1, startKey: <>, server:
127.0.0.1:40076}
2009-02-24 09:36:05,667 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer:
MSG_CALL_SERVER_STARTUP
2009-02-24 09:36:05,668 INFO
org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner
scan of 1 row(s) of meta region {regionname: -ROOT-,,0, startKey: <>,
server: 127.0.0.1:40
076} complete
2009-02-24 09:36:05,884 INFO
org.apache.hadoop.hbase.master.RegionServerOperation: process shutdown
of server 127.0.0.1:40076: logSplit: false, rootRescanned: false,
numberOfMetaRegions: 1,
onlineMetaRegions.size(): 1
2009-02-24 09:36:17,689 INFO
org.apache.hadoop.hbase.regionserver.HLog: Splitting 1 log(s) in
hdfs://localhost:9000/hbase/log_127.0.0.1_1235427119323_40076
2009-02-24 09:36:25,034 INFO
org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner
scan of 2 row(s) of meta region {regionname: .META.,,1, startKey: <>,
server: 127.0.0.1:40
076} complete
2009-02-24 09:36:25,034 INFO
org.apache.hadoop.hbase.master.BaseScanner: All 1 .META. region(s)
scanned
2009-02-24 09:36:36,598 INFO
org.apache.hadoop.hbase.regionserver.HLog: log file splitting
completed for hdfs://localhost:9000/hbase/
log_127.0.0.1_1235427119323_40076
2009-02-24 09:36:56,953 INFO
org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner
scanning meta region {regionname: .META.,,1, startKey: <>, server:
127.0.0.1:40076}
2009-02-24 09:37:05,279 ERROR
org.apache.hadoop.hbase.regionserver.HRegionServer: Failed openScanner
org.apache.hadoop.hbase.NotServingRegionException: .META.,,1
at
org
.apache
.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:
2065)
at
org
.apache
.hadoop
.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1699)
at sun.reflect.GeneratedMethodAccessor22.invoke(Unknown Source)
at
sun
.reflect
.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:
25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.HBaseRPC
$Server.call(HBaseRPC.java:632)
at org.apache.hadoop.hbase.ipc.HBaseServer
$Handler.run(HBaseServer.java:895)
2009-02-24 09:37:06,102 INFO org.apache.hadoop.ipc.HBaseServer: IPC
Server handler 4 on 40076, call openScanner([...@1ac5a78, [...@6799dc,
[...@1a66cfc, 9223372036854775807, null) from 127.0.0.1
:50139: error:
org.apache.hadoop.hbase.NotServingRegionException: .META.,,1
org.apache.hadoop.hbase.NotServingRegionException: .META.,,1
at
org
.apache
.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:
2065)
at
org
.apache
.hadoop
.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1699)
at sun.reflect.GeneratedMethodAccessor22.invoke(Unknown Source)
at
sun
.reflect
.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:
25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.HBaseRPC
$Server.call(HBaseRPC.java:632)
at org.apache.hadoop.hbase.ipc.HBaseServer
$Handler.run(HBaseServer.java:895)
2009-02-24 09:37:07,051 WARN
org.apache.hadoop.hbase.master.BaseScanner: Scan one META region:
{regionname: .META.,,1, startKey: <>, server: 127.0.0.1:40076}
org.apache.hadoop.hbase.NotServingRegionException:
org.apache.hadoop.hbase.NotServingRegionException: .META.,,1
at
org
.apache
.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:
2065)
at
org
.apache
.hadoop
.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1699)
at sun.reflect.GeneratedMethodAccessor22.invoke(Unknown Source)
at
sun
.reflect
.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:
25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.HBaseRPC
$Server.call(HBaseRPC.java:632)
at org.apache.hadoop.hbase.ipc.HBaseServer
$Handler.run(HBaseServer.java:895)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun
.reflect
.NativeConstructorAccessorImpl
.newInstance(NativeConstructorAccessorImpl.java:39)
at
sun
.reflect
.DelegatingConstructorAccessorImpl
.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:
513)
at
org
.apache
.hadoop
.hbase
.RemoteExceptionHandler
.decodeRemoteException(RemoteExceptionHandler.java:95)
at
org.apache.hadoop.hbase.master.BaseScanner.scanRegion(BaseScanner.java:
185)
at
org
.apache
.hadoop.hbase.master.MetaScanner.scanOneMetaRegion(MetaScanner.java:73)
at
org
.apache
.hadoop.hbase.master.MetaScanner.maintenanceScan(MetaScanner.java:129)
at
org.apache.hadoop.hbase.master.BaseScanner.chore(BaseScanner.java:137)
at org.apache.hadoop.hbase.Chore.run(Chore.java:65)
2009-02-24 09:37:07,230 INFO
org.apache.hadoop.hbase.master.BaseScanner: All 1 .META. region(s)
scanned
2009-02-24 09:37:50,642 INFO
org.apache.hadoop.hbase.regionserver.HRegion: Closed .META.,,1
2009-02-24 09:37:56,955 INFO
org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner
scanning meta region {regionname: .META.,,1, startKey: <>, server:
127.0.0.1:40076}
2009-02-24 09:37:57,126 ERROR
org.apache.hadoop.hbase.regionserver.HRegionServer: Failed openScanner
org.apache.hadoop.hbase.NotServingRegionException: .META.,,1
at
org
.apache
.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:
2065)
at
org
.apache
.hadoop
.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1699)
at sun.reflect.GeneratedMethodAccessor22.invoke(Unknown Source)
at
sun
.reflect
.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:
25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.HBaseRPC
$Server.call(HBaseRPC.java:632)
at org.apache.hadoop.hbase.ipc.HBaseServer
$Handler.run(HBaseServer.java:895)
2009-02-24 09:37:59,129 INFO org.apache.hadoop.ipc.HBaseServer: IPC
Server handler 2 on 40076, call openScanner([...@155bf4e, [...@82b436,
[...@1056757, 9223372036854775807, null) from 127.0.0.1
:35804: error:
org.apache.hadoop.hbase.NotServingRegionException: .META.,,1
org.apache.hadoop.hbase.NotServingRegionException: .META.,,1
at
org
.apache
.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:
2065)
at
org
.apache
.hadoop
.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1699)
at sun.reflect.GeneratedMethodAccessor22.invoke(Unknown Source)
at
sun
.reflect
.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:
25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.HBaseRPC
$Server.call(HBaseRPC.java:632)
at org.apache.hadoop.hbase.ipc.HBaseServer
$Handler.run(HBaseServer.java:895)
2009-02-24 09:38:00,412 WARN
org.apache.hadoop.hbase.master.BaseScanner: Scan one META region:
{regionname: .META.,,1, startKey: <>, server: 127.0.0.1:40076}
org.apache.hadoop.hbase.NotServingRegionException:
org.apache.hadoop.hbase.NotServingRegionException: .META.,,1
at
org
.apache
.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:
2065)
at
org
.apache
.hadoop
.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1699)
at sun.reflect.GeneratedMethodAccessor22.invoke(Unknown Source)
at
sun
.reflect
.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:
25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.HBaseRPC
$Server.call(HBaseRPC.java:632)
at org.apache.hadoop.hbase.ipc.HBaseServer
$Handler.run(HBaseServer.java:895)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun
.reflect
.NativeConstructorAccessorImpl
.newInstance(NativeConstructorAccessorImpl.java:39)
at
sun
.reflect
.DelegatingConstructorAccessorImpl
.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:
513)
at
org
.apache
.hadoop
.hbase
.RemoteExceptionHandler
.decodeRemoteException(RemoteExceptionHandler.java:95)
at
org.apache.hadoop.hbase.master.BaseScanner.scanRegion(BaseScanner.java:
185)
at
org
.apache
.hadoop.hbase.master.MetaScanner.scanOneMetaRegion(MetaScanner.java:73)
at
org
.apache
.hadoop.hbase.master.MetaScanner.maintenanceScan(MetaScanner.java:129)
at
org.apache.hadoop.hbase.master.BaseScanner.chore(BaseScanner.java:137)
at org.apache.hadoop.hbase.Chore.run(Chore.java:65)
This just repeats forever, until I restart HBase.
Any help would be greatly appreciated.
Andrew
