on a devel system i don't have any monitoring in place. i just leave it and
hope the log files can give me hints when it breaks. in the master log i
see the log info below. the machine wasn't under any heavy work ... been
the same amount of load for the past 48 hours or so. for the tests
everything is on one machine...
2009-05-15 00:52:49,186 INFO org.apache.hadoop.hbase.master.BaseScanner:
RegionManager.metaScanner scanning meta region {regionname: .META.,,1,
startKey: <>,server: 127.0.0.1:60020}
2009-05-15 00:52:49,336 INFO org.apache.hadoop.hbase.master.BaseScanner:
RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0,
startKey: <>,server: 127.0.0.1:60020}
2009-05-15 00:54:43,959 INFO org.apache.hadoop.hbase.master.ServerManager:
127.0.0.1:60020 lease expired
2009-05-15 00:54:47,417 INFO
org.apache.hadoop.hbase.master.RegionServerOperation: process shutdown of
server 127.0.0.1:60020: logSplit: false, rootRescanned: false,
numberOfMetaRegions: 1, onlineMetaRegions.size(): 1
2009-05-15 00:54:50,049 INFO org.apache.hadoop.hbase.regionserver.HLog:
Splitting 27 log(s) in hdfs://
foo.bar.net:9000/hbase/log_127.0.0.1_1242258287822_60020
2009-05-15 00:55:54,866 INFO org.apache.hadoop.hbase.master.BaseScanner:
RegionManager.rootScanner scan of 1 row(s) of meta region {regionname:
-ROOT-,,0, startKey: <>, server: 127.0.0.1:60020} complete
2009-05-15 00:55:57,246 INFO org.apache.hadoop.hbase.master.BaseScanner:
RegionManager.metaScanner scan of 11 row(s) of meta region {regionname:
.META.,,1, startKey: <>, server: 127.0.0.1:60020} complete
2009-05-15 00:55:57,246 INFO org.apache.hadoop.hbase.master.BaseScanner: All
1 .META. region(s) scanned
2009-05-15 00:55:57,246 INFO org.apache.hadoop.hbase.master.BaseScanner:
RegionManager.metaScanner scanning meta region {regionname: .META.,,1,
startKey: <>,server: 127.0.0.1:60020}
2009-05-15 00:55:57,727 WARN org.apache.hadoop.hbase.master.BaseScanner:
Scan one META region: {regionname: .META.,,1, startKey: <>, server:
127.0.0.1:60020}
org.apache.hadoop.hbase.NotServingRegionException:
org.apache.hadoop.hbase.NotServingRegionException: .META.,,1
at
org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2076)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1710)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:912)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at
org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:94)
at
org.apache.hadoop.hbase.master.BaseScanner.scanRegion(BaseScanner.java:185)
at
org.apache.hadoop.hbase.master.MetaScanner.scanOneMetaRegion(MetaScanner.java:73)
at
org.apache.hadoop.hbase.master.MetaScanner.maintenanceScan(MetaScanner.java:129)
at
org.apache.hadoop.hbase.master.BaseScanner.chore(BaseScanner.java:137)
at org.apache.hadoop.hbase.Chore.run(Chore.java:65)
2009-05-15 00:55:57,977 INFO org.apache.hadoop.hbase.master.BaseScanner: All
1 .META. region(s) scanned
2009-05-15 00:56:57,252 INFO org.apache.hadoop.hbase.master.BaseScanner:
RegionManager.metaScanner scanning meta region {regionname: .META.,,1,
startKey: <>,
server: 127.0.0.1:60020}
2009-05-15 00:57:02,214 WARN org.apache.hadoop.hbase.master.BaseScanner:
Scan one META region: {regionname: .META.,,1, startKey: <>, server:
127.0.0.1:60
020}
org.apache.hadoop.hbase.NotServingRegionException:
org.apache.hadoop.hbase.NotServingRegionException: .META.,,1
at
org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2076)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1710)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:912)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at
org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:94)
at
org.apache.hadoop.hbase.master.BaseScanner.scanRegion(BaseScanner.java:185)
at
org.apache.hadoop.hbase.master.MetaScanner.scanOneMetaRegion(MetaScanner.java:73)
at
org.apache.hadoop.hbase.master.MetaScanner.maintenanceScan(MetaScanner.java:129)
On Fri, May 15, 2009 at 5:13 PM, Andrew Purtell <[email protected]> wrote:
> > 2009-05-15 00:55:53,090 WARN
> > org.apache.hadoop.hbase.regionserver.HRegionServer: unable to report to
> > master for 189261 milliseconds - retrying
>
> What do you see in the master log around this time?
>
> Was your cluster heavily loaded and/or in swap at this time? Do you have
> monitoring in place (i.e. sar (sysstat), Ganglia, Nagios) where you can go
> back in time and look at what was going on at the OS or network level at the
> time?
>
> Best regards,
>
> - Andy
>
>
>
>
> ________________________________
> From: Sasha Dolgy <[email protected]>
> To: [email protected]
> Sent: Friday, May 15, 2009 1:53:37 AM
> Subject: HRegionServer: Failed openScanner
>
> Hi there,
>
> Thought I would run a count 'table-name' to see how many records are in a
> table. The count didn't work so I went and took a look in the
> regionserver
> log file and see the below. Now, to be honest, not quite sure why it's
> just
> stopped working. The lines are from the start of the log file (from may
> 15). The log file from May 14 has no errors in it. Is there any way I can
> enable debugging to find out why it's stopped working? I had added this in
> the past 36 hours but it didn't appear to cause any problems and was
> working
> fine for atleast 24 hours:
>
> <property>
> <name>hbase.regionserver.class</name>
> <value>org.apache.hadoop.hbase.ipc.IndexedRegionInterface</value>
> <description>enable indexing</description>
> </property>
>
> <property>
> <name>hbase.regionserver.impl</name>
>
>
> <value>org.apache.hadoop.hbase.regionserver.tableindexed.IndexedRegionServer</value>
> <description>enable indexing</description>
> </property>
>
>
> $HBASE_HOME/bin/stop-all.sh and $HBASE_HOME/bin/start-all.sh fixed
> everything and it's all working fine again.
>
> 2009-05-15 00:55:53,090 WARN
> org.apache.hadoop.hbase.regionserver.HRegionServer: unable to report to
> master for 189261 milliseconds - retrying
> 2009-05-15 00:55:56,789 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer:
> MSG_CALL_SERVER_STARTUP:
> safeMode=false
> 2009-05-15 00:55:57,249 ERROR
> org.apache.hadoop.hbase.regionserver.HRegionServer: Failed openScanner
> org.apache.hadoop.hbase.NotServingRegionException: .META.,,1
> at
>
> org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2076)
> at
>
> org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1710)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
> at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:912)
> 2009-05-15 00:55:57,252 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
> handler 9 on 60020, call openScanner([...@233dfbc1, [...@3a5b4dfa,
> [...@405c7604,
> 9223372036854775807, null) from 127.0.0.1:47277: error:
> org.apache.hadoop.hbase.NotServingRegionException: .META.,,1
> org.apache.hadoop.hbase.NotServingRegionException: .META.,,1
> at
>
> org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2076)
> at
>
> org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1710)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
> at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:912)
> 2009-05-15 00:56:57,751 ERROR
> org.apache.hadoop.hbase.regionserver.HRegionServer: Failed openScanner
> org.apache.hadoop.hbase.NotServingRegionException: .META.,,1
> at
>
> org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2076)
> at
>
> org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1710)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
> at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:912)
>
> --
> Sasha Dolgy
> [email protected]
>
>
>
>
>
--
Sasha Dolgy
[email protected]
