> 2009-05-15 00:55:53,090 WARN > org.apache.hadoop.hbase.regionserver.HRegionServer: unable to report to > master for 189261 milliseconds - retrying
What do you see in the master log around this time? Was your cluster heavily loaded and/or in swap at this time? Do you have monitoring in place (i.e. sar (sysstat), Ganglia, Nagios) where you can go back in time and look at what was going on at the OS or network level at the time? Best regards, - Andy ________________________________ From: Sasha Dolgy <[email protected]> To: [email protected] Sent: Friday, May 15, 2009 1:53:37 AM Subject: HRegionServer: Failed openScanner Hi there, Thought I would run a count 'table-name' to see how many records are in a table. The count didn't work so I went and took a look in the regionserver log file and see the below. Now, to be honest, not quite sure why it's just stopped working. The lines are from the start of the log file (from may 15). The log file from May 14 has no errors in it. Is there any way I can enable debugging to find out why it's stopped working? I had added this in the past 36 hours but it didn't appear to cause any problems and was working fine for atleast 24 hours: <property> <name>hbase.regionserver.class</name> <value>org.apache.hadoop.hbase.ipc.IndexedRegionInterface</value> <description>enable indexing</description> </property> <property> <name>hbase.regionserver.impl</name> <value>org.apache.hadoop.hbase.regionserver.tableindexed.IndexedRegionServer</value> <description>enable indexing</description> </property> $HBASE_HOME/bin/stop-all.sh and $HBASE_HOME/bin/start-all.sh fixed everything and it's all working fine again. 2009-05-15 00:55:53,090 WARN org.apache.hadoop.hbase.regionserver.HRegionServer: unable to report to master for 189261 milliseconds - retrying 2009-05-15 00:55:56,789 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_CALL_SERVER_STARTUP: safeMode=false 2009-05-15 00:55:57,249 ERROR org.apache.hadoop.hbase.regionserver.HRegionServer: Failed openScanner org.apache.hadoop.hbase.NotServingRegionException: .META.,,1 at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2076) at org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1710) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632) at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:912) 2009-05-15 00:55:57,252 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 9 on 60020, call openScanner([...@233dfbc1, [...@3a5b4dfa, [...@405c7604, 9223372036854775807, null) from 127.0.0.1:47277: error: org.apache.hadoop.hbase.NotServingRegionException: .META.,,1 org.apache.hadoop.hbase.NotServingRegionException: .META.,,1 at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2076) at org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1710) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632) at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:912) 2009-05-15 00:56:57,751 ERROR org.apache.hadoop.hbase.regionserver.HRegionServer: Failed openScanner org.apache.hadoop.hbase.NotServingRegionException: .META.,,1 at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2076) at org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1710) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632) at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:912) -- Sasha Dolgy [email protected]
