More updates in case it can help: In the UI I get the following regions mapping for .META.: Table Regions
Name Region Server Start Key End Key Requests .META.,,1.1028785192 172.25.98.66:11401 - When I run 'hbase hbck' I fail, with the following error which indicates is thinks this region is placed in a different server (65 and not 66) 12/11/08 14:23:02 DEBUG client.HConnectionManager$HConnectionImplementation: Removed .META.,,1.1028785192 for tableName=.META. from cache because of 12/11/08 14:23:02 DEBUG client.HConnectionManager$HConnectionImplementation: Cached location for .META.,,1.1028785192 is 172.25.98.65:11411 Exception in thread "main" org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=10, exceptions: Thu Nov 08 14:22:23 UTC 2012, org.apache.hadoop.hbase.client.ScannerCallable@39ed1b0b, org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: Region is not online: .META.,,1 How can there be such a mismatch? On Nov 7, 2012, at 8:08 AM, Mesika, Asaf wrote: > After restart, running hbck results in: > 12/11/07 06:07:28 DEBUG client.MetaScanner: Scanning .META. starting at row= > for max=2147483647 rows using > org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@51b4a74b > > 12/11/07 06:07:28 INFO util.HBaseFsck: getHTableDescriptors == tableNames => > [] > Exception in thread "main" java.lang.NullPointerException > at > org.apache.hadoop.hbase.util.HBaseFsck.reportTablesInFlux(HBaseFsck.java:560) > at > org.apache.hadoop.hbase.util.HBaseFsck.onlineConsistencyRepair(HBaseFsck.java:346) > at org.apache.hadoop.hbase.util.HBaseFsck.onlineHbck(HBaseFsck.java:382) > at org.apache.hadoop.hbase.util.HBaseFsck.main(HBaseFsck.java:3120) > > HBase is version 0.94.0 > > > On Nov 7, 2012, at 7:28 AM, Mesika, Asaf wrote: > >> Hi, >> >> I have a cluster with 10 RS and 1 Master. >> From some reason, any call to the HBase fails. I ran hbck and got the same >> failure: >> >> Tue Nov 06 13:27:07 UTC 2012, >> org.apache.hadoop.hbase.client.ScannerCallable@7d8a8ce2, >> org.apache.hadoop.hbase.NotServingRegionException: >> org.apache.hadoop.hbase.NotServingRegionException: Region is not online: >> .META.,,1 >> >> at >> org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:183) >> at >> org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:196) >> at >> org.apache.hadoop.hbase.client.MetaScanner.access$000(MetaScanner.java:48) >> at >> org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:126) >> at >> org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:123) >> at >> org.apache.hadoop.hbase.client.HConnectionManager.execute(HConnectionManager.java:359) >> at >> org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:123) >> at >> org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:99) >> at >> org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:77) >> at >> org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:61) >> at >> org.apache.hadoop.hbase.util.HBaseFsck.loadMetaEntries(HBaseFsck.java:2234) >> at >> org.apache.hadoop.hbase.util.HBaseFsck.onlineConsistencyRepair(HBaseFsck.java:334) >> at org.apache.hadoop.hbase.util.HBaseFsck.onlineHbck(HBaseFsck.java:382) >> at org.apache.hadoop.hbase.util.HBaseFsck.main(HBaseFsck.java:3120) >> >> >> Anyone has any idea how to fix it without the obvious restart? >> >
