I went back from 01-13 all the way to hbase-lilaifeng-master-CentOS.log.2014-01-07 I saw:
2014-01-07 17:46:05,700 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: Cached location for .META.,,1.1028785192 is Slave2:60020 Do you have region server log for Slave2 ? On Tue, Jan 14, 2014 at 3:56 PM, 李来锋 <[email protected]> wrote: > Hi, > Since the blackout caused part of the table can not be accessed, use > "hbase HBCK - fix" an error occurred when repair: > 14/01/14 23:12:59 DEBUG > client.HConnectionManager$HConnectionImplementation: Removed > .META.,,1.1028785192 for tableName=.META. from cache because of > 14/01/14 23:12:59 DEBUG > client.HConnectionManager$HConnectionImplementation: Cached location for > .META.,,1.1028785192 is Slave2:60020 > Exception in thread "main" > org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after > attempts=10, exceptions: > Tue Jan 14 22:59:55 CST 2014, > org.apache.hadoop.hbase.client.ScannerCallable@51a19458, > org.apache.hadoop.hbase.NotServingRegionException: > org.apache.hadoop.hbase.NotServingRegionException: Re online: .META.,,1 > Tue Jan 14 23:00:15 CST 2014, > org.apache.hadoop.hbase.client.ScannerCallable@51a19458, > org.apache.hadoop.hbase.NotServingRegionException: > org.apache.hadoop.hbase.NotServingRegionException: Re online: .META.,,1 > Tue Jan 14 23:00:35 CST 2014, > org.apache.hadoop.hbase.client.ScannerCallable@51a19458, > org.apache.hadoop.hbase.NotServingRegionException: > org.apache.hadoop.hbase.NotServingRegionException: Re online: .META.,,1 > Tue Jan 14 23:00:56 CST 2014, > org.apache.hadoop.hbase.client.ScannerCallable@51a19458, > org.apache.hadoop.hbase.NotServingRegionException: > org.apache.hadoop.hbase.NotServingRegionException: Re online: .META.,,1 > Tue Jan 14 23:01:36 CST 2014, > org.apache.hadoop.hbase.client.ScannerCallable@51a19458, > org.apache.hadoop.hbase.NotServingRegionException: > org.apache.hadoop.hbase.NotServingRegionException: Re online: .META.,,1 > Tue Jan 14 23:02:16 CST 2014, > org.apache.hadoop.hbase.client.ScannerCallable@51a19458, > org.apache.hadoop.hbase.NotServingRegionException: > org.apache.hadoop.hbase.NotServingRegionException: Re online: .META.,,1 > Tue Jan 14 23:03:37 CST 2014, > org.apache.hadoop.hbase.client.ScannerCallable@51a19458, > org.apache.hadoop.hbase.NotServingRegionException: > org.apache.hadoop.hbase.NotServingRegionException: Re online: .META.,,1 > Tue Jan 14 23:04:57 CST 2014, > org.apache.hadoop.hbase.client.ScannerCallable@51a19458, > org.apache.hadoop.hbase.NotServingRegionException: > org.apache.hadoop.hbase.NotServingRegionException: Re online: .META.,,1 > Tue Jan 14 23:07:38 CST 2014, > org.apache.hadoop.hbase.client.ScannerCallable@51a19458, > org.apache.hadoop.hbase.NotServingRegionException: > org.apache.hadoop.hbase.NotServingRegionException: Re online: .META.,,1 > Tue Jan 14 23:12:59 CST 2014, > org.apache.hadoop.hbase.client.ScannerCallable@51a19458, > org.apache.hadoop.hbase.NotServingRegionException: > org.apache.hadoop.hbase.NotServingRegionException: Re online: .META.,,1 > at > org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:183) > at > org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:206) > at > org.apache.hadoop.hbase.client.MetaScanner.access$000(MetaScanner.java:54) > at > org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:133) > at > org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:130) > at > org.apache.hadoop.hbase.client.HConnectionManager.execute(HConnectionManager.java:383) > at > org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:130) > at > org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:105) > at > org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:83) > at > org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:67) > at > org.apache.hadoop.hbase.util.HBaseFsck.loadMetaEntries(HBaseFsck.java:2584) > at > org.apache.hadoop.hbase.util.HBaseFsck.onlineConsistencyRepair(HBaseFsck.java:384) > at org.apache.hadoop.hbase.util.HBaseFsck.onlineHbck(HBaseFsck.java:437) > at org.apache.hadoop.hbase.util.HBaseFsck.exec(HBaseFsck.java:3633) > at org.apache.hadoop.hbase.util.HBaseFsck.run(HBaseFsck.java:3452) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79) > at org.apache.hadoop.hbase.util.HBaseFsck.main(HBaseFsck.java:3446) > > This is the error log when the power outage: > 2014-01-13 09:26:10,899 INFO org.apache.hadoop.hbase.master.LoadBalancer: > Skipping load balancing because balanced cluster; servers=2 regions=1 > average=0.5 mostloaded=1 leastloaded=0 > 2014-01-13 09:26:10,899 INFO org.apache.hadoop.hbase.master.LoadBalancer: > Skipping load balancing because balanced cluster; servers=2 regions=1 > average=0.5 mostloaded=1 leastloaded=0 > 2014-01-13 09:26:10,899 INFO org.apache.hadoop.hbase.master.LoadBalancer: > Skipping load balancing because balanced cluster; servers=2 regions=1 > average=0.5 mostloaded=1 leastloaded=0 > 2014-01-13 09:26:10,899 INFO org.apache.hadoop.hbase.master.LoadBalancer: > Skipping load balancing because balanced cluster; servers=2 regions=1 > average=0.5 mostloaded=1 leastloaded=0 > 2014-01-13 09:26:10,899 INFO org.apache.hadoop.hbase.master.LoadBalancer: > Skipping load balancing because balanced cluster; servers=2 regions=1 > average=0.5 mostloaded=1 leastloaded=0 > 2014-01-13 09:26:10,899 INFO org.apache.hadoop.hbase.master.LoadBalancer: > Skipping load balancing because balanced cluster; servers=2 regions=1 > average=0.5 mostloaded=1 leastloaded=0 > 2014-01-13 09:26:10,899 INFO org.apache.hadoop.hbase.master.LoadBalancer: > Skipping load balancing because balanced cluster; servers=2 regions=1 > average=0.5 mostloaded=1 leastloaded=0 > 2014-01-13 09:26:10,899 INFO org.apache.hadoop.hbase.master.LoadBalancer: > Skipping load balancing because balanced cluster; servers=2 regions=1 > average=0.5 mostloaded=1 leastloaded=0 > 2014-01-13 09:26:10,899 INFO org.apache.hadoop.hbase.master.LoadBalancer: > Skipping load balancing because balanced cluster; servers=2 regions=1 > average=0.5 mostloaded=1 leastloaded=0 > 2014-01-13 09:26:10,899 INFO org.apache.hadoop.hbase.master.LoadBalancer: > Skipping load balancing because balanced cluster; servers=2 regions=1 > average=0.5 mostloaded=1 leastloaded=0 > 2014-01-13 09:26:10,923 DEBUG > org.apache.hadoop.hbase.client.ClientScanner: Creating scanner over .META. > starting at key '' > 2014-01-13 09:26:10,923 DEBUG > org.apache.hadoop.hbase.client.ClientScanner: Advancing internal scanner to > startKey at '' > 2014-01-13 09:26:10,929 DEBUG > org.apache.hadoop.hbase.client.ClientScanner: Finished with scanning at > {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192,} > 2014-01-13 09:26:10,929 DEBUG > org.apache.hadoop.hbase.master.CatalogJanitor: Scanned 10 catalog row(s) > and gc'd 0 unreferenced parent region(s) > > Attachment is the log files. > > Thanks > > 致 > 礼 > > 李来锋 > 北京中恒博瑞数字电力科技有限公司 > 邮编: 100085 > 地址: 北京市海淀区西二旗大街39号领秀新硅谷D区B座 > TEL: 010-82781699-8181，18612035758 > >
