ok... Grep the meta row key in master log and see if you can tell a story about what happened to that region....
grep 'e3d331be0a8b19acd6e9e40e858cf59d' logs/hbase-master.log St.Ack On Sat, Oct 23, 2010 at 1:38 PM, Sebastian Bauer <[email protected]> wrote: > this was on production so i created full backup of /hbase dir and > deleted .META. from hadoop, in monday i will have access to test servers > so i will give you more logs. > but before restart i saw many of this: > > 2010-10-23 22:32:37,942 DEBUG > org.apache.hadoop.hbase.master.AssignmentManager: Starting unassignment > of region > Golden_UTA,0C4780DF3A4817A55C54824B53D45F91-7519_d_2010_9_10,1287069029997. > 2010-10-23 22:32:37,945 WARN > org.apache.hadoop.hbase.master.CatalogJanitor: REGIONINFO_QUALIFIER is > empty in > keyvalues={Golden_SUN_new,,1286037947793.e3d331be0a8b19acd6e9e40e858cf59d./inf > 2010-10-23 22:32:37,945 ERROR > org.apache.hadoop.hbase.master.CatalogJanitor: Caught > exception > > java.lang.NullPointerException > > at > org.apache.hadoop.hbase.master.CatalogJanitor$1.visit(CatalogJanitor.java:101) > > at > org.apache.hadoop.hbase.catalog.MetaReader.fullScan(MetaReader.java:197) > > at > org.apache.hadoop.hbase.master.CatalogJanitor.scan(CatalogJanitor.java:107) > > at > org.apache.hadoop.hbase.master.CatalogJanitor.chore(CatalogJanitor.java:77) > > at > org.apache.hadoop.hbase.Chore.run(Chore.java:66) > > > W dniu 23.10.2010 22:17, Stack pisze: >> If you restart master again, still NPE? What version of hbase? So, >> your meta has an empty info:regioninfo cell. When you do ./bin/hbase, >> is there an hbck to run? If so, run it and what does it report? >> StAck >> >> On Sat, Oct 23, 2010 at 11:21 AM, Sebastian Bauer <[email protected]> wrote: >>> Today i found this problem after restarting hbase: >>> >>> >>> 2010-10-23 20:16:17,882 DEBUG >>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: >>> Lookedup root region location, >>> connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionIm >>> 2010-10-23 20:16:17,890 DEBUG >>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: >>> Cached location for .META.,,1.1028785192 is >>> db2a.goldenline.pl:60020 >>> 2010-10-23 20:16:18,432 FATAL org.apache.hadoop.hbase.master.HMaster: >>> Unhandled exception. Starting >>> shutdown. >>> >>> java.lang.NullPointerException >>> >>> at >>> org.apache.hadoop.hbase.util.Writables.getWritable(Writables.java:75) >>> >>> at >>> org.apache.hadoop.hbase.util.Writables.getHRegionInfo(Writables.java:119) >>> >>> at >>> org.apache.hadoop.hbase.client.MetaScanner$1.processRow(MetaScanner.java:188) >>> >>> at >>> org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:157) >>> >>> at >>> org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:69) >>> >>> at >>> org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:54) >>> >>> at >>> org.apache.hadoop.hbase.client.MetaScanner.listAllRegions(MetaScanner.java:195) >>> >>> at >>> org.apache.hadoop.hbase.master.AssignmentManager.assignAllUserRegions(AssignmentManager.java:1048) >>> >>> at >>> org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:379) >>> >>> at >>> org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:265) >>> >>> 2010-10-23 20:16:18,433 INFO org.apache.hadoop.hbase.master.HMaster: >>> Aborting >>> >>> 2010-10-23 20:16:18,433 DEBUG org.apache.hadoop.hbase.master.HMaster: >>> Stopping service threads >>> > > > -- > > Pozdrawiam > Sebastian Bauer > ----------------------------------------------------- > http://tikecik.pl > >
