It's a known problem with the .META. table involving old deleted rows and new rows. There is a brief summary here:
http://issues.apache.org/jira/browse/HBASE-1383 I believe that we are making truncate in the shell actually issue the major_compact itself too now. On Tue, Jul 14, 2009 at 3:40 PM, Haijun Cao<[email protected]> wrote: > Ryan, > > It worked. thank you! > > I have used truncate command on TestTable before, but I then decided to drop > the table completely and start the test from scratch. This problem seem to > happen after the hbase server crashed. I experienced both > hbase-1634/HBASE-1638 and this count issue. Running the script in 1638 > resolve the NPE problem (Thanks stack), running flush/major_compact meta > table resolved the count issue. > > For my education, do you mind explaining the cause/fix for this count issue I > am seeing? > > Thanks > Haijun > > > > > ________________________________ > From: Ryan Rawson <[email protected]> > To: [email protected] > Sent: Tuesday, July 14, 2009 3:14:33 PM > Subject: Re: java.io.IOException: HRegionInfo was null or empty in .META. > > Try doing this on the shell: > > flush '.META.' > major_compact '.META.' > > did you use 'truncate' command at any point? > > On Tue, Jul 14, 2009 at 3:10 PM, Haijun Cao<[email protected]> wrote: >> >> Hi >> >> I am running hbase PE test, loaded TestTable with 10M records without any >> problem, later hbase crashed (during another sequentialWrite test), after >> restart, I can't count TestTable, it always get stuck at 1.1 mil records. >> >> >> I am wondering if anybody has encountered the same problem (data got >> corrupted after server crash)? How do you recover? At this point, it is just >> test data, so it is ok for me to lose some data. Is there a way to >> drop/repair the bad region (0001192603)? Does hbase have a auto-repair tool >> like fsck? >> >> Your help is greatly appreciated. >> >> Haijun >> >> >> Error stack: >> >> Current count: 1100000, row: 0001099999 >> NativeException: java.lang.RuntimeException: >> org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact >> region server null for region , row '0001192603', but failed after 5 >> attempts. >> Exceptions: >> java.io.IOException: HRegionInfo was null or empty in .META. >> java.io.IOException: HRegionInfo was null or empty in .META. >> java.io.IOException: HRegionInfo was null or empty in .META. >> java.io.IOException: HRegionInfo was null or empty in .META. >> java.io.IOException: HRegionInfo was null or empty in .META. >> >> from org/apache/hadoop/hbase/client/HTable.java:2083:in `hasNext' >> from sun.reflect.GeneratedMethodAccessor9:-1:in `invoke' >> from sun/reflect/DelegatingMethodAccessorImpl.java:25:in `invoke' >> from java/lang/reflect/Method.java:597:in `invoke' >> from org/jruby/javasupport/JavaMethod.java:298:in >> `invokeWithExceptionHandling' >> from org/jruby/javasupport/JavaMethod.java:259:in `invoke' >> from org/jruby/java/invokers/InstanceMethodInvoker.java:36:in `call' >> from org/jruby/runtime/callsite/CachingCallSite.java:70:in `call' >> from org/jruby/ast/CallNoArgNode.java:61:in `interpret' >> from org/jruby/ast/WhileNode.java:127:in `interpret' >> from org/jruby/ast/NewlineNode.java:104:in `interpret' >> from org/jruby/ast/BlockNode.java:71:in `interpret' >> from org/jruby/internal/runtime/methods/InterpretedMethod.java:163:in >> `call' >> from org/jruby/internal/runtime/methods/DefaultMethod.java:144:in >> `call' >> from org/jruby/runtime/callsite/CachingCallSite.java:273:in >> `cacheAndCall' >> from org/jruby/runtime/callsite/CachingCallSite.java:112:in `call' >> >> >> I scan the .META. table and noticed that region 0001192603 has no info >> columns, only historian columns. I also check the hadoop file system, region >> 0001192603 has a oldlogfile.log file in its directory (other regions don't): >> >> Found 3 items >> -rw-r--r-- 3 bamboo supergroup 619 2009-07-13 16:28 >> /user/bamboo/hbase/TestTable/954997373/.regioninfo >> drwxr-xr-x - bamboo supergroup 0 2009-07-13 16:28 >> /user/bamboo/hbase/TestTable/954997373/info >> -rw-r--r-- 3 bamboo supergroup 39417817 2009-07-13 16:52 >> /user/bamboo/hbase/TestTable/954997373/oldlogfile.log >> >> >> > > > >
