Hi
I am running hbase PE test, loaded TestTable with 10M records without any
problem, later hbase crashed (during another sequentialWrite test), after
restart, I can't count TestTable, it always get stuck at 1.1 mil records.
I am wondering if anybody has encountered the same problem (data got corrupted
after server crash)? How do you recover? At this point, it is just test data,
so it is ok for me to lose some data. Is there a way to drop/repair the bad
region (0001192603)? Does hbase have a auto-repair tool like fsck?
Your help is greatly appreciated.
Haijun
Error stack:
Current count: 1100000, row: 0001099999
NativeException: java.lang.RuntimeException:
org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact
region server null for region , row '0001192603', but failed after 5 attempts.
Exceptions:
java.io.IOException: HRegionInfo was null or empty in .META.
java.io.IOException: HRegionInfo was null or empty in .META.
java.io.IOException: HRegionInfo was null or empty in .META.
java.io.IOException: HRegionInfo was null or empty in .META.
java.io.IOException: HRegionInfo was null or empty in .META.
from org/apache/hadoop/hbase/client/HTable.java:2083:in `hasNext'
from sun.reflect.GeneratedMethodAccessor9:-1:in `invoke'
from sun/reflect/DelegatingMethodAccessorImpl.java:25:in `invoke'
from java/lang/reflect/Method.java:597:in `invoke'
from org/jruby/javasupport/JavaMethod.java:298:in
`invokeWithExceptionHandling'
from org/jruby/javasupport/JavaMethod.java:259:in `invoke'
from org/jruby/java/invokers/InstanceMethodInvoker.java:36:in `call'
from org/jruby/runtime/callsite/CachingCallSite.java:70:in `call'
from org/jruby/ast/CallNoArgNode.java:61:in `interpret'
from org/jruby/ast/WhileNode.java:127:in `interpret'
from org/jruby/ast/NewlineNode.java:104:in `interpret'
from org/jruby/ast/BlockNode.java:71:in `interpret'
from org/jruby/internal/runtime/methods/InterpretedMethod.java:163:in
`call'
from org/jruby/internal/runtime/methods/DefaultMethod.java:144:in `call'
from org/jruby/runtime/callsite/CachingCallSite.java:273:in
`cacheAndCall'
from org/jruby/runtime/callsite/CachingCallSite.java:112:in `call'
I scan the .META. table and noticed that region 0001192603 has no info columns,
only historian columns. I also check the hadoop file system, region 0001192603
has a oldlogfile.log file in its directory (other regions don't):
Found 3 items
-rw-r--r-- 3 bamboo supergroup 619 2009-07-13 16:28
/user/bamboo/hbase/TestTable/954997373/.regioninfo
drwxr-xr-x - bamboo supergroup 0 2009-07-13 16:28
/user/bamboo/hbase/TestTable/954997373/info
-rw-r--r-- 3 bamboo supergroup 39417817 2009-07-13 16:52
/user/bamboo/hbase/TestTable/954997373/oldlogfile.log
