The script ran without the previous problem, but it did not fix the problem. When I ran hbck or check_meta.rb again they indicated that the problem was still there. Do I need to do something else in preparation before running check_meta?
Thanks, Robert -----Original Message----- From: [email protected] [mailto:[email protected]] On Behalf Of Stack Sent: Tuesday, May 31, 2011 2:57 PM To: [email protected] Subject: Re: wrong region exception Try adding this change: Index: bin/check_meta.rb =================================================================== --- bin/check_meta.rb (revision 1129468) +++ bin/check_meta.rb (working copy) @@ -127,11 +127,13 @@ scan = Scan.new() scanner = metatable.getScanner(scan) oldHRI = nil -bad = nil +bad = 0 while (result = scanner.next()) rowid = Bytes.toString(result.getRow()) rowidStr = java.lang.String.new(rowid) bytes = result.getValue(HConstants::CATALOG_FAMILY, HConstants::REGIONINFO_QUALIFIER) + next if not bytes + next if bytes.length == 0 hri = Writables.getHRegionInfo(bytes) if oldHRI if oldHRI.isOffline() && Bytes.equals(oldHRI.getStartKey(), hri.getStartKey()) You might print out the result you have if the qualifier is null just to see what are the rows missing an HRegionInfo. St.Ack On Tue, May 31, 2011 at 9:02 AM, Robert Gonzalez <[email protected]> wrote: > I'm trying my "nuclear" option: basically copy the data from the old db to a > new one, skipping over bad regions. The bad news is that it is taking > forever. > > > I get a stack trace just trying to run check_meta.rb: > > maxpoint@c1-m02:/usr/lib/hbase/bin$ ./hbase org.jruby.Main > check_meta.rb Writables.java:75:in > `org.apache.hadoop.hbase.util.Writables.getWritable': > java.lang.NullPointerException: null (NativeException) > from Writables.java:119:in > `org.apache.hadoop.hbase.util.Writables.getHRegionInfo' > from NativeMethodAccessorImpl.java:-2:in > `sun.reflect.NativeMethodAccessorImpl.invoke0' > from NativeMethodAccessorImpl.java:39:in > `sun.reflect.NativeMethodAccessorImpl.invoke' > from DelegatingMethodAccessorImpl.java:25:in > `sun.reflect.DelegatingMethodAccessorImpl.invoke' > from Method.java:597:in `java.lang.reflect.Method.invoke' > from JavaMethod.java:196:in > `org.jruby.javasupport.JavaMethod.invokeWithExceptionHandling' > from JavaMethod.java:182:in > `org.jruby.javasupport.JavaMethod.invoke_static' > from JavaClass.java:371:in > `org.jruby.javasupport.JavaClass$StaticMethodInvoker.execute' > ... 17 levels... > from Main.java:183:in `org.jruby.Main.runInterpreter' > from Main.java:120:in `org.jruby.Main.run' > from Main.java:95:in `org.jruby.Main.main' > Complete Java stackTrace > java.lang.NullPointerException > at > org.apache.hadoop.hbase.util.Writables.getWritable(Writables.java:75) > at > org.apache.hadoop.hbase.util.Writables.getHRegionInfo(Writables.java:1 > 19) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.j > ava:39) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccess > orImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at > org.jruby.javasupport.JavaMethod.invokeWithExceptionHandling(JavaMetho > d.java:196) > at > org.jruby.javasupport.JavaMethod.invoke_static(JavaMethod.java:182) > at > org.jruby.javasupport.JavaClass$StaticMethodInvoker.execute(JavaClass. > java:371) > at > org.jruby.internal.runtime.methods.SimpleCallbackMethod.call(SimpleCal > lbackMethod.java:81) > at > org.jruby.evaluator.EvaluationState.callNode(EvaluationState.java:571) > at > org.jruby.evaluator.EvaluationState.evalInternal(EvaluationState.java: > 207) > at > org.jruby.evaluator.EvaluationState.localAsgnNode(EvaluationState.java > :1254) > at > org.jruby.evaluator.EvaluationState.evalInternal(EvaluationState.java: > 286) > at > org.jruby.evaluator.EvaluationState.blockNode(EvaluationState.java:533 > ) > at > org.jruby.evaluator.EvaluationState.evalInternal(EvaluationState.java: > 201) > at > org.jruby.evaluator.EvaluationState.whileNode(EvaluationState.java:179 > 3) > at > org.jruby.evaluator.EvaluationState.evalInternal(EvaluationState.java: > 387) > at > org.jruby.evaluator.EvaluationState.blockNode(EvaluationState.java:533 > ) > at > org.jruby.evaluator.EvaluationState.evalInternal(EvaluationState.java: > 201) > at > org.jruby.evaluator.EvaluationState.rootNode(EvaluationState.java:1628 > ) > at > org.jruby.evaluator.EvaluationState.evalInternal(EvaluationState.java: > 356) > at > org.jruby.evaluator.EvaluationState.eval(EvaluationState.java:164) > at org.jruby.Ruby.eval(Ruby.java:278) > at org.jruby.Ruby.compileOrFallbackAndRun(Ruby.java:306) > at org.jruby.Main.runInterpreter(Main.java:238) > at org.jruby.Main.runInterpreter(Main.java:183) > at org.jruby.Main.run(Main.java:120) > at org.jruby.Main.main(Main.java:95) > > -----Original Message----- > From: [email protected] [mailto:[email protected]] On Behalf Of > Stack > Sent: Friday, May 27, 2011 12:43 AM > To: [email protected] > Subject: Re: wrong region exception > > Robert: > > Looks like script already exists. Check bin/check_meta.rb. If you pass it > --fix it should plug the hole. Read the head of the script for how to run it. > > Good luck, > St.Ack > > On Thu, May 26, 2011 at 1:06 PM, Robert Gonzalez > <[email protected]> wrote: >> I sent the meta.txt to your saint.ack@gmail .com account due to the >> attachment. >> >> -----Original Message----- >> From: [email protected] [mailto:[email protected]] On Behalf Of >> Stack >> Sent: Thursday, May 26, 2011 1:35 PM >> To: [email protected] >> Subject: Re: wrong region exception >> >> On Thu, May 26, 2011 at 8:06 AM, Robert Gonzalez >> <[email protected]> wrote: >>> It looks like an entire region is missing, here is the online table: >>> >>> urlhashv2,7FF1A5BF839C37078083B4F8267008F6,1303028235302.b0d55566fd0 >>> e >>> 0 2ae98541618396aa7b1. >>> c1-s03.atxd.maxpointinteractive.com:60030 >>> 7FF1A5BF839C37078083B4F8267008F6 >>> 80116D7E506D87ED39EAFFE784B5B590 >>> urlhashv2,8031483E0B3B7F587020FCBB764272D9,1305226123483.3ed065ad87f >>> 8 >>> 9 aece6b994dd31b42b2a. >>> c1-s33.atxd.maxpointinteractive.com:60030 >>> 8031483E0B3B7F587020FCBB764272D9 >>> 8041346D0B05617FA4B9152BFE9B18B9 >>> >>> One ends at 80116D7E506D87ED39EAFFE784B5B590, but the next one doesn't >>> start there. >>> >> >> So, make sure you actually have a hole. Dump out your meta table: >> >> echo "scan '.META.'"| ./bin/hbase shell &> /tmp/meta.txt >> >> Then look ensure that there is a hole between the above regions (compare >> start and end keys... the end key of one region needs to match the start key >> of the next). >> >> If indeed a hole, you need to do a little surgery inserting a new missing >> region (hbck should fix this but it doesn't have the smarts just yet). >> >> Basically, you create a new region with start and end keys to fill the hole >> then you insert it into .META. and then assign it. There are some scripts >> in our bin directory that do various parts of this. I'm pretty sure its >> beyond any but a few figuring this mess out so if you do the above foot work >> and provide a few more details, I'll hack up something for you (and >> hopefully something generalized to be use by others later, and later to be >> integrated into hbck). >> >> St.Ack >> >
