I'm trying my "nuclear" option: basically copy the data from the old db to a
new one, skipping over bad regions. The bad news is that it is taking forever.
I get a stack trace just trying to run check_meta.rb:
maxpoint@c1-m02:/usr/lib/hbase/bin$ ./hbase org.jruby.Main check_meta.rb
Writables.java:75:in `org.apache.hadoop.hbase.util.Writables.getWritable':
java.lang.NullPointerException: null (NativeException)
from Writables.java:119:in
`org.apache.hadoop.hbase.util.Writables.getHRegionInfo'
from NativeMethodAccessorImpl.java:-2:in
`sun.reflect.NativeMethodAccessorImpl.invoke0'
from NativeMethodAccessorImpl.java:39:in
`sun.reflect.NativeMethodAccessorImpl.invoke'
from DelegatingMethodAccessorImpl.java:25:in
`sun.reflect.DelegatingMethodAccessorImpl.invoke'
from Method.java:597:in `java.lang.reflect.Method.invoke'
from JavaMethod.java:196:in
`org.jruby.javasupport.JavaMethod.invokeWithExceptionHandling'
from JavaMethod.java:182:in
`org.jruby.javasupport.JavaMethod.invoke_static'
from JavaClass.java:371:in
`org.jruby.javasupport.JavaClass$StaticMethodInvoker.execute'
... 17 levels...
from Main.java:183:in `org.jruby.Main.runInterpreter'
from Main.java:120:in `org.jruby.Main.run'
from Main.java:95:in `org.jruby.Main.main'
Complete Java stackTrace
java.lang.NullPointerException
at org.apache.hadoop.hbase.util.Writables.getWritable(Writables.java:75)
at
org.apache.hadoop.hbase.util.Writables.getHRegionInfo(Writables.java:119)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.jruby.javasupport.JavaMethod.invokeWithExceptionHandling(JavaMethod.java:196)
at org.jruby.javasupport.JavaMethod.invoke_static(JavaMethod.java:182)
at
org.jruby.javasupport.JavaClass$StaticMethodInvoker.execute(JavaClass.java:371)
at
org.jruby.internal.runtime.methods.SimpleCallbackMethod.call(SimpleCallbackMethod.java:81)
at
org.jruby.evaluator.EvaluationState.callNode(EvaluationState.java:571)
at
org.jruby.evaluator.EvaluationState.evalInternal(EvaluationState.java:207)
at
org.jruby.evaluator.EvaluationState.localAsgnNode(EvaluationState.java:1254)
at
org.jruby.evaluator.EvaluationState.evalInternal(EvaluationState.java:286)
at
org.jruby.evaluator.EvaluationState.blockNode(EvaluationState.java:533)
at
org.jruby.evaluator.EvaluationState.evalInternal(EvaluationState.java:201)
at
org.jruby.evaluator.EvaluationState.whileNode(EvaluationState.java:1793)
at
org.jruby.evaluator.EvaluationState.evalInternal(EvaluationState.java:387)
at
org.jruby.evaluator.EvaluationState.blockNode(EvaluationState.java:533)
at
org.jruby.evaluator.EvaluationState.evalInternal(EvaluationState.java:201)
at
org.jruby.evaluator.EvaluationState.rootNode(EvaluationState.java:1628)
at
org.jruby.evaluator.EvaluationState.evalInternal(EvaluationState.java:356)
at org.jruby.evaluator.EvaluationState.eval(EvaluationState.java:164)
at org.jruby.Ruby.eval(Ruby.java:278)
at org.jruby.Ruby.compileOrFallbackAndRun(Ruby.java:306)
at org.jruby.Main.runInterpreter(Main.java:238)
at org.jruby.Main.runInterpreter(Main.java:183)
at org.jruby.Main.run(Main.java:120)
at org.jruby.Main.main(Main.java:95)
-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of Stack
Sent: Friday, May 27, 2011 12:43 AM
To: [email protected]
Subject: Re: wrong region exception
Robert:
Looks like script already exists. Check bin/check_meta.rb. If you pass it
--fix it should plug the hole. Read the head of the script for how to run it.
Good luck,
St.Ack
On Thu, May 26, 2011 at 1:06 PM, Robert Gonzalez
<[email protected]> wrote:
> I sent the meta.txt to your saint.ack@gmail .com account due to the
> attachment.
>
> -----Original Message-----
> From: [email protected] [mailto:[email protected]] On Behalf Of
> Stack
> Sent: Thursday, May 26, 2011 1:35 PM
> To: [email protected]
> Subject: Re: wrong region exception
>
> On Thu, May 26, 2011 at 8:06 AM, Robert Gonzalez
> <[email protected]> wrote:
>> It looks like an entire region is missing, here is the online table:
>>
>> urlhashv2,7FF1A5BF839C37078083B4F8267008F6,1303028235302.b0d55566fd0e
>> 0 2ae98541618396aa7b1.
>> c1-s03.atxd.maxpointinteractive.com:60030
>> 7FF1A5BF839C37078083B4F8267008F6
>> 80116D7E506D87ED39EAFFE784B5B590
>> urlhashv2,8031483E0B3B7F587020FCBB764272D9,1305226123483.3ed065ad87f8
>> 9 aece6b994dd31b42b2a. c1-s33.atxd.maxpointinteractive.com:60030
>> 8031483E0B3B7F587020FCBB764272D9
>> 8041346D0B05617FA4B9152BFE9B18B9
>>
>> One ends at 80116D7E506D87ED39EAFFE784B5B590, but the next one doesn't start
>> there.
>>
>
> So, make sure you actually have a hole. Dump out your meta table:
>
> echo "scan '.META.'"| ./bin/hbase shell &> /tmp/meta.txt
>
> Then look ensure that there is a hole between the above regions (compare
> start and end keys... the end key of one region needs to match the start key
> of the next).
>
> If indeed a hole, you need to do a little surgery inserting a new missing
> region (hbck should fix this but it doesn't have the smarts just yet).
>
> Basically, you create a new region with start and end keys to fill the hole
> then you insert it into .META. and then assign it. There are some scripts in
> our bin directory that do various parts of this. I'm pretty sure its beyond
> any but a few figuring this mess out so if you do the above foot work and
> provide a few more details, I'll hack up something for you (and hopefully
> something generalized to be use by others later, and later to be integrated
> into hbck).
>
> St.Ack
>
