[
https://issues.apache.org/jira/browse/HBASE-5356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13204137#comment-13204137
]
Jonathan Hsieh commented on HBASE-5356:
---------------------------------------
rephrase of previous comment: The region_mover script initially caches a list
of regions to move. If the table is deleted after the cached list is gathered
but before all regions are moved, the script can get stuck attempting to move a
deleted region.
In a related but likely separate issue -- if a new presplit table is created,
as the region_mover is emptying an RS, the emptying RS is a candidate for new
regions and will get some of the new regions. Ideally this is fenced off so
that it does not get regions assigned to it, but this would require some ZK.
This seems less important because the majority of regions will be moved off the
RS and the few new regions can rely on the automatically fail over to other
RS's.
> region_mover.rb can hang if table region it belongs to is deleted.
> ------------------------------------------------------------------
>
> Key: HBASE-5356
> URL: https://issues.apache.org/jira/browse/HBASE-5356
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.90.3, 0.94.0, 0.92.0
> Reporter: Jonathan Hsieh
> Assignee: Jonathan Hsieh
> Priority: Minor
>
> I was testing the region_mover.rb script on a loaded hbase and noticed that
> it can hang (thus hanging graceful shutdown) if a region that it is
> attempting to move gets deleted (by a table delete operation).
> Here's the start of the relevent stack dump
> {code}
> 12/02/08 13:27:13 WARN client.HConnectionManager$HConnectionImplementation:
> Encountered problems when prefetch META table:
> org.apache.hadoop.hbase.TableNotFoundException: Cannot find row in .META. for
> table: TestLoadAndVerify_1328735001040, row=TestLoadAnd\
> Verify_1328735001040,yC^P\xD7\x945\xD4,99999999999999
> at
> org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:136)
> at
> org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:95)
> at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.prefetchRegionCache(HConnectionManager.java:64\
> 9)
> at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:703\
> )
> at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:594)
> at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.relocateRegion(HConnectionManager.java:565)
> at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionLocation(HConnectionManager.java:416)
> at
> org.apache.hadoop.hbase.client.ServerCallable.instantiateServer(ServerCallable.java:57)
> at
> org.apache.hadoop.hbase.client.ScannerCallable.instantiateServer(ScannerCallable.java:63)
> at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionServerWithRetries(HConnectionManager.\
> java:1018)
> at
> org.apache.hadoop.hbase.client.HTable$ClientScanner.nextScanner(HTable.java:1104)
> at
> org.apache.hadoop.hbase.client.HTable$ClientScanner.initialize(HTable.java:1027)
> at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:535)
> at sun.reflect.GeneratedMethodAccessor24.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(JavaMethod.java:525)
> at org.jruby.javasupport.JavaMethod.invokeDirect(JavaMethod.java:380)
> at
> org.jruby.java.invokers.InstanceMethodInvoker.call(InstanceMethodInvoker.java:58)
> at
> org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:137)
> at
> usr.lib.hbase.bin.region_mover.method__7$RUBY$isSuccessfulScan(/usr/lib/hbase/bin/region_mover.rb:133)
> at
> usr$lib$hbase$bin$region_mover#method__7$RUBY$isSuccessfulScan.call(usr$lib$hbase$bin$region_mover#method__7$RUBY$isSucces\
> sfulScan:65535)
> at
> usr$lib$hbase$bin$region_mover#method__7$RUBY$isSuccessfulScan.call(usr$lib$hbase$bin$region_mover#method__7$RUBY$isSucces\
> sfulScan:65535)
> at
> org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:171)
> at
> usr.lib.hbase.bin.region_mover.block_4$RUBY$__for__(/usr/lib/hbase/bin/region_mover.rb:326)
> at
> usr$lib$hbase$bin$region_mover#block_4$RUBY$__for__.call(usr$lib$hbase$bin$region_mover#block_4$RUBY$__for__:65535)
> at org.jruby.runtime.CompiledBlock.yield(CompiledBlock.java:133)
> at org.jruby.runtime.BlockBody.call(BlockBody.java:73)
> at org.jruby.runtime.Block.call(Block.java:89)
> at org.jruby.RubyProc.call(RubyProc.java:268)
> at org.jruby.RubyProc.call(RubyProc.java:228)
> at org.jruby.RubyProc$i$0$0$call.call(RubyProc$i$0$0$call.gen:65535)
> at
> org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:209)
> at
> org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:205)
> at
> org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:137)
> at org.jruby.ast.CallOneArgNode.interpret(CallOneArgNode.java:57)
> at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:103)
> at org.jruby.ast.WhileNode.interpret(WhileNode.java:131)
> at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:103)
> at org.jruby.ast.BlockNode.interpret(BlockNode.java:71)
> at
> org.jruby.evaluator.ASTInterpreter.INTERPRET_METHOD(ASTInterpreter.java:74)
> at
> org.jruby.internal.runtime.methods.InterpretedMethod.call(InterpretedMethod.java:169)
> at
> org.jruby.internal.runtime.methods.DefaultMethod.call(DefaultMethod.java:171)
> at
> org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:272)
> at
> org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:114)
> at
> org.jruby.runtime.callsite.CachingCallSite.callIter(CachingCallSite.java:123)
> at
> usr.lib.hbase.bin.region_mover.chained_26_rescue_4$RUBY$SYNTHETICunloadRegions(/usr/lib/hbase/bin/region_mover.rb:319)
> at
> usr.lib.hbase.bin.region_mover.method__25$RUBY$unloadRegions(/usr/lib/hbase/bin/region_mover.rb:313)
> at
> usr$lib$hbase$bin$region_mover#method__25$RUBY$unloadRegions.call(usr$lib$hbase$bin$region_mover#method__25$RUBY$unloadRegions:65535)
> at
> usr$lib$hbase$bin$region_mover#method__25$RUBY$unloadRegions.call(usr$lib$hbase$bin$region_mover#method__25$RUBY$unloadRegions:65535)
> at
> org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:302)
> at
> org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:173)
> at
> usr.lib.hbase.bin.region_mover.__file__(/usr/lib/hbase/bin/region_mover.rb:430)
> at
> usr.lib.hbase.bin.region_mover.load(/usr/lib/hbase/bin/region_mover.rb)
> at org.jruby.Ruby.runScript(Ruby.java:670)
> at org.jruby.Ruby.runNormally(Ruby.java:574)
> at org.jruby.Ruby.runFromMain(Ruby.java:423)
> at org.jruby.Main.doRunFromMain(Main.java:278)
> at org.jruby.Main.internalRun(Main.java:198)
> at org.jruby.Main.run(Main.java:164)
> at org.jruby.Main.run(Main.java:148)
> at org.jruby.Main.main(Main.java:128)
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
