[
https://issues.apache.org/jira/browse/HBASE-13083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14333291#comment-14333291
]
Hadoop QA commented on HBASE-13083:
-----------------------------------
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12700172/HBASE-13083-branch-1.patch
against branch-1 branch at commit 6a4bca86e2b4dba2140f9886a6d4e9b59916a5a7.
ATTACHMENT ID: 12700172
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.1 2.5.2 2.6.0)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/12941//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12941//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12941//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12941//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12941//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12941//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12941//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12941//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12941//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12941//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12941//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12941//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/12941//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/12941//console
This message is automatically generated.
> Master can be dead-locked while assigning META.
> -----------------------------------------------
>
> Key: HBASE-13083
> URL: https://issues.apache.org/jira/browse/HBASE-13083
> Project: HBase
> Issue Type: Bug
> Components: master, Region Assignment
> Affects Versions: 2.0.0, 1.1.0
> Reporter: Andrey Stepachev
> Assignee: Andrey Stepachev
> Attachments: HBASE-13083-branch-1.patch, HBASE-13083.patch
>
>
> We got situation when master is deadlocked.
> It seems we have deadlock in master code. In SSH it calls
> RegionStates#serverOffline which in turn
> aquires synchronized(this) effectively block all requests to RegionStates.
> In another thread it processes assignMeta, which tries to access region
> states and blocks.
> Finally any assignment operations try to access meta for table states and
> region operations, but
> cannot do that due of locked RegionStates class.
> serverOffline() waiting for meta availability
> {code}
> Thread 17019: (state = BLOCKED)
> - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
> may be imprecise)
> - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14,
> line=186 (Interpreted frame)
> -
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(java.util.concurrent.SynchronousQueue$TransferStack$SNode,
> boolean, long) @bci=158, line=458 (Compiled frame)
> /serverOffline
> - java.lang.Thread.sleep(long) @bci=0 (Interpreted frame)
> -
> org.apache.hadoop.hbase.zookeeper.MetaTableLocator.blockUntilAvailable(org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher,
> int, long) @bci=74, line=605 (Interpreted frame)
> -
> org.apache.hadoop.hbase.zookeeper.MetaTableLocator.blockUntilAvailable(org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher,
> long) @bci=4, line=580 (Interpreted frame)
> -
> org.apache.hadoop.hbase.zookeeper.MetaTableLocator.blockUntilAvailable(org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher,
> long, org.apache.hadoop.conf.Configuration) @bci=65, line=559 (Interpreted
> frame)
> - org.apache.hadoop.hbase.client.ZooKeeperRegistry.getMetaRegionLocation()
> @bci=69, line=58 (Interpreted frame)
> -
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateMeta(org.apache.hadoop.hbase.TableName,
> boolean, int) @bci=83, line=1131 (Compiled frame)
> -
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(org.apache.hadoop.hbase.TableName,
> byte[], boolean, boolean, int) @bci=74, line=1098 (Compiled frame)
> -
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.findAllLocationsOrFail(org.apache.hadoop.hbase.client.Action,
> boolean) @bci=73, line=940 (Compiled frame)
> -
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.groupAndSendMultiAction(java.util.List,
> int) @bci=48, line=857 (Compiled frame)
> -
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.access$100(org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl,
> java.util.List, int) @bci=3, line=575 (Compiled frame)
> -
> org.apache.hadoop.hbase.client.AsyncProcess.submitAll(java.util.concurrent.ExecutorService,
> org.apache.hadoop.hbase.TableName, java.util.List,
> org.apache.hadoop.hbase.client.coprocessor.Batch$Callback,
> java.lang.Object[]) @bci=195, line=557 (Compiled frame)
> -
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.processBatchCallback(java.util.List,
> org.apache.hadoop.hbase.TableName, java.util.concurrent.ExecutorService,
> java.lang.Object[],
> org.apache.hadoop.hbase.client.coprocessor.Batch$Callback) @bci=11, line=2136
> (Compiled frame)
> -
> org.apache.hadoop.hbase.util.MultiHConnection.processBatchCallback(java.util.List,
> org.apache.hadoop.hbase.TableName, java.lang.Object[],
> org.apache.hadoop.hbase.client.coprocessor.Batch$Callback) @bci=24, line=125
> (Compiled frame)
> - org.apache.hadoop.hbase.master.RegionStateStore.updateRegionState(long,
> org.apache.hadoop.hbase.master.RegionState,
> org.apache.hadoop.hbase.master.RegionState) @bci=421, line=244 (Compiled
> frame)
> -
> org.apache.hadoop.hbase.master.RegionStates.updateRegionState(org.apache.hadoop.hbase.HRegionInfo,
> org.apache.hadoop.hbase.master.RegionState$State,
> org.apache.hadoop.hbase.ServerName, long) @bci=149, line=1109 (Compiled frame)
> -
> org.apache.hadoop.hbase.master.RegionStates.updateRegionState(org.apache.hadoop.hbase.HRegionInfo,
> org.apache.hadoop.hbase.master.RegionState$State,
> org.apache.hadoop.hbase.ServerName) @bci=7, line=425 (Compiled frame)
> -
> org.apache.hadoop.hbase.master.RegionStates.updateRegionState(org.apache.hadoop.hbase.HRegionInfo,
> org.apache.hadoop.hbase.master.RegionState$State) @bci=24, line=383
> (Compiled frame)
> -
> org.apache.hadoop.hbase.master.RegionStates.regionOffline(org.apache.hadoop.hbase.HRegionInfo,
> org.apache.hadoop.hbase.master.RegionState$State) @bci=83, line=586
> (Interpreted frame)
> -
> org.apache.hadoop.hbase.master.RegionStates.regionOffline(org.apache.hadoop.hbase.HRegionInfo)
> @bci=3, line=566 (Interpreted frame)
> -
> org.apache.hadoop.hbase.master.RegionStates.serverOffline(org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher,
> org.apache.hadoop.hbase.ServerName) @bci=494, line=667 (Interpreted frame)
> -
> org.apache.hadoop.hbase.master.AssignmentManager.processServerShutdown(org.apache.hadoop.hbase.ServerName)
> @bci=101, line=3334 (Interpreted frame)
> - org.apache.hadoop.hbase.master.handler.ServerShutdownHandler.process()
> @bci=626, line=237 (Interpreted frame)
> - org.apache.hadoop.hbase.executor.EventHandler.run() @bci=33, line=128
> (Interpreted frame)
> -
> java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
> @bci=95, line=1145 (Interpreted frame)
> - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line
> {code}
> Blocked meta looks like:
> {code}
> Thread 18357: (state = BLOCKED)
> -
> org.apache.hadoop.hbase.master.RegionStates.getRegionState(java.lang.String)
> @bci=0, line=1053 (Compiled frame)
> -
> org.apache.hadoop.hbase.master.RegionStates.getRegionState(org.apache.hadoop.hbase.HRegionInfo)
> @bci=5, line=1036 (Compiled frame)
> -
> org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(org.apache.hadoop.hbase.HRegionInfo,
> boolean) @bci=5, line=1915 (Interpreted frame)
> -
> org.apache.hadoop.hbase.master.AssignmentManager.assign(org.apache.hadoop.hbase.HRegionInfo,
> boolean, boolean) @bci=29, line=1564 (Interpreted frame)
> -
> org.apache.hadoop.hbase.master.AssignmentManager.assign(org.apache.hadoop.hbase.HRegionInfo,
> boolean) @bci=4, line=1550 (Interpreted frame)
> -
> org.apache.hadoop.hbase.master.AssignmentManager.assignMeta(org.apache.hadoop.hbase.HRegionInfo)
> @bci=23, line=2636 (Interpreted frame)
> -
> org.apache.hadoop.hbase.master.handler.MetaServerShutdownHandler.verifyAndAssignMeta()
> @bci=64, line=159 (Interpreted frame)
> -
> org.apache.hadoop.hbase.master.handler.MetaServerShutdownHandler.verifyAndAssignMetaWithRetries()
> @bci=39, line=184 (Interpreted frame)
> - org.apache.hadoop.hbase.master.handler.MetaServerShutdownHandler.process()
> @bci=276, line=93 (Interpreted frame)
> - org.apache.hadoop.hbase.executor.EventHandler.run() @bci=33, line=128
> (Interpreted frame)
> -
> java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
> @bci=95, line=1145 (Compiled frame)
> - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=615
> (Interpreted frame)
> - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
