[jira] [Commented] (HBASE-10897) On master start, deadlock if refresh UI
[
https://issues.apache.org/jira/browse/HBASE-10897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13966267#comment-13966267
]
stack commented on HBASE-10897:
---
Go for it Jimmy
On master start, deadlock if refresh UI
---
Key: HBASE-10897
URL: https://issues.apache.org/jira/browse/HBASE-10897
Project: HBase
Issue Type: Bug
Affects Versions: 0.99.0
Reporter: stack
Assignee: Jimmy Xiang
Fix For: 0.99.0
Attachments: hbase-10897.patch, hbase-10897_v2.patch,
hbase-10897_v3.patch, hbase-10897_v4.patch
Playing w/ MTTR recovery on trunk, master starting up deadlocked:
Waiting to finish active master initialization:
{code}
ActiveMasterManager daemon prio=10 tid=0x7fafb5dc3800 nid=0x5fb5
waiting for monitor entry [0x7faf8f57d000]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getKeepAliveZooKeeperWatcher(ConnectionManager.java:1683)
- waiting to lock 0x00064ab4b9a8 (a java.lang.Object)
at
org.apache.hadoop.hbase.client.ZooKeeperRegistry.getMetaRegionLocation(ZooKeeperRegistry.java:53)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1029)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:989)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getRegionLocation(ConnectionManager.java:830)
at
org.apache.hadoop.hbase.client.ConnectionAdapter.getRegionLocation(ConnectionAdapter.java:305)
at
org.apache.hadoop.hbase.client.RegionServerCallable.prepare(RegionServerCallable.java:77)
at
org.apache.hadoop.hbase.client.ScannerCallable.prepare(ScannerCallable.java:118)
at
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:101)
at
org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:264)
at
org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:169)
at
org.apache.hadoop.hbase.client.ClientScanner.init(ClientScanner.java:164)
at
org.apache.hadoop.hbase.client.ClientScanner.init(ClientScanner.java:107)
at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:766)
at
org.apache.hadoop.hbase.catalog.MetaReader.fullScan(MetaReader.java:539)
at
org.apache.hadoop.hbase.catalog.MetaReader.fullScanOfMeta(MetaReader.java:140)
at
org.apache.hadoop.hbase.catalog.MetaMigrationConvertingToPB.isMetaTableUpdated(MetaMigrationConvertingToPB.java:164)
at
org.apache.hadoop.hbase.catalog.MetaMigrationConvertingToPB.updateMetaIfNecessary(MetaMigrationConvertingToPB.java:131)
at
org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:567)
at org.apache.hadoop.hbase.master.HMaster.access$500(HMaster.java:147)
at org.apache.hadoop.hbase.master.HMaster$1.run(HMaster.java:1242)
at java.lang.Thread.run(Thread.java:744)
{code}
... but the master servlet has the lock while trying to access master:
{code}
686004346@qtp-2101021459-0 daemon prio=10 tid=0x7fafb5d2a800 nid=0x5fb1
waiting on condition [0x7faf8f87f000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$StubMaker.makeStub(ConnectionManager.java:1562)
- locked 0x00064ab4b9a8 (a java.lang.Object)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$MasterServiceStubMaker.makeStub(ConnectionManager.java:1597)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getKeepAliveMasterService(ConnectionManager.java:1805)
- locked 0x00064ab4b9a8 (a java.lang.Object)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.listTables(ConnectionManager.java:2481)
at
org.apache.hadoop.hbase.client.HBaseAdmin.listTables(HBaseAdmin.java:321)
at
org.apache.hadoop.hbase.tmpl.master.MasterStatusTmplImpl.__jamon_innerUnit__userTables(MasterStatusTmplImpl.java:530)
at
org.apache.hadoop.hbase.tmpl.master.MasterStatusTmplImpl.renderNoFlush(MasterStatusTmplImpl.java:255)
at
org.apache.hadoop.hbase.tmpl.master.MasterStatusTmpl.renderNoFlush(MasterStatusTmpl.java:382)
at
org.apache.hadoop.hbase.tmpl.master.MasterStatusTmpl.render(MasterStatusTmpl.java:372)
at
[jira] [Commented] (HBASE-10897) On master start, deadlock if refresh UI
[
https://issues.apache.org/jira/browse/HBASE-10897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13966997#comment-13966997
]
Hudson commented on HBASE-10897:
SUCCESS: Integrated in HBase-TRUNK #5077 (See
[https://builds.apache.org/job/HBase-TRUNK/5077/])
HBASE-10897 On master start, deadlock if refresh UI (jxiang: rev 1586705)
*
/hbase/trunk/hbase-server/src/main/jamon/org/apache/hadoop/hbase/tmpl/master/MasterStatusTmpl.jamon
*
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
*
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterRpcServices.java
*
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterStatusServlet.java
*
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/ServerManager.java
On master start, deadlock if refresh UI
---
Key: HBASE-10897
URL: https://issues.apache.org/jira/browse/HBASE-10897
Project: HBase
Issue Type: Bug
Affects Versions: 0.99.0
Reporter: stack
Assignee: Jimmy Xiang
Fix For: 0.99.0
Attachments: hbase-10897.patch, hbase-10897_v2.patch,
hbase-10897_v3.patch, hbase-10897_v4.patch
Playing w/ MTTR recovery on trunk, master starting up deadlocked:
Waiting to finish active master initialization:
{code}
ActiveMasterManager daemon prio=10 tid=0x7fafb5dc3800 nid=0x5fb5
waiting for monitor entry [0x7faf8f57d000]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getKeepAliveZooKeeperWatcher(ConnectionManager.java:1683)
- waiting to lock 0x00064ab4b9a8 (a java.lang.Object)
at
org.apache.hadoop.hbase.client.ZooKeeperRegistry.getMetaRegionLocation(ZooKeeperRegistry.java:53)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1029)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:989)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getRegionLocation(ConnectionManager.java:830)
at
org.apache.hadoop.hbase.client.ConnectionAdapter.getRegionLocation(ConnectionAdapter.java:305)
at
org.apache.hadoop.hbase.client.RegionServerCallable.prepare(RegionServerCallable.java:77)
at
org.apache.hadoop.hbase.client.ScannerCallable.prepare(ScannerCallable.java:118)
at
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:101)
at
org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:264)
at
org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:169)
at
org.apache.hadoop.hbase.client.ClientScanner.init(ClientScanner.java:164)
at
org.apache.hadoop.hbase.client.ClientScanner.init(ClientScanner.java:107)
at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:766)
at
org.apache.hadoop.hbase.catalog.MetaReader.fullScan(MetaReader.java:539)
at
org.apache.hadoop.hbase.catalog.MetaReader.fullScanOfMeta(MetaReader.java:140)
at
org.apache.hadoop.hbase.catalog.MetaMigrationConvertingToPB.isMetaTableUpdated(MetaMigrationConvertingToPB.java:164)
at
org.apache.hadoop.hbase.catalog.MetaMigrationConvertingToPB.updateMetaIfNecessary(MetaMigrationConvertingToPB.java:131)
at
org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:567)
at org.apache.hadoop.hbase.master.HMaster.access$500(HMaster.java:147)
at org.apache.hadoop.hbase.master.HMaster$1.run(HMaster.java:1242)
at java.lang.Thread.run(Thread.java:744)
{code}
... but the master servlet has the lock while trying to access master:
{code}
686004346@qtp-2101021459-0 daemon prio=10 tid=0x7fafb5d2a800 nid=0x5fb1
waiting on condition [0x7faf8f87f000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$StubMaker.makeStub(ConnectionManager.java:1562)
- locked 0x00064ab4b9a8 (a java.lang.Object)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$MasterServiceStubMaker.makeStub(ConnectionManager.java:1597)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getKeepAliveMasterService(ConnectionManager.java:1805)
- locked 0x00064ab4b9a8 (a java.lang.Object)
at
[jira] [Commented] (HBASE-10897) On master start, deadlock if refresh UI
[
https://issues.apache.org/jira/browse/HBASE-10897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13965535#comment-13965535
]
Jimmy Xiang commented on HBASE-10897:
-
{quote}
Is this right?
+ if (master.getMasterCoprocessorHost() == null) {
What do CPs have to do w/ this?
{quote}
It doesn't look right. Let me do it in a different way.
{quote}
Has to be public?
+ public boolean isCatalogJanitorEnabled() {
{quote}
I will make it packaged protected.
On master start, deadlock if refresh UI
---
Key: HBASE-10897
URL: https://issues.apache.org/jira/browse/HBASE-10897
Project: HBase
Issue Type: Bug
Affects Versions: 0.99.0
Reporter: stack
Assignee: Jimmy Xiang
Fix For: 0.99.0
Attachments: hbase-10897.patch, hbase-10897_v2.patch,
hbase-10897_v3.patch
Playing w/ MTTR recovery on trunk, master starting up deadlocked:
Waiting to finish active master initialization:
{code}
ActiveMasterManager daemon prio=10 tid=0x7fafb5dc3800 nid=0x5fb5
waiting for monitor entry [0x7faf8f57d000]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getKeepAliveZooKeeperWatcher(ConnectionManager.java:1683)
- waiting to lock 0x00064ab4b9a8 (a java.lang.Object)
at
org.apache.hadoop.hbase.client.ZooKeeperRegistry.getMetaRegionLocation(ZooKeeperRegistry.java:53)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1029)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:989)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getRegionLocation(ConnectionManager.java:830)
at
org.apache.hadoop.hbase.client.ConnectionAdapter.getRegionLocation(ConnectionAdapter.java:305)
at
org.apache.hadoop.hbase.client.RegionServerCallable.prepare(RegionServerCallable.java:77)
at
org.apache.hadoop.hbase.client.ScannerCallable.prepare(ScannerCallable.java:118)
at
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:101)
at
org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:264)
at
org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:169)
at
org.apache.hadoop.hbase.client.ClientScanner.init(ClientScanner.java:164)
at
org.apache.hadoop.hbase.client.ClientScanner.init(ClientScanner.java:107)
at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:766)
at
org.apache.hadoop.hbase.catalog.MetaReader.fullScan(MetaReader.java:539)
at
org.apache.hadoop.hbase.catalog.MetaReader.fullScanOfMeta(MetaReader.java:140)
at
org.apache.hadoop.hbase.catalog.MetaMigrationConvertingToPB.isMetaTableUpdated(MetaMigrationConvertingToPB.java:164)
at
org.apache.hadoop.hbase.catalog.MetaMigrationConvertingToPB.updateMetaIfNecessary(MetaMigrationConvertingToPB.java:131)
at
org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:567)
at org.apache.hadoop.hbase.master.HMaster.access$500(HMaster.java:147)
at org.apache.hadoop.hbase.master.HMaster$1.run(HMaster.java:1242)
at java.lang.Thread.run(Thread.java:744)
{code}
... but the master servlet has the lock while trying to access master:
{code}
686004346@qtp-2101021459-0 daemon prio=10 tid=0x7fafb5d2a800 nid=0x5fb1
waiting on condition [0x7faf8f87f000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$StubMaker.makeStub(ConnectionManager.java:1562)
- locked 0x00064ab4b9a8 (a java.lang.Object)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$MasterServiceStubMaker.makeStub(ConnectionManager.java:1597)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getKeepAliveMasterService(ConnectionManager.java:1805)
- locked 0x00064ab4b9a8 (a java.lang.Object)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.listTables(ConnectionManager.java:2481)
at
org.apache.hadoop.hbase.client.HBaseAdmin.listTables(HBaseAdmin.java:321)
at
org.apache.hadoop.hbase.tmpl.master.MasterStatusTmplImpl.__jamon_innerUnit__userTables(MasterStatusTmplImpl.java:530)
at
org.apache.hadoop.hbase.tmpl.master.MasterStatusTmplImpl.renderNoFlush(MasterStatusTmplImpl.java:255)
at
[jira] [Commented] (HBASE-10897) On master start, deadlock if refresh UI
[
https://issues.apache.org/jira/browse/HBASE-10897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13965752#comment-13965752
]
Hadoop QA commented on HBASE-10897:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12639610/hbase-10897_v4.patch
against trunk revision .
ATTACHMENT ID: 12639610
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/9248//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9248//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9248//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9248//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9248//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9248//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9248//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9248//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9248//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9248//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/9248//console
This message is automatically generated.
On master start, deadlock if refresh UI
---
Key: HBASE-10897
URL: https://issues.apache.org/jira/browse/HBASE-10897
Project: HBase
Issue Type: Bug
Affects Versions: 0.99.0
Reporter: stack
Assignee: Jimmy Xiang
Fix For: 0.99.0
Attachments: hbase-10897.patch, hbase-10897_v2.patch,
hbase-10897_v3.patch, hbase-10897_v4.patch
Playing w/ MTTR recovery on trunk, master starting up deadlocked:
Waiting to finish active master initialization:
{code}
ActiveMasterManager daemon prio=10 tid=0x7fafb5dc3800 nid=0x5fb5
waiting for monitor entry [0x7faf8f57d000]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getKeepAliveZooKeeperWatcher(ConnectionManager.java:1683)
- waiting to lock 0x00064ab4b9a8 (a java.lang.Object)
at
org.apache.hadoop.hbase.client.ZooKeeperRegistry.getMetaRegionLocation(ZooKeeperRegistry.java:53)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1029)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:989)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getRegionLocation(ConnectionManager.java:830)
at
org.apache.hadoop.hbase.client.ConnectionAdapter.getRegionLocation(ConnectionAdapter.java:305)
at
org.apache.hadoop.hbase.client.RegionServerCallable.prepare(RegionServerCallable.java:77)
at
org.apache.hadoop.hbase.client.ScannerCallable.prepare(ScannerCallable.java:118)
at
[jira] [Commented] (HBASE-10897) On master start, deadlock if refresh UI
[
https://issues.apache.org/jira/browse/HBASE-10897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13964587#comment-13964587
]
Hadoop QA commented on HBASE-10897:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12639438/hbase-10897_v2.patch
against trunk revision .
ATTACHMENT ID: 12639438
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.regionserver.handler.TestCloseRegionHandler
org.apache.hadoop.hbase.master.TestAssignmentManager
{color:red}-1 core zombie tests{color}. There are 1 zombie test(s):
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/9231//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9231//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9231//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9231//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9231//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9231//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9231//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9231//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9231//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9231//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/9231//console
This message is automatically generated.
On master start, deadlock if refresh UI
---
Key: HBASE-10897
URL: https://issues.apache.org/jira/browse/HBASE-10897
Project: HBase
Issue Type: Bug
Affects Versions: 0.99.0
Reporter: stack
Assignee: Jimmy Xiang
Fix For: 0.99.0
Attachments: hbase-10897.patch, hbase-10897_v2.patch
Playing w/ MTTR recovery on trunk, master starting up deadlocked:
Waiting to finish active master initialization:
{code}
ActiveMasterManager daemon prio=10 tid=0x7fafb5dc3800 nid=0x5fb5
waiting for monitor entry [0x7faf8f57d000]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getKeepAliveZooKeeperWatcher(ConnectionManager.java:1683)
- waiting to lock 0x00064ab4b9a8 (a java.lang.Object)
at
org.apache.hadoop.hbase.client.ZooKeeperRegistry.getMetaRegionLocation(ZooKeeperRegistry.java:53)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1029)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:989)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getRegionLocation(ConnectionManager.java:830)
at
org.apache.hadoop.hbase.client.ConnectionAdapter.getRegionLocation(ConnectionAdapter.java:305)
at
[jira] [Commented] (HBASE-10897) On master start, deadlock if refresh UI
[
https://issues.apache.org/jira/browse/HBASE-10897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13964920#comment-13964920
]
stack commented on HBASE-10897:
---
Is this right?
+ if (master.getMasterCoprocessorHost() == null) {
What do CPs have to do w/ this?
Has to be public?
+ public boolean isCatalogJanitorEnabled() {
Does + public boolean isCatalogJanitorEnabled() { signify Master is up? If
so, should the test be isMasterUp? (Do we have one of these already?)
Else its great that the UI comes up early.
On master start, deadlock if refresh UI
---
Key: HBASE-10897
URL: https://issues.apache.org/jira/browse/HBASE-10897
Project: HBase
Issue Type: Bug
Affects Versions: 0.99.0
Reporter: stack
Assignee: Jimmy Xiang
Fix For: 0.99.0
Attachments: hbase-10897.patch, hbase-10897_v2.patch,
hbase-10897_v3.patch
Playing w/ MTTR recovery on trunk, master starting up deadlocked:
Waiting to finish active master initialization:
{code}
ActiveMasterManager daemon prio=10 tid=0x7fafb5dc3800 nid=0x5fb5
waiting for monitor entry [0x7faf8f57d000]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getKeepAliveZooKeeperWatcher(ConnectionManager.java:1683)
- waiting to lock 0x00064ab4b9a8 (a java.lang.Object)
at
org.apache.hadoop.hbase.client.ZooKeeperRegistry.getMetaRegionLocation(ZooKeeperRegistry.java:53)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1029)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:989)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getRegionLocation(ConnectionManager.java:830)
at
org.apache.hadoop.hbase.client.ConnectionAdapter.getRegionLocation(ConnectionAdapter.java:305)
at
org.apache.hadoop.hbase.client.RegionServerCallable.prepare(RegionServerCallable.java:77)
at
org.apache.hadoop.hbase.client.ScannerCallable.prepare(ScannerCallable.java:118)
at
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:101)
at
org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:264)
at
org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:169)
at
org.apache.hadoop.hbase.client.ClientScanner.init(ClientScanner.java:164)
at
org.apache.hadoop.hbase.client.ClientScanner.init(ClientScanner.java:107)
at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:766)
at
org.apache.hadoop.hbase.catalog.MetaReader.fullScan(MetaReader.java:539)
at
org.apache.hadoop.hbase.catalog.MetaReader.fullScanOfMeta(MetaReader.java:140)
at
org.apache.hadoop.hbase.catalog.MetaMigrationConvertingToPB.isMetaTableUpdated(MetaMigrationConvertingToPB.java:164)
at
org.apache.hadoop.hbase.catalog.MetaMigrationConvertingToPB.updateMetaIfNecessary(MetaMigrationConvertingToPB.java:131)
at
org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:567)
at org.apache.hadoop.hbase.master.HMaster.access$500(HMaster.java:147)
at org.apache.hadoop.hbase.master.HMaster$1.run(HMaster.java:1242)
at java.lang.Thread.run(Thread.java:744)
{code}
... but the master servlet has the lock while trying to access master:
{code}
686004346@qtp-2101021459-0 daemon prio=10 tid=0x7fafb5d2a800 nid=0x5fb1
waiting on condition [0x7faf8f87f000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$StubMaker.makeStub(ConnectionManager.java:1562)
- locked 0x00064ab4b9a8 (a java.lang.Object)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$MasterServiceStubMaker.makeStub(ConnectionManager.java:1597)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getKeepAliveMasterService(ConnectionManager.java:1805)
- locked 0x00064ab4b9a8 (a java.lang.Object)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.listTables(ConnectionManager.java:2481)
at
org.apache.hadoop.hbase.client.HBaseAdmin.listTables(HBaseAdmin.java:321)
at
org.apache.hadoop.hbase.tmpl.master.MasterStatusTmplImpl.__jamon_innerUnit__userTables(MasterStatusTmplImpl.java:530)
at
[jira] [Commented] (HBASE-10897) On master start, deadlock if refresh UI
[
https://issues.apache.org/jira/browse/HBASE-10897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13964922#comment-13964922
]
Hadoop QA commented on HBASE-10897:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12639484/hbase-10897_v3.patch
against trunk revision .
ATTACHMENT ID: 12639484
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.regionserver.handler.TestCloseRegionHandler
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/9237//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9237//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9237//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9237//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9237//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9237//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9237//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9237//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9237//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9237//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/9237//console
This message is automatically generated.
On master start, deadlock if refresh UI
---
Key: HBASE-10897
URL: https://issues.apache.org/jira/browse/HBASE-10897
Project: HBase
Issue Type: Bug
Affects Versions: 0.99.0
Reporter: stack
Assignee: Jimmy Xiang
Fix For: 0.99.0
Attachments: hbase-10897.patch, hbase-10897_v2.patch,
hbase-10897_v3.patch
Playing w/ MTTR recovery on trunk, master starting up deadlocked:
Waiting to finish active master initialization:
{code}
ActiveMasterManager daemon prio=10 tid=0x7fafb5dc3800 nid=0x5fb5
waiting for monitor entry [0x7faf8f57d000]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getKeepAliveZooKeeperWatcher(ConnectionManager.java:1683)
- waiting to lock 0x00064ab4b9a8 (a java.lang.Object)
at
org.apache.hadoop.hbase.client.ZooKeeperRegistry.getMetaRegionLocation(ZooKeeperRegistry.java:53)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1029)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:989)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getRegionLocation(ConnectionManager.java:830)
at
org.apache.hadoop.hbase.client.ConnectionAdapter.getRegionLocation(ConnectionAdapter.java:305)
at
org.apache.hadoop.hbase.client.RegionServerCallable.prepare(RegionServerCallable.java:77)
at
org.apache.hadoop.hbase.client.ScannerCallable.prepare(ScannerCallable.java:118)
at
[jira] [Commented] (HBASE-10897) On master start, deadlock if refresh UI
[
https://issues.apache.org/jira/browse/HBASE-10897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13964926#comment-13964926
]
stack commented on HBASE-10897:
---
A commit just broke
org.apache.hadoop.hbase.regionserver.handler.TestCloseRegionHandler ? This one
seems to show up a bunch now lately.
On master start, deadlock if refresh UI
---
Key: HBASE-10897
URL: https://issues.apache.org/jira/browse/HBASE-10897
Project: HBase
Issue Type: Bug
Affects Versions: 0.99.0
Reporter: stack
Assignee: Jimmy Xiang
Fix For: 0.99.0
Attachments: hbase-10897.patch, hbase-10897_v2.patch,
hbase-10897_v3.patch
Playing w/ MTTR recovery on trunk, master starting up deadlocked:
Waiting to finish active master initialization:
{code}
ActiveMasterManager daemon prio=10 tid=0x7fafb5dc3800 nid=0x5fb5
waiting for monitor entry [0x7faf8f57d000]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getKeepAliveZooKeeperWatcher(ConnectionManager.java:1683)
- waiting to lock 0x00064ab4b9a8 (a java.lang.Object)
at
org.apache.hadoop.hbase.client.ZooKeeperRegistry.getMetaRegionLocation(ZooKeeperRegistry.java:53)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1029)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:989)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getRegionLocation(ConnectionManager.java:830)
at
org.apache.hadoop.hbase.client.ConnectionAdapter.getRegionLocation(ConnectionAdapter.java:305)
at
org.apache.hadoop.hbase.client.RegionServerCallable.prepare(RegionServerCallable.java:77)
at
org.apache.hadoop.hbase.client.ScannerCallable.prepare(ScannerCallable.java:118)
at
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:101)
at
org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:264)
at
org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:169)
at
org.apache.hadoop.hbase.client.ClientScanner.init(ClientScanner.java:164)
at
org.apache.hadoop.hbase.client.ClientScanner.init(ClientScanner.java:107)
at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:766)
at
org.apache.hadoop.hbase.catalog.MetaReader.fullScan(MetaReader.java:539)
at
org.apache.hadoop.hbase.catalog.MetaReader.fullScanOfMeta(MetaReader.java:140)
at
org.apache.hadoop.hbase.catalog.MetaMigrationConvertingToPB.isMetaTableUpdated(MetaMigrationConvertingToPB.java:164)
at
org.apache.hadoop.hbase.catalog.MetaMigrationConvertingToPB.updateMetaIfNecessary(MetaMigrationConvertingToPB.java:131)
at
org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:567)
at org.apache.hadoop.hbase.master.HMaster.access$500(HMaster.java:147)
at org.apache.hadoop.hbase.master.HMaster$1.run(HMaster.java:1242)
at java.lang.Thread.run(Thread.java:744)
{code}
... but the master servlet has the lock while trying to access master:
{code}
686004346@qtp-2101021459-0 daemon prio=10 tid=0x7fafb5d2a800 nid=0x5fb1
waiting on condition [0x7faf8f87f000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$StubMaker.makeStub(ConnectionManager.java:1562)
- locked 0x00064ab4b9a8 (a java.lang.Object)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$MasterServiceStubMaker.makeStub(ConnectionManager.java:1597)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getKeepAliveMasterService(ConnectionManager.java:1805)
- locked 0x00064ab4b9a8 (a java.lang.Object)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.listTables(ConnectionManager.java:2481)
at
org.apache.hadoop.hbase.client.HBaseAdmin.listTables(HBaseAdmin.java:321)
at
org.apache.hadoop.hbase.tmpl.master.MasterStatusTmplImpl.__jamon_innerUnit__userTables(MasterStatusTmplImpl.java:530)
at
org.apache.hadoop.hbase.tmpl.master.MasterStatusTmplImpl.renderNoFlush(MasterStatusTmplImpl.java:255)
at
org.apache.hadoop.hbase.tmpl.master.MasterStatusTmpl.renderNoFlush(MasterStatusTmpl.java:382)
at
[jira] [Commented] (HBASE-10897) On master start, deadlock if refresh UI
[
https://issues.apache.org/jira/browse/HBASE-10897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13964936#comment-13964936
]
Mikhail Antonov commented on HBASE-10897:
-
I think it was in HBASE-10813. Shall it be reopened or new jira filed?
On master start, deadlock if refresh UI
---
Key: HBASE-10897
URL: https://issues.apache.org/jira/browse/HBASE-10897
Project: HBase
Issue Type: Bug
Affects Versions: 0.99.0
Reporter: stack
Assignee: Jimmy Xiang
Fix For: 0.99.0
Attachments: hbase-10897.patch, hbase-10897_v2.patch,
hbase-10897_v3.patch
Playing w/ MTTR recovery on trunk, master starting up deadlocked:
Waiting to finish active master initialization:
{code}
ActiveMasterManager daemon prio=10 tid=0x7fafb5dc3800 nid=0x5fb5
waiting for monitor entry [0x7faf8f57d000]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getKeepAliveZooKeeperWatcher(ConnectionManager.java:1683)
- waiting to lock 0x00064ab4b9a8 (a java.lang.Object)
at
org.apache.hadoop.hbase.client.ZooKeeperRegistry.getMetaRegionLocation(ZooKeeperRegistry.java:53)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1029)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:989)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getRegionLocation(ConnectionManager.java:830)
at
org.apache.hadoop.hbase.client.ConnectionAdapter.getRegionLocation(ConnectionAdapter.java:305)
at
org.apache.hadoop.hbase.client.RegionServerCallable.prepare(RegionServerCallable.java:77)
at
org.apache.hadoop.hbase.client.ScannerCallable.prepare(ScannerCallable.java:118)
at
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:101)
at
org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:264)
at
org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:169)
at
org.apache.hadoop.hbase.client.ClientScanner.init(ClientScanner.java:164)
at
org.apache.hadoop.hbase.client.ClientScanner.init(ClientScanner.java:107)
at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:766)
at
org.apache.hadoop.hbase.catalog.MetaReader.fullScan(MetaReader.java:539)
at
org.apache.hadoop.hbase.catalog.MetaReader.fullScanOfMeta(MetaReader.java:140)
at
org.apache.hadoop.hbase.catalog.MetaMigrationConvertingToPB.isMetaTableUpdated(MetaMigrationConvertingToPB.java:164)
at
org.apache.hadoop.hbase.catalog.MetaMigrationConvertingToPB.updateMetaIfNecessary(MetaMigrationConvertingToPB.java:131)
at
org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:567)
at org.apache.hadoop.hbase.master.HMaster.access$500(HMaster.java:147)
at org.apache.hadoop.hbase.master.HMaster$1.run(HMaster.java:1242)
at java.lang.Thread.run(Thread.java:744)
{code}
... but the master servlet has the lock while trying to access master:
{code}
686004346@qtp-2101021459-0 daemon prio=10 tid=0x7fafb5d2a800 nid=0x5fb1
waiting on condition [0x7faf8f87f000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$StubMaker.makeStub(ConnectionManager.java:1562)
- locked 0x00064ab4b9a8 (a java.lang.Object)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$MasterServiceStubMaker.makeStub(ConnectionManager.java:1597)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getKeepAliveMasterService(ConnectionManager.java:1805)
- locked 0x00064ab4b9a8 (a java.lang.Object)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.listTables(ConnectionManager.java:2481)
at
org.apache.hadoop.hbase.client.HBaseAdmin.listTables(HBaseAdmin.java:321)
at
org.apache.hadoop.hbase.tmpl.master.MasterStatusTmplImpl.__jamon_innerUnit__userTables(MasterStatusTmplImpl.java:530)
at
org.apache.hadoop.hbase.tmpl.master.MasterStatusTmplImpl.renderNoFlush(MasterStatusTmplImpl.java:255)
at
org.apache.hadoop.hbase.tmpl.master.MasterStatusTmpl.renderNoFlush(MasterStatusTmpl.java:382)
at
org.apache.hadoop.hbase.tmpl.master.MasterStatusTmpl.render(MasterStatusTmpl.java:372)
at
[jira] [Commented] (HBASE-10897) On master start, deadlock if refresh UI
[
https://issues.apache.org/jira/browse/HBASE-10897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13957837#comment-13957837
]
Jimmy Xiang commented on HBASE-10897:
-
I noticed this issue too. I think we should show Master not ready before
master finishes the initialization. Let me play with a simple patch.
On master start, deadlock if refresh UI
---
Key: HBASE-10897
URL: https://issues.apache.org/jira/browse/HBASE-10897
Project: HBase
Issue Type: Bug
Affects Versions: 0.99.0
Reporter: stack
Playing w/ MTTR recovery on trunk, master starting up deadlocked:
Waiting to finish active master initialization:
{code}
ActiveMasterManager daemon prio=10 tid=0x7fafb5dc3800 nid=0x5fb5
waiting for monitor entry [0x7faf8f57d000]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getKeepAliveZooKeeperWatcher(ConnectionManager.java:1683)
- waiting to lock 0x00064ab4b9a8 (a java.lang.Object)
at
org.apache.hadoop.hbase.client.ZooKeeperRegistry.getMetaRegionLocation(ZooKeeperRegistry.java:53)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1029)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:989)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getRegionLocation(ConnectionManager.java:830)
at
org.apache.hadoop.hbase.client.ConnectionAdapter.getRegionLocation(ConnectionAdapter.java:305)
at
org.apache.hadoop.hbase.client.RegionServerCallable.prepare(RegionServerCallable.java:77)
at
org.apache.hadoop.hbase.client.ScannerCallable.prepare(ScannerCallable.java:118)
at
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:101)
at
org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:264)
at
org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:169)
at
org.apache.hadoop.hbase.client.ClientScanner.init(ClientScanner.java:164)
at
org.apache.hadoop.hbase.client.ClientScanner.init(ClientScanner.java:107)
at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:766)
at
org.apache.hadoop.hbase.catalog.MetaReader.fullScan(MetaReader.java:539)
at
org.apache.hadoop.hbase.catalog.MetaReader.fullScanOfMeta(MetaReader.java:140)
at
org.apache.hadoop.hbase.catalog.MetaMigrationConvertingToPB.isMetaTableUpdated(MetaMigrationConvertingToPB.java:164)
at
org.apache.hadoop.hbase.catalog.MetaMigrationConvertingToPB.updateMetaIfNecessary(MetaMigrationConvertingToPB.java:131)
at
org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:567)
at org.apache.hadoop.hbase.master.HMaster.access$500(HMaster.java:147)
at org.apache.hadoop.hbase.master.HMaster$1.run(HMaster.java:1242)
at java.lang.Thread.run(Thread.java:744)
{code}
... but the master servlet has the lock while trying to access master:
{code}
686004346@qtp-2101021459-0 daemon prio=10 tid=0x7fafb5d2a800 nid=0x5fb1
waiting on condition [0x7faf8f87f000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$StubMaker.makeStub(ConnectionManager.java:1562)
- locked 0x00064ab4b9a8 (a java.lang.Object)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$MasterServiceStubMaker.makeStub(ConnectionManager.java:1597)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getKeepAliveMasterService(ConnectionManager.java:1805)
- locked 0x00064ab4b9a8 (a java.lang.Object)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.listTables(ConnectionManager.java:2481)
at
org.apache.hadoop.hbase.client.HBaseAdmin.listTables(HBaseAdmin.java:321)
at
org.apache.hadoop.hbase.tmpl.master.MasterStatusTmplImpl.__jamon_innerUnit__userTables(MasterStatusTmplImpl.java:530)
at
org.apache.hadoop.hbase.tmpl.master.MasterStatusTmplImpl.renderNoFlush(MasterStatusTmplImpl.java:255)
at
org.apache.hadoop.hbase.tmpl.master.MasterStatusTmpl.renderNoFlush(MasterStatusTmpl.java:382)
at
org.apache.hadoop.hbase.tmpl.master.MasterStatusTmpl.render(MasterStatusTmpl.java:372)
at
org.apache.hadoop.hbase.master.MasterStatusServlet.doGet(MasterStatusServlet.java:102)
...
[jira] [Commented] (HBASE-10897) On master start, deadlock if refresh UI
[
https://issues.apache.org/jira/browse/HBASE-10897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13957857#comment-13957857
]
stack commented on HBASE-10897:
---
Lets try it. Good one Jimmy +1
On master start, deadlock if refresh UI
---
Key: HBASE-10897
URL: https://issues.apache.org/jira/browse/HBASE-10897
Project: HBase
Issue Type: Bug
Affects Versions: 0.99.0
Reporter: stack
Attachments: hbase-10897.patch
Playing w/ MTTR recovery on trunk, master starting up deadlocked:
Waiting to finish active master initialization:
{code}
ActiveMasterManager daemon prio=10 tid=0x7fafb5dc3800 nid=0x5fb5
waiting for monitor entry [0x7faf8f57d000]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getKeepAliveZooKeeperWatcher(ConnectionManager.java:1683)
- waiting to lock 0x00064ab4b9a8 (a java.lang.Object)
at
org.apache.hadoop.hbase.client.ZooKeeperRegistry.getMetaRegionLocation(ZooKeeperRegistry.java:53)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1029)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:989)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getRegionLocation(ConnectionManager.java:830)
at
org.apache.hadoop.hbase.client.ConnectionAdapter.getRegionLocation(ConnectionAdapter.java:305)
at
org.apache.hadoop.hbase.client.RegionServerCallable.prepare(RegionServerCallable.java:77)
at
org.apache.hadoop.hbase.client.ScannerCallable.prepare(ScannerCallable.java:118)
at
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:101)
at
org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:264)
at
org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:169)
at
org.apache.hadoop.hbase.client.ClientScanner.init(ClientScanner.java:164)
at
org.apache.hadoop.hbase.client.ClientScanner.init(ClientScanner.java:107)
at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:766)
at
org.apache.hadoop.hbase.catalog.MetaReader.fullScan(MetaReader.java:539)
at
org.apache.hadoop.hbase.catalog.MetaReader.fullScanOfMeta(MetaReader.java:140)
at
org.apache.hadoop.hbase.catalog.MetaMigrationConvertingToPB.isMetaTableUpdated(MetaMigrationConvertingToPB.java:164)
at
org.apache.hadoop.hbase.catalog.MetaMigrationConvertingToPB.updateMetaIfNecessary(MetaMigrationConvertingToPB.java:131)
at
org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:567)
at org.apache.hadoop.hbase.master.HMaster.access$500(HMaster.java:147)
at org.apache.hadoop.hbase.master.HMaster$1.run(HMaster.java:1242)
at java.lang.Thread.run(Thread.java:744)
{code}
... but the master servlet has the lock while trying to access master:
{code}
686004346@qtp-2101021459-0 daemon prio=10 tid=0x7fafb5d2a800 nid=0x5fb1
waiting on condition [0x7faf8f87f000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$StubMaker.makeStub(ConnectionManager.java:1562)
- locked 0x00064ab4b9a8 (a java.lang.Object)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$MasterServiceStubMaker.makeStub(ConnectionManager.java:1597)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getKeepAliveMasterService(ConnectionManager.java:1805)
- locked 0x00064ab4b9a8 (a java.lang.Object)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.listTables(ConnectionManager.java:2481)
at
org.apache.hadoop.hbase.client.HBaseAdmin.listTables(HBaseAdmin.java:321)
at
org.apache.hadoop.hbase.tmpl.master.MasterStatusTmplImpl.__jamon_innerUnit__userTables(MasterStatusTmplImpl.java:530)
at
org.apache.hadoop.hbase.tmpl.master.MasterStatusTmplImpl.renderNoFlush(MasterStatusTmplImpl.java:255)
at
org.apache.hadoop.hbase.tmpl.master.MasterStatusTmpl.renderNoFlush(MasterStatusTmpl.java:382)
at
org.apache.hadoop.hbase.tmpl.master.MasterStatusTmpl.render(MasterStatusTmpl.java:372)
at
org.apache.hadoop.hbase.master.MasterStatusServlet.doGet(MasterStatusServlet.java:102)
...
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-10897) On master start, deadlock if refresh UI
[
https://issues.apache.org/jira/browse/HBASE-10897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13957860#comment-13957860
]
Jimmy Xiang commented on HBASE-10897:
-
With this patch, I don't see the problem any more. [~stack], you want to give
it a try?
If the master hasn't finished the initialization, currently, we show this:
{noformat}
HTTP ERROR 503
Problem accessing /master-status. Reason:
Master not ready
Powered by Jetty://
{noformat}
Should we show something like this, which looks like more friendly?
{noformat}
The Master is initializing!
{noformat}
On master start, deadlock if refresh UI
---
Key: HBASE-10897
URL: https://issues.apache.org/jira/browse/HBASE-10897
Project: HBase
Issue Type: Bug
Affects Versions: 0.99.0
Reporter: stack
Attachments: hbase-10897.patch
Playing w/ MTTR recovery on trunk, master starting up deadlocked:
Waiting to finish active master initialization:
{code}
ActiveMasterManager daemon prio=10 tid=0x7fafb5dc3800 nid=0x5fb5
waiting for monitor entry [0x7faf8f57d000]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getKeepAliveZooKeeperWatcher(ConnectionManager.java:1683)
- waiting to lock 0x00064ab4b9a8 (a java.lang.Object)
at
org.apache.hadoop.hbase.client.ZooKeeperRegistry.getMetaRegionLocation(ZooKeeperRegistry.java:53)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1029)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:989)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getRegionLocation(ConnectionManager.java:830)
at
org.apache.hadoop.hbase.client.ConnectionAdapter.getRegionLocation(ConnectionAdapter.java:305)
at
org.apache.hadoop.hbase.client.RegionServerCallable.prepare(RegionServerCallable.java:77)
at
org.apache.hadoop.hbase.client.ScannerCallable.prepare(ScannerCallable.java:118)
at
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:101)
at
org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:264)
at
org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:169)
at
org.apache.hadoop.hbase.client.ClientScanner.init(ClientScanner.java:164)
at
org.apache.hadoop.hbase.client.ClientScanner.init(ClientScanner.java:107)
at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:766)
at
org.apache.hadoop.hbase.catalog.MetaReader.fullScan(MetaReader.java:539)
at
org.apache.hadoop.hbase.catalog.MetaReader.fullScanOfMeta(MetaReader.java:140)
at
org.apache.hadoop.hbase.catalog.MetaMigrationConvertingToPB.isMetaTableUpdated(MetaMigrationConvertingToPB.java:164)
at
org.apache.hadoop.hbase.catalog.MetaMigrationConvertingToPB.updateMetaIfNecessary(MetaMigrationConvertingToPB.java:131)
at
org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:567)
at org.apache.hadoop.hbase.master.HMaster.access$500(HMaster.java:147)
at org.apache.hadoop.hbase.master.HMaster$1.run(HMaster.java:1242)
at java.lang.Thread.run(Thread.java:744)
{code}
... but the master servlet has the lock while trying to access master:
{code}
686004346@qtp-2101021459-0 daemon prio=10 tid=0x7fafb5d2a800 nid=0x5fb1
waiting on condition [0x7faf8f87f000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$StubMaker.makeStub(ConnectionManager.java:1562)
- locked 0x00064ab4b9a8 (a java.lang.Object)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$MasterServiceStubMaker.makeStub(ConnectionManager.java:1597)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getKeepAliveMasterService(ConnectionManager.java:1805)
- locked 0x00064ab4b9a8 (a java.lang.Object)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.listTables(ConnectionManager.java:2481)
at
org.apache.hadoop.hbase.client.HBaseAdmin.listTables(HBaseAdmin.java:321)
at
org.apache.hadoop.hbase.tmpl.master.MasterStatusTmplImpl.__jamon_innerUnit__userTables(MasterStatusTmplImpl.java:530)
at
org.apache.hadoop.hbase.tmpl.master.MasterStatusTmplImpl.renderNoFlush(MasterStatusTmplImpl.java:255)
at
[jira] [Commented] (HBASE-10897) On master start, deadlock if refresh UI
[
https://issues.apache.org/jira/browse/HBASE-10897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13957871#comment-13957871
]
stack commented on HBASE-10897:
---
[~jxiang] commit it. I'm testing something else but as side effect will notice
if this comes up again.
On master start, deadlock if refresh UI
---
Key: HBASE-10897
URL: https://issues.apache.org/jira/browse/HBASE-10897
Project: HBase
Issue Type: Bug
Affects Versions: 0.99.0
Reporter: stack
Attachments: hbase-10897.patch
Playing w/ MTTR recovery on trunk, master starting up deadlocked:
Waiting to finish active master initialization:
{code}
ActiveMasterManager daemon prio=10 tid=0x7fafb5dc3800 nid=0x5fb5
waiting for monitor entry [0x7faf8f57d000]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getKeepAliveZooKeeperWatcher(ConnectionManager.java:1683)
- waiting to lock 0x00064ab4b9a8 (a java.lang.Object)
at
org.apache.hadoop.hbase.client.ZooKeeperRegistry.getMetaRegionLocation(ZooKeeperRegistry.java:53)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1029)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:989)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getRegionLocation(ConnectionManager.java:830)
at
org.apache.hadoop.hbase.client.ConnectionAdapter.getRegionLocation(ConnectionAdapter.java:305)
at
org.apache.hadoop.hbase.client.RegionServerCallable.prepare(RegionServerCallable.java:77)
at
org.apache.hadoop.hbase.client.ScannerCallable.prepare(ScannerCallable.java:118)
at
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:101)
at
org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:264)
at
org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:169)
at
org.apache.hadoop.hbase.client.ClientScanner.init(ClientScanner.java:164)
at
org.apache.hadoop.hbase.client.ClientScanner.init(ClientScanner.java:107)
at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:766)
at
org.apache.hadoop.hbase.catalog.MetaReader.fullScan(MetaReader.java:539)
at
org.apache.hadoop.hbase.catalog.MetaReader.fullScanOfMeta(MetaReader.java:140)
at
org.apache.hadoop.hbase.catalog.MetaMigrationConvertingToPB.isMetaTableUpdated(MetaMigrationConvertingToPB.java:164)
at
org.apache.hadoop.hbase.catalog.MetaMigrationConvertingToPB.updateMetaIfNecessary(MetaMigrationConvertingToPB.java:131)
at
org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:567)
at org.apache.hadoop.hbase.master.HMaster.access$500(HMaster.java:147)
at org.apache.hadoop.hbase.master.HMaster$1.run(HMaster.java:1242)
at java.lang.Thread.run(Thread.java:744)
{code}
... but the master servlet has the lock while trying to access master:
{code}
686004346@qtp-2101021459-0 daemon prio=10 tid=0x7fafb5d2a800 nid=0x5fb1
waiting on condition [0x7faf8f87f000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$StubMaker.makeStub(ConnectionManager.java:1562)
- locked 0x00064ab4b9a8 (a java.lang.Object)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$MasterServiceStubMaker.makeStub(ConnectionManager.java:1597)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getKeepAliveMasterService(ConnectionManager.java:1805)
- locked 0x00064ab4b9a8 (a java.lang.Object)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.listTables(ConnectionManager.java:2481)
at
org.apache.hadoop.hbase.client.HBaseAdmin.listTables(HBaseAdmin.java:321)
at
org.apache.hadoop.hbase.tmpl.master.MasterStatusTmplImpl.__jamon_innerUnit__userTables(MasterStatusTmplImpl.java:530)
at
org.apache.hadoop.hbase.tmpl.master.MasterStatusTmplImpl.renderNoFlush(MasterStatusTmplImpl.java:255)
at
org.apache.hadoop.hbase.tmpl.master.MasterStatusTmpl.renderNoFlush(MasterStatusTmpl.java:382)
at
org.apache.hadoop.hbase.tmpl.master.MasterStatusTmpl.render(MasterStatusTmpl.java:372)
at
org.apache.hadoop.hbase.master.MasterStatusServlet.doGet(MasterStatusServlet.java:102)
...
{code}
--
[jira] [Commented] (HBASE-10897) On master start, deadlock if refresh UI
[
https://issues.apache.org/jira/browse/HBASE-10897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13958125#comment-13958125
]
Hudson commented on HBASE-10897:
SUCCESS: Integrated in HBase-TRUNK #5058 (See
[https://builds.apache.org/job/HBase-TRUNK/5058/])
HBASE-10897 On master start, deadlock if refresh UI (jxiang: rev 1584091)
*
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterStatusServlet.java
On master start, deadlock if refresh UI
---
Key: HBASE-10897
URL: https://issues.apache.org/jira/browse/HBASE-10897
Project: HBase
Issue Type: Bug
Affects Versions: 0.99.0
Reporter: stack
Assignee: Jimmy Xiang
Fix For: 0.99.0
Attachments: hbase-10897.patch
Playing w/ MTTR recovery on trunk, master starting up deadlocked:
Waiting to finish active master initialization:
{code}
ActiveMasterManager daemon prio=10 tid=0x7fafb5dc3800 nid=0x5fb5
waiting for monitor entry [0x7faf8f57d000]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getKeepAliveZooKeeperWatcher(ConnectionManager.java:1683)
- waiting to lock 0x00064ab4b9a8 (a java.lang.Object)
at
org.apache.hadoop.hbase.client.ZooKeeperRegistry.getMetaRegionLocation(ZooKeeperRegistry.java:53)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1029)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:989)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getRegionLocation(ConnectionManager.java:830)
at
org.apache.hadoop.hbase.client.ConnectionAdapter.getRegionLocation(ConnectionAdapter.java:305)
at
org.apache.hadoop.hbase.client.RegionServerCallable.prepare(RegionServerCallable.java:77)
at
org.apache.hadoop.hbase.client.ScannerCallable.prepare(ScannerCallable.java:118)
at
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:101)
at
org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:264)
at
org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:169)
at
org.apache.hadoop.hbase.client.ClientScanner.init(ClientScanner.java:164)
at
org.apache.hadoop.hbase.client.ClientScanner.init(ClientScanner.java:107)
at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:766)
at
org.apache.hadoop.hbase.catalog.MetaReader.fullScan(MetaReader.java:539)
at
org.apache.hadoop.hbase.catalog.MetaReader.fullScanOfMeta(MetaReader.java:140)
at
org.apache.hadoop.hbase.catalog.MetaMigrationConvertingToPB.isMetaTableUpdated(MetaMigrationConvertingToPB.java:164)
at
org.apache.hadoop.hbase.catalog.MetaMigrationConvertingToPB.updateMetaIfNecessary(MetaMigrationConvertingToPB.java:131)
at
org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:567)
at org.apache.hadoop.hbase.master.HMaster.access$500(HMaster.java:147)
at org.apache.hadoop.hbase.master.HMaster$1.run(HMaster.java:1242)
at java.lang.Thread.run(Thread.java:744)
{code}
... but the master servlet has the lock while trying to access master:
{code}
686004346@qtp-2101021459-0 daemon prio=10 tid=0x7fafb5d2a800 nid=0x5fb1
waiting on condition [0x7faf8f87f000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$StubMaker.makeStub(ConnectionManager.java:1562)
- locked 0x00064ab4b9a8 (a java.lang.Object)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$MasterServiceStubMaker.makeStub(ConnectionManager.java:1597)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getKeepAliveMasterService(ConnectionManager.java:1805)
- locked 0x00064ab4b9a8 (a java.lang.Object)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.listTables(ConnectionManager.java:2481)
at
org.apache.hadoop.hbase.client.HBaseAdmin.listTables(HBaseAdmin.java:321)
at
org.apache.hadoop.hbase.tmpl.master.MasterStatusTmplImpl.__jamon_innerUnit__userTables(MasterStatusTmplImpl.java:530)
at
org.apache.hadoop.hbase.tmpl.master.MasterStatusTmplImpl.renderNoFlush(MasterStatusTmplImpl.java:255)
at
[jira] [Commented] (HBASE-10897) On master start, deadlock if refresh UI
[
https://issues.apache.org/jira/browse/HBASE-10897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13957327#comment-13957327
]
stack commented on HBASE-10897:
---
We should not allow listing tables until active master [~jxiang]?
On master start, deadlock if refresh UI
---
Key: HBASE-10897
URL: https://issues.apache.org/jira/browse/HBASE-10897
Project: HBase
Issue Type: Bug
Affects Versions: 0.99.0
Reporter: stack
Playing w/ MTTR recovery on trunk, master starting up deadlocked:
Waiting to finish active master initialization:
{code}
ActiveMasterManager daemon prio=10 tid=0x7fafb5dc3800 nid=0x5fb5
waiting for monitor entry [0x7faf8f57d000]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getKeepAliveZooKeeperWatcher(ConnectionManager.java:1683)
- waiting to lock 0x00064ab4b9a8 (a java.lang.Object)
at
org.apache.hadoop.hbase.client.ZooKeeperRegistry.getMetaRegionLocation(ZooKeeperRegistry.java:53)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1029)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:989)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getRegionLocation(ConnectionManager.java:830)
at
org.apache.hadoop.hbase.client.ConnectionAdapter.getRegionLocation(ConnectionAdapter.java:305)
at
org.apache.hadoop.hbase.client.RegionServerCallable.prepare(RegionServerCallable.java:77)
at
org.apache.hadoop.hbase.client.ScannerCallable.prepare(ScannerCallable.java:118)
at
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:101)
at
org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:264)
at
org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:169)
at
org.apache.hadoop.hbase.client.ClientScanner.init(ClientScanner.java:164)
at
org.apache.hadoop.hbase.client.ClientScanner.init(ClientScanner.java:107)
at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:766)
at
org.apache.hadoop.hbase.catalog.MetaReader.fullScan(MetaReader.java:539)
at
org.apache.hadoop.hbase.catalog.MetaReader.fullScanOfMeta(MetaReader.java:140)
at
org.apache.hadoop.hbase.catalog.MetaMigrationConvertingToPB.isMetaTableUpdated(MetaMigrationConvertingToPB.java:164)
at
org.apache.hadoop.hbase.catalog.MetaMigrationConvertingToPB.updateMetaIfNecessary(MetaMigrationConvertingToPB.java:131)
at
org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:567)
at org.apache.hadoop.hbase.master.HMaster.access$500(HMaster.java:147)
at org.apache.hadoop.hbase.master.HMaster$1.run(HMaster.java:1242)
at java.lang.Thread.run(Thread.java:744)
{code}
... but the master servlet has the lock while trying to access master:
{code}
686004346@qtp-2101021459-0 daemon prio=10 tid=0x7fafb5d2a800 nid=0x5fb1
waiting on condition [0x7faf8f87f000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$StubMaker.makeStub(ConnectionManager.java:1562)
- locked 0x00064ab4b9a8 (a java.lang.Object)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$MasterServiceStubMaker.makeStub(ConnectionManager.java:1597)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getKeepAliveMasterService(ConnectionManager.java:1805)
- locked 0x00064ab4b9a8 (a java.lang.Object)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.listTables(ConnectionManager.java:2481)
at
org.apache.hadoop.hbase.client.HBaseAdmin.listTables(HBaseAdmin.java:321)
at
org.apache.hadoop.hbase.tmpl.master.MasterStatusTmplImpl.__jamon_innerUnit__userTables(MasterStatusTmplImpl.java:530)
at
org.apache.hadoop.hbase.tmpl.master.MasterStatusTmplImpl.renderNoFlush(MasterStatusTmplImpl.java:255)
at
org.apache.hadoop.hbase.tmpl.master.MasterStatusTmpl.renderNoFlush(MasterStatusTmpl.java:382)
at
org.apache.hadoop.hbase.tmpl.master.MasterStatusTmpl.render(MasterStatusTmpl.java:372)
at
org.apache.hadoop.hbase.master.MasterStatusServlet.doGet(MasterStatusServlet.java:102)
...
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
