[jira] [Commented] (HBASE-7259) Deadlock in HBaseClient when KeeperException occured
[
https://issues.apache.org/jira/browse/HBASE-7259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=1357#comment-1357
]
Hudson commented on HBASE-7259:
---
Integrated in HBase-0.94-security-on-Hadoop-23 #10 (See
[https://builds.apache.org/job/HBase-0.94-security-on-Hadoop-23/10/])
HBASE-7259 Deadlock in HBaseClient when KeeperException occured (liwei)
(Revision 1424233)
Result = FAILURE
larsh :
Files :
*
/hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java
Deadlock in HBaseClient when KeeperException occured
Key: HBASE-7259
URL: https://issues.apache.org/jira/browse/HBASE-7259
Project: HBase
Issue Type: Bug
Components: Zookeeper
Affects Versions: 0.94.0, 0.94.1, 0.94.2
Reporter: sina-liwei
Priority: Critical
Fix For: 0.94.4
Attachments: 7259-0.94-branch.txt, HBASE-7259-0.94.2.txt
HBaseClient was running after a period of time, all of get operation became
too slow.
From the client logs I could see the following:
1. Unable to get data of znode /hbase/root-region-server
{code}
java.lang.InterruptedException
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1253)
at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1129)
at
org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:264)
at
org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataInternal(ZKUtil.java:522)
at
org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataAndWatch(ZKUtil.java:498)
at
org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.getData(ZooKeeperNodeTracker.java:156)
at
org.apache.hadoop.hbase.zookeeper.RootRegionTracker.getRootRegionLocation(RootRegionTracker.java:62)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:821)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:933)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:832)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:234)
at org.apache.hadoop.hbase.client.HTable.init(HTable.java:174)
at
org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:150)
at
org.apache.hadoop.hbase.client.MetaScanner.access$000(MetaScanner.java:48)
at
org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:126)
at
org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:123)
at
org.apache.hadoop.hbase.client.HConnectionManager.execute(HConnectionManager.java:359)
at
org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:123)
at
org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:99)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.prefetchRegionCache(HConnectionManager.java:894)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:948)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:836)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionLocation(HConnectionManager.java:725)
at
org.apache.hadoop.hbase.client.ServerCallable.connect(ServerCallable.java:82)
at
org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:162)
at org.apache.hadoop.hbase.client.HTable.get(HTable.java:685)
at
org.apache.hadoop.hbase.client.HTablePool$PooledHTable.get(HTablePool.java:366)
{code}
2. Catalina.out found one Java-level deadlock:
{code}
=
catalina-exec-800:
waiting to lock monitor 0x5f1f6530 (object 0x000731902200, a
java.lang.Object),
which is held by catalina-exec-710
catalina-exec-710:
waiting to lock monitor 0x2aaab9a05bd0 (object 0x0007321f8708, a
[jira] [Commented] (HBASE-7259) Deadlock in HBaseClient when KeeperException occured
[
https://issues.apache.org/jira/browse/HBASE-7259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13537919#comment-13537919
]
Hudson commented on HBASE-7259:
---
Integrated in HBase-0.94-security #87 (See
[https://builds.apache.org/job/HBase-0.94-security/87/])
HBASE-7259 Deadlock in HBaseClient when KeeperException occured (liwei)
(Revision 1424233)
Result = SUCCESS
larsh :
Files :
*
/hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java
Deadlock in HBaseClient when KeeperException occured
Key: HBASE-7259
URL: https://issues.apache.org/jira/browse/HBASE-7259
Project: HBase
Issue Type: Bug
Components: Zookeeper
Affects Versions: 0.94.0, 0.94.1, 0.94.2
Reporter: liwei
Priority: Critical
Fix For: 0.94.4
Attachments: 7259-0.94-branch.txt, HBASE-7259-0.94.2.txt
HBaseClient was running after a period of time, all of get operation became
too slow.
From the client logs I could see the following:
1. Unable to get data of znode /hbase/root-region-server
{code}
java.lang.InterruptedException
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1253)
at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1129)
at
org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:264)
at
org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataInternal(ZKUtil.java:522)
at
org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataAndWatch(ZKUtil.java:498)
at
org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.getData(ZooKeeperNodeTracker.java:156)
at
org.apache.hadoop.hbase.zookeeper.RootRegionTracker.getRootRegionLocation(RootRegionTracker.java:62)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:821)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:933)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:832)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:234)
at org.apache.hadoop.hbase.client.HTable.init(HTable.java:174)
at
org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:150)
at
org.apache.hadoop.hbase.client.MetaScanner.access$000(MetaScanner.java:48)
at
org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:126)
at
org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:123)
at
org.apache.hadoop.hbase.client.HConnectionManager.execute(HConnectionManager.java:359)
at
org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:123)
at
org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:99)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.prefetchRegionCache(HConnectionManager.java:894)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:948)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:836)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionLocation(HConnectionManager.java:725)
at
org.apache.hadoop.hbase.client.ServerCallable.connect(ServerCallable.java:82)
at
org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:162)
at org.apache.hadoop.hbase.client.HTable.get(HTable.java:685)
at
org.apache.hadoop.hbase.client.HTablePool$PooledHTable.get(HTablePool.java:366)
{code}
2. Catalina.out found one Java-level deadlock:
{code}
=
catalina-exec-800:
waiting to lock monitor 0x5f1f6530 (object 0x000731902200, a
java.lang.Object),
which is held by catalina-exec-710
catalina-exec-710:
waiting to lock monitor 0x2aaab9a05bd0 (object 0x0007321f8708, a
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation),
which
[jira] [Commented] (HBASE-7259) Deadlock in HBaseClient when KeeperException occured
[
https://issues.apache.org/jira/browse/HBASE-7259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13536574#comment-13536574
]
Lars Hofhansl commented on HBASE-7259:
--
Going to commit this to 0.94 (after fixing reseting to resetting :) ), unless I
hear objections.
Deadlock in HBaseClient when KeeperException occured
Key: HBASE-7259
URL: https://issues.apache.org/jira/browse/HBASE-7259
Project: HBase
Issue Type: Bug
Components: Zookeeper
Affects Versions: 0.94.0, 0.94.1, 0.94.2
Reporter: liwei
Priority: Critical
Fix For: 0.94.4
Attachments: 7259-0.94-branch.txt, HBASE-7259-0.94.2.txt
HBaseClient was running after a period of time, all of get operation became
too slow.
From the client logs I could see the following:
1. Unable to get data of znode /hbase/root-region-server
{code}
java.lang.InterruptedException
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1253)
at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1129)
at
org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:264)
at
org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataInternal(ZKUtil.java:522)
at
org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataAndWatch(ZKUtil.java:498)
at
org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.getData(ZooKeeperNodeTracker.java:156)
at
org.apache.hadoop.hbase.zookeeper.RootRegionTracker.getRootRegionLocation(RootRegionTracker.java:62)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:821)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:933)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:832)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:234)
at org.apache.hadoop.hbase.client.HTable.init(HTable.java:174)
at
org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:150)
at
org.apache.hadoop.hbase.client.MetaScanner.access$000(MetaScanner.java:48)
at
org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:126)
at
org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:123)
at
org.apache.hadoop.hbase.client.HConnectionManager.execute(HConnectionManager.java:359)
at
org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:123)
at
org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:99)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.prefetchRegionCache(HConnectionManager.java:894)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:948)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:836)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionLocation(HConnectionManager.java:725)
at
org.apache.hadoop.hbase.client.ServerCallable.connect(ServerCallable.java:82)
at
org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:162)
at org.apache.hadoop.hbase.client.HTable.get(HTable.java:685)
at
org.apache.hadoop.hbase.client.HTablePool$PooledHTable.get(HTablePool.java:366)
{code}
2. Catalina.out found one Java-level deadlock:
{code}
=
catalina-exec-800:
waiting to lock monitor 0x5f1f6530 (object 0x000731902200, a
java.lang.Object),
which is held by catalina-exec-710
catalina-exec-710:
waiting to lock monitor 0x2aaab9a05bd0 (object 0x0007321f8708, a
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation),
which is held by catalina-exec-29-EventThread
catalina-exec-29-EventThread:
waiting to lock monitor 0x5f9f0af0 (object 0x000732a9c7e0, a
org.apache.hadoop.hbase.zookeeper.RootRegionTracker),
which is held
[jira] [Commented] (HBASE-7259) Deadlock in HBaseClient when KeeperException occured
[
https://issues.apache.org/jira/browse/HBASE-7259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13536609#comment-13536609
]
Lars Hofhansl commented on HBASE-7259:
--
Committed to 0.94. Thanks for the patch [~boneylw]
Deadlock in HBaseClient when KeeperException occured
Key: HBASE-7259
URL: https://issues.apache.org/jira/browse/HBASE-7259
Project: HBase
Issue Type: Bug
Components: Zookeeper
Affects Versions: 0.94.0, 0.94.1, 0.94.2
Reporter: liwei
Priority: Critical
Fix For: 0.94.4
Attachments: 7259-0.94-branch.txt, HBASE-7259-0.94.2.txt
HBaseClient was running after a period of time, all of get operation became
too slow.
From the client logs I could see the following:
1. Unable to get data of znode /hbase/root-region-server
{code}
java.lang.InterruptedException
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1253)
at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1129)
at
org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:264)
at
org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataInternal(ZKUtil.java:522)
at
org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataAndWatch(ZKUtil.java:498)
at
org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.getData(ZooKeeperNodeTracker.java:156)
at
org.apache.hadoop.hbase.zookeeper.RootRegionTracker.getRootRegionLocation(RootRegionTracker.java:62)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:821)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:933)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:832)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:234)
at org.apache.hadoop.hbase.client.HTable.init(HTable.java:174)
at
org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:150)
at
org.apache.hadoop.hbase.client.MetaScanner.access$000(MetaScanner.java:48)
at
org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:126)
at
org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:123)
at
org.apache.hadoop.hbase.client.HConnectionManager.execute(HConnectionManager.java:359)
at
org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:123)
at
org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:99)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.prefetchRegionCache(HConnectionManager.java:894)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:948)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:836)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionLocation(HConnectionManager.java:725)
at
org.apache.hadoop.hbase.client.ServerCallable.connect(ServerCallable.java:82)
at
org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:162)
at org.apache.hadoop.hbase.client.HTable.get(HTable.java:685)
at
org.apache.hadoop.hbase.client.HTablePool$PooledHTable.get(HTablePool.java:366)
{code}
2. Catalina.out found one Java-level deadlock:
{code}
=
catalina-exec-800:
waiting to lock monitor 0x5f1f6530 (object 0x000731902200, a
java.lang.Object),
which is held by catalina-exec-710
catalina-exec-710:
waiting to lock monitor 0x2aaab9a05bd0 (object 0x0007321f8708, a
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation),
which is held by catalina-exec-29-EventThread
catalina-exec-29-EventThread:
waiting to lock monitor 0x5f9f0af0 (object 0x000732a9c7e0, a
org.apache.hadoop.hbase.zookeeper.RootRegionTracker),
which is held by catalina-exec-710
Java stack information
[jira] [Commented] (HBASE-7259) Deadlock in HBaseClient when KeeperException occured
[
https://issues.apache.org/jira/browse/HBASE-7259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13536708#comment-13536708
]
Hudson commented on HBASE-7259:
---
Integrated in HBase-0.94 #647 (See
[https://builds.apache.org/job/HBase-0.94/647/])
HBASE-7259 Deadlock in HBaseClient when KeeperException occured (liwei)
(Revision 1424233)
Result = FAILURE
larsh :
Files :
*
/hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java
Deadlock in HBaseClient when KeeperException occured
Key: HBASE-7259
URL: https://issues.apache.org/jira/browse/HBASE-7259
Project: HBase
Issue Type: Bug
Components: Zookeeper
Affects Versions: 0.94.0, 0.94.1, 0.94.2
Reporter: liwei
Priority: Critical
Fix For: 0.94.4
Attachments: 7259-0.94-branch.txt, HBASE-7259-0.94.2.txt
HBaseClient was running after a period of time, all of get operation became
too slow.
From the client logs I could see the following:
1. Unable to get data of znode /hbase/root-region-server
{code}
java.lang.InterruptedException
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1253)
at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1129)
at
org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:264)
at
org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataInternal(ZKUtil.java:522)
at
org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataAndWatch(ZKUtil.java:498)
at
org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.getData(ZooKeeperNodeTracker.java:156)
at
org.apache.hadoop.hbase.zookeeper.RootRegionTracker.getRootRegionLocation(RootRegionTracker.java:62)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:821)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:933)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:832)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:234)
at org.apache.hadoop.hbase.client.HTable.init(HTable.java:174)
at
org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:150)
at
org.apache.hadoop.hbase.client.MetaScanner.access$000(MetaScanner.java:48)
at
org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:126)
at
org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:123)
at
org.apache.hadoop.hbase.client.HConnectionManager.execute(HConnectionManager.java:359)
at
org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:123)
at
org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:99)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.prefetchRegionCache(HConnectionManager.java:894)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:948)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:836)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionLocation(HConnectionManager.java:725)
at
org.apache.hadoop.hbase.client.ServerCallable.connect(ServerCallable.java:82)
at
org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:162)
at org.apache.hadoop.hbase.client.HTable.get(HTable.java:685)
at
org.apache.hadoop.hbase.client.HTablePool$PooledHTable.get(HTablePool.java:366)
{code}
2. Catalina.out found one Java-level deadlock:
{code}
=
catalina-exec-800:
waiting to lock monitor 0x5f1f6530 (object 0x000731902200, a
java.lang.Object),
which is held by catalina-exec-710
catalina-exec-710:
waiting to lock monitor 0x2aaab9a05bd0 (object 0x0007321f8708, a
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation),
which is held by
[jira] [Commented] (HBASE-7259) Deadlock in HBaseClient when KeeperException occured
[
https://issues.apache.org/jira/browse/HBASE-7259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13535646#comment-13535646
]
Lars Hofhansl commented on HBASE-7259:
--
Can we wrap that logic into resetZookeeperTrackers?
Otherwise I'm fine to commit this.
Deadlock in HBaseClient when KeeperException occured
Key: HBASE-7259
URL: https://issues.apache.org/jira/browse/HBASE-7259
Project: HBase
Issue Type: Bug
Components: Zookeeper
Affects Versions: 0.94.0, 0.94.1, 0.94.2
Reporter: liwei
Priority: Critical
Fix For: 0.94.4
Attachments: 7259-0.94-branch.txt, HBASE-7259-0.94.2.txt
HBaseClient was running after a period of time, all of get operation became
too slow.
From the client logs I could see the following:
1. Unable to get data of znode /hbase/root-region-server
{code}
java.lang.InterruptedException
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1253)
at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1129)
at
org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:264)
at
org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataInternal(ZKUtil.java:522)
at
org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataAndWatch(ZKUtil.java:498)
at
org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.getData(ZooKeeperNodeTracker.java:156)
at
org.apache.hadoop.hbase.zookeeper.RootRegionTracker.getRootRegionLocation(RootRegionTracker.java:62)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:821)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:933)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:832)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:234)
at org.apache.hadoop.hbase.client.HTable.init(HTable.java:174)
at
org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:150)
at
org.apache.hadoop.hbase.client.MetaScanner.access$000(MetaScanner.java:48)
at
org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:126)
at
org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:123)
at
org.apache.hadoop.hbase.client.HConnectionManager.execute(HConnectionManager.java:359)
at
org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:123)
at
org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:99)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.prefetchRegionCache(HConnectionManager.java:894)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:948)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:836)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionLocation(HConnectionManager.java:725)
at
org.apache.hadoop.hbase.client.ServerCallable.connect(ServerCallable.java:82)
at
org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:162)
at org.apache.hadoop.hbase.client.HTable.get(HTable.java:685)
at
org.apache.hadoop.hbase.client.HTablePool$PooledHTable.get(HTablePool.java:366)
{code}
2. Catalina.out found one Java-level deadlock:
{code}
=
catalina-exec-800:
waiting to lock monitor 0x5f1f6530 (object 0x000731902200, a
java.lang.Object),
which is held by catalina-exec-710
catalina-exec-710:
waiting to lock monitor 0x2aaab9a05bd0 (object 0x0007321f8708, a
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation),
which is held by catalina-exec-29-EventThread
catalina-exec-29-EventThread:
waiting to lock monitor 0x5f9f0af0 (object 0x000732a9c7e0, a
org.apache.hadoop.hbase.zookeeper.RootRegionTracker),
which is held by
[jira] [Commented] (HBASE-7259) Deadlock in HBaseClient when KeeperException occured
[
https://issues.apache.org/jira/browse/HBASE-7259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13532892#comment-13532892
]
Sergey Shelukhin commented on HBASE-7259:
-
This appears to have stalled...
Deadlock in HBaseClient when KeeperException occured
Key: HBASE-7259
URL: https://issues.apache.org/jira/browse/HBASE-7259
Project: HBase
Issue Type: Bug
Components: Zookeeper
Affects Versions: 0.94.0, 0.94.1, 0.94.2
Reporter: liwei
Priority: Critical
Fix For: 0.94.4
Attachments: 7259-0.94-branch.txt, HBASE-7259-0.94.2.txt
HBaseClient was running after a period of time, all of get operation became
too slow.
From the client logs I could see the following:
1. Unable to get data of znode /hbase/root-region-server
{code}
java.lang.InterruptedException
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1253)
at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1129)
at
org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:264)
at
org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataInternal(ZKUtil.java:522)
at
org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataAndWatch(ZKUtil.java:498)
at
org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.getData(ZooKeeperNodeTracker.java:156)
at
org.apache.hadoop.hbase.zookeeper.RootRegionTracker.getRootRegionLocation(RootRegionTracker.java:62)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:821)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:933)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:832)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:234)
at org.apache.hadoop.hbase.client.HTable.init(HTable.java:174)
at
org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:150)
at
org.apache.hadoop.hbase.client.MetaScanner.access$000(MetaScanner.java:48)
at
org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:126)
at
org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:123)
at
org.apache.hadoop.hbase.client.HConnectionManager.execute(HConnectionManager.java:359)
at
org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:123)
at
org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:99)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.prefetchRegionCache(HConnectionManager.java:894)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:948)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:836)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionLocation(HConnectionManager.java:725)
at
org.apache.hadoop.hbase.client.ServerCallable.connect(ServerCallable.java:82)
at
org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:162)
at org.apache.hadoop.hbase.client.HTable.get(HTable.java:685)
at
org.apache.hadoop.hbase.client.HTablePool$PooledHTable.get(HTablePool.java:366)
{code}
2. Catalina.out found one Java-level deadlock:
{code}
=
catalina-exec-800:
waiting to lock monitor 0x5f1f6530 (object 0x000731902200, a
java.lang.Object),
which is held by catalina-exec-710
catalina-exec-710:
waiting to lock monitor 0x2aaab9a05bd0 (object 0x0007321f8708, a
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation),
which is held by catalina-exec-29-EventThread
catalina-exec-29-EventThread:
waiting to lock monitor 0x5f9f0af0 (object 0x000732a9c7e0, a
org.apache.hadoop.hbase.zookeeper.RootRegionTracker),
which is held by catalina-exec-710
Java stack information for the
[jira] [Commented] (HBASE-7259) Deadlock in HBaseClient when KeeperException occured
[ https://issues.apache.org/jira/browse/HBASE-7259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13508573#comment-13508573 ] Ted Yu commented on HBASE-7259: --- Can you provide patch for trunk so that Hadoop QA can run through test suite ? Thanks Deadlock in HBaseClient when KeeperException occured Key: HBASE-7259 URL: https://issues.apache.org/jira/browse/HBASE-7259 Project: HBase Issue Type: Bug Components: Zookeeper Affects Versions: 0.94.0, 0.94.1, 0.94.2 Reporter: liwei Priority: Critical Attachments: ZookeeperNodeTracker.patch HBaseClient was running after a period of time, all of get operation became too slow. From the client logs I could see the following: 1. Unable to get data of znode /hbase/root-region-server java.lang.InterruptedException at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:485) at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1253) at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1129) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:264) at org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataInternal(ZKUtil.java:522) at org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataAndWatch(ZKUtil.java:498) at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.getData(ZooKeeperNodeTracker.java:156) at org.apache.hadoop.hbase.zookeeper.RootRegionTracker.getRootRegionLocation(RootRegionTracker.java:62) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:821) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:933) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:832) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801) at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:234) at org.apache.hadoop.hbase.client.HTable.init(HTable.java:174) at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:150) at org.apache.hadoop.hbase.client.MetaScanner.access$000(MetaScanner.java:48) at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:126) at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:123) at org.apache.hadoop.hbase.client.HConnectionManager.execute(HConnectionManager.java:359) at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:123) at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:99) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.prefetchRegionCache(HConnectionManager.java:894) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:948) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:836) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionLocation(HConnectionManager.java:725) at org.apache.hadoop.hbase.client.ServerCallable.connect(ServerCallable.java:82) at org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:162) at org.apache.hadoop.hbase.client.HTable.get(HTable.java:685) at org.apache.hadoop.hbase.client.HTablePool$PooledHTable.get(HTablePool.java:366) 2. Catalina.out found one Java-level deadlock: = catalina-exec-800: waiting to lock monitor 0x5f1f6530 (object 0x000731902200, a java.lang.Object), which is held by catalina-exec-710 catalina-exec-710: waiting to lock monitor 0x2aaab9a05bd0 (object 0x0007321f8708, a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation), which is held by catalina-exec-29-EventThread catalina-exec-29-EventThread: waiting to lock monitor 0x5f9f0af0 (object 0x000732a9c7e0, a org.apache.hadoop.hbase.zookeeper.RootRegionTracker), which is held by catalina-exec-710 Java stack information for the threads listed above:
[jira] [Commented] (HBASE-7259) Deadlock in HBaseClient when KeeperException occured
[ https://issues.apache.org/jira/browse/HBASE-7259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13508779#comment-13508779 ] Ted Yu commented on HBASE-7259: --- The latest patch was generated from 0.94 branch. Please check out trunk repo: http://svn.apache.org/repos/asf/hbase/trunk And attach patch based on trunk. Normally the patch filename contains the JIRA number (and optionally short description). Thanks Deadlock in HBaseClient when KeeperException occured Key: HBASE-7259 URL: https://issues.apache.org/jira/browse/HBASE-7259 Project: HBase Issue Type: Bug Components: Zookeeper Affects Versions: 0.94.0, 0.94.1, 0.94.2 Reporter: liwei Priority: Critical Attachments: HConnectionManager.patch HBaseClient was running after a period of time, all of get operation became too slow. From the client logs I could see the following: 1. Unable to get data of znode /hbase/root-region-server java.lang.InterruptedException at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:485) at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1253) at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1129) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:264) at org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataInternal(ZKUtil.java:522) at org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataAndWatch(ZKUtil.java:498) at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.getData(ZooKeeperNodeTracker.java:156) at org.apache.hadoop.hbase.zookeeper.RootRegionTracker.getRootRegionLocation(RootRegionTracker.java:62) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:821) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:933) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:832) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801) at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:234) at org.apache.hadoop.hbase.client.HTable.init(HTable.java:174) at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:150) at org.apache.hadoop.hbase.client.MetaScanner.access$000(MetaScanner.java:48) at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:126) at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:123) at org.apache.hadoop.hbase.client.HConnectionManager.execute(HConnectionManager.java:359) at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:123) at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:99) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.prefetchRegionCache(HConnectionManager.java:894) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:948) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:836) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionLocation(HConnectionManager.java:725) at org.apache.hadoop.hbase.client.ServerCallable.connect(ServerCallable.java:82) at org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:162) at org.apache.hadoop.hbase.client.HTable.get(HTable.java:685) at org.apache.hadoop.hbase.client.HTablePool$PooledHTable.get(HTablePool.java:366) 2. Catalina.out found one Java-level deadlock: = catalina-exec-800: waiting to lock monitor 0x5f1f6530 (object 0x000731902200, a java.lang.Object), which is held by catalina-exec-710 catalina-exec-710: waiting to lock monitor 0x2aaab9a05bd0 (object 0x0007321f8708, a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation), which is held by catalina-exec-29-EventThread catalina-exec-29-EventThread: waiting to lock monitor 0x5f9f0af0 (object 0x000732a9c7e0, a
[jira] [Commented] (HBASE-7259) Deadlock in HBaseClient when KeeperException occured
[ https://issues.apache.org/jira/browse/HBASE-7259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13508863#comment-13508863 ] liwei commented on HBASE-7259: -- I don't have privilege to commit. How to do it? Deadlock in HBaseClient when KeeperException occured Key: HBASE-7259 URL: https://issues.apache.org/jira/browse/HBASE-7259 Project: HBase Issue Type: Bug Components: Zookeeper Affects Versions: 0.94.0, 0.94.1, 0.94.2 Reporter: liwei Priority: Critical Attachments: HConnectionManager.patch HBaseClient was running after a period of time, all of get operation became too slow. From the client logs I could see the following: 1. Unable to get data of znode /hbase/root-region-server java.lang.InterruptedException at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:485) at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1253) at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1129) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:264) at org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataInternal(ZKUtil.java:522) at org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataAndWatch(ZKUtil.java:498) at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.getData(ZooKeeperNodeTracker.java:156) at org.apache.hadoop.hbase.zookeeper.RootRegionTracker.getRootRegionLocation(RootRegionTracker.java:62) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:821) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:933) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:832) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801) at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:234) at org.apache.hadoop.hbase.client.HTable.init(HTable.java:174) at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:150) at org.apache.hadoop.hbase.client.MetaScanner.access$000(MetaScanner.java:48) at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:126) at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:123) at org.apache.hadoop.hbase.client.HConnectionManager.execute(HConnectionManager.java:359) at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:123) at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:99) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.prefetchRegionCache(HConnectionManager.java:894) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:948) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:836) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionLocation(HConnectionManager.java:725) at org.apache.hadoop.hbase.client.ServerCallable.connect(ServerCallable.java:82) at org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:162) at org.apache.hadoop.hbase.client.HTable.get(HTable.java:685) at org.apache.hadoop.hbase.client.HTablePool$PooledHTable.get(HTablePool.java:366) 2. Catalina.out found one Java-level deadlock: = catalina-exec-800: waiting to lock monitor 0x5f1f6530 (object 0x000731902200, a java.lang.Object), which is held by catalina-exec-710 catalina-exec-710: waiting to lock monitor 0x2aaab9a05bd0 (object 0x0007321f8708, a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation), which is held by catalina-exec-29-EventThread catalina-exec-29-EventThread: waiting to lock monitor 0x5f9f0af0 (object 0x000732a9c7e0, a org.apache.hadoop.hbase.zookeeper.RootRegionTracker), which is held by catalina-exec-710 Java stack information for the threads listed above: ===
[jira] [Commented] (HBASE-7259) Deadlock in HBaseClient when KeeperException occured
[ https://issues.apache.org/jira/browse/HBASE-7259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13508871#comment-13508871 ] Ted Yu commented on HBASE-7259: --- I was talking about how to create the patch. 1. svn co http://svn.apache.org/repos/asf/hbase/trunk 2. cd trunk 3. modify HConnectionManager.java 4. at the root of trunk (your workspace) issue: svn diff 7259-remove-deadlock-in-case-keeper-exception.txt Deadlock in HBaseClient when KeeperException occured Key: HBASE-7259 URL: https://issues.apache.org/jira/browse/HBASE-7259 Project: HBase Issue Type: Bug Components: Zookeeper Affects Versions: 0.94.0, 0.94.1, 0.94.2 Reporter: liwei Priority: Critical Attachments: HConnectionManager.patch HBaseClient was running after a period of time, all of get operation became too slow. From the client logs I could see the following: 1. Unable to get data of znode /hbase/root-region-server java.lang.InterruptedException at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:485) at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1253) at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1129) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:264) at org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataInternal(ZKUtil.java:522) at org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataAndWatch(ZKUtil.java:498) at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.getData(ZooKeeperNodeTracker.java:156) at org.apache.hadoop.hbase.zookeeper.RootRegionTracker.getRootRegionLocation(RootRegionTracker.java:62) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:821) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:933) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:832) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801) at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:234) at org.apache.hadoop.hbase.client.HTable.init(HTable.java:174) at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:150) at org.apache.hadoop.hbase.client.MetaScanner.access$000(MetaScanner.java:48) at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:126) at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:123) at org.apache.hadoop.hbase.client.HConnectionManager.execute(HConnectionManager.java:359) at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:123) at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:99) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.prefetchRegionCache(HConnectionManager.java:894) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:948) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:836) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionLocation(HConnectionManager.java:725) at org.apache.hadoop.hbase.client.ServerCallable.connect(ServerCallable.java:82) at org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:162) at org.apache.hadoop.hbase.client.HTable.get(HTable.java:685) at org.apache.hadoop.hbase.client.HTablePool$PooledHTable.get(HTablePool.java:366) 2. Catalina.out found one Java-level deadlock: = catalina-exec-800: waiting to lock monitor 0x5f1f6530 (object 0x000731902200, a java.lang.Object), which is held by catalina-exec-710 catalina-exec-710: waiting to lock monitor 0x2aaab9a05bd0 (object 0x0007321f8708, a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation), which is held by catalina-exec-29-EventThread catalina-exec-29-EventThread: waiting to lock monitor 0x5f9f0af0 (object 0x000732a9c7e0, a
[jira] [Commented] (HBASE-7259) Deadlock in HBaseClient when KeeperException occured
[ https://issues.apache.org/jira/browse/HBASE-7259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13508873#comment-13508873 ] Lars Hofhansl commented on HBASE-7259: -- Things are much different in trunk as far as the client is concerned. A trunk patch for this neither realistic nor useful. Thanks for the analysis [~boneylw] Deadlock in HBaseClient when KeeperException occured Key: HBASE-7259 URL: https://issues.apache.org/jira/browse/HBASE-7259 Project: HBase Issue Type: Bug Components: Zookeeper Affects Versions: 0.94.0, 0.94.1, 0.94.2 Reporter: liwei Priority: Critical Attachments: HConnectionManager.patch HBaseClient was running after a period of time, all of get operation became too slow. From the client logs I could see the following: 1. Unable to get data of znode /hbase/root-region-server java.lang.InterruptedException at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:485) at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1253) at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1129) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:264) at org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataInternal(ZKUtil.java:522) at org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataAndWatch(ZKUtil.java:498) at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.getData(ZooKeeperNodeTracker.java:156) at org.apache.hadoop.hbase.zookeeper.RootRegionTracker.getRootRegionLocation(RootRegionTracker.java:62) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:821) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:933) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:832) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801) at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:234) at org.apache.hadoop.hbase.client.HTable.init(HTable.java:174) at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:150) at org.apache.hadoop.hbase.client.MetaScanner.access$000(MetaScanner.java:48) at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:126) at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:123) at org.apache.hadoop.hbase.client.HConnectionManager.execute(HConnectionManager.java:359) at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:123) at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:99) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.prefetchRegionCache(HConnectionManager.java:894) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:948) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:836) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionLocation(HConnectionManager.java:725) at org.apache.hadoop.hbase.client.ServerCallable.connect(ServerCallable.java:82) at org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:162) at org.apache.hadoop.hbase.client.HTable.get(HTable.java:685) at org.apache.hadoop.hbase.client.HTablePool$PooledHTable.get(HTablePool.java:366) 2. Catalina.out found one Java-level deadlock: = catalina-exec-800: waiting to lock monitor 0x5f1f6530 (object 0x000731902200, a java.lang.Object), which is held by catalina-exec-710 catalina-exec-710: waiting to lock monitor 0x2aaab9a05bd0 (object 0x0007321f8708, a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation), which is held by catalina-exec-29-EventThread catalina-exec-29-EventThread: waiting to lock monitor 0x5f9f0af0 (object 0x000732a9c7e0, a org.apache.hadoop.hbase.zookeeper.RootRegionTracker), which is held by
[jira] [Commented] (HBASE-7259) Deadlock in HBaseClient when KeeperException occured
[
https://issues.apache.org/jira/browse/HBASE-7259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13508884#comment-13508884
]
Lars Hofhansl commented on HBASE-7259:
--
Patch looks good.
Deadlock in HBaseClient when KeeperException occured
Key: HBASE-7259
URL: https://issues.apache.org/jira/browse/HBASE-7259
Project: HBase
Issue Type: Bug
Components: Zookeeper
Affects Versions: 0.94.0, 0.94.1, 0.94.2
Reporter: liwei
Priority: Critical
Attachments: HConnectionManager.patch
HBaseClient was running after a period of time, all of get operation became
too slow.
From the client logs I could see the following:
1. Unable to get data of znode /hbase/root-region-server
{code}
java.lang.InterruptedException
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1253)
at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1129)
at
org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:264)
at
org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataInternal(ZKUtil.java:522)
at
org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataAndWatch(ZKUtil.java:498)
at
org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.getData(ZooKeeperNodeTracker.java:156)
at
org.apache.hadoop.hbase.zookeeper.RootRegionTracker.getRootRegionLocation(RootRegionTracker.java:62)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:821)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:933)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:832)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:234)
at org.apache.hadoop.hbase.client.HTable.init(HTable.java:174)
at
org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:150)
at
org.apache.hadoop.hbase.client.MetaScanner.access$000(MetaScanner.java:48)
at
org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:126)
at
org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:123)
at
org.apache.hadoop.hbase.client.HConnectionManager.execute(HConnectionManager.java:359)
at
org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:123)
at
org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:99)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.prefetchRegionCache(HConnectionManager.java:894)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:948)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:836)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionLocation(HConnectionManager.java:725)
at
org.apache.hadoop.hbase.client.ServerCallable.connect(ServerCallable.java:82)
at
org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:162)
at org.apache.hadoop.hbase.client.HTable.get(HTable.java:685)
at
org.apache.hadoop.hbase.client.HTablePool$PooledHTable.get(HTablePool.java:366)
{code}
2. Catalina.out found one Java-level deadlock:
{code}
=
catalina-exec-800:
waiting to lock monitor 0x5f1f6530 (object 0x000731902200, a
java.lang.Object),
which is held by catalina-exec-710
catalina-exec-710:
waiting to lock monitor 0x2aaab9a05bd0 (object 0x0007321f8708, a
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation),
which is held by catalina-exec-29-EventThread
catalina-exec-29-EventThread:
waiting to lock monitor 0x5f9f0af0 (object 0x000732a9c7e0, a
org.apache.hadoop.hbase.zookeeper.RootRegionTracker),
which is held by catalina-exec-710
Java stack information for the threads listed above:
===
