[jira] [Updated] (HBASE-7259) Deadlock in HBaseClient when KeeperException occured

2012-12-04 Thread liwei (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HBASE-7259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

liwei updated HBASE-7259:
-

Attachment: 7259-0.94-branch.txt

 Deadlock in HBaseClient when KeeperException occured
 

 Key: HBASE-7259
 URL: https://issues.apache.org/jira/browse/HBASE-7259
 Project: HBase
  Issue Type: Bug
  Components: Zookeeper
Affects Versions: 0.94.0, 0.94.1, 0.94.2
Reporter: liwei
Priority: Critical
 Fix For: 0.94.4

 Attachments: 7259-0.94-branch.txt, HBASE-7259-0.94.2.txt


 HBaseClient was running after a period of time, all of get operation became 
 too slow.
 From the client logs I could see the following:
 1. Unable to get data of znode /hbase/root-region-server
 {code}
 java.lang.InterruptedException
 at java.lang.Object.wait(Native Method)
 at java.lang.Object.wait(Object.java:485)
 at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1253)
 at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1129)
 at 
 org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:264)
 at 
 org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataInternal(ZKUtil.java:522)
 at 
 org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataAndWatch(ZKUtil.java:498)
 at 
 org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.getData(ZooKeeperNodeTracker.java:156)
 at 
 org.apache.hadoop.hbase.zookeeper.RootRegionTracker.getRootRegionLocation(RootRegionTracker.java:62)
 at 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:821)
 at 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
 at 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:933)
 at 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:832)
 at 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
 at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:234)
 at org.apache.hadoop.hbase.client.HTable.init(HTable.java:174)
 at 
 org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:150)
 at 
 org.apache.hadoop.hbase.client.MetaScanner.access$000(MetaScanner.java:48)
 at 
 org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:126)
 at 
 org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:123)
 at 
 org.apache.hadoop.hbase.client.HConnectionManager.execute(HConnectionManager.java:359)
 at 
 org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:123)
 at 
 org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:99)
 at 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.prefetchRegionCache(HConnectionManager.java:894)
 at 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:948)
 at 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:836)
 at 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
 at 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionLocation(HConnectionManager.java:725)
 at 
 org.apache.hadoop.hbase.client.ServerCallable.connect(ServerCallable.java:82)
 at 
 org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:162)
 at org.apache.hadoop.hbase.client.HTable.get(HTable.java:685)
 at 
 org.apache.hadoop.hbase.client.HTablePool$PooledHTable.get(HTablePool.java:366)
 {code}
 2. Catalina.out found one Java-level deadlock:
 {code}
 =
 catalina-exec-800:
   waiting to lock monitor 0x5f1f6530 (object 0x000731902200, a 
 java.lang.Object),
   which is held by catalina-exec-710
 catalina-exec-710:
   waiting to lock monitor 0x2aaab9a05bd0 (object 0x0007321f8708, a 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation),
   which is held by catalina-exec-29-EventThread
 catalina-exec-29-EventThread:
   waiting to lock monitor 0x5f9f0af0 (object 0x000732a9c7e0, a 
 org.apache.hadoop.hbase.zookeeper.RootRegionTracker),
   which is held by catalina-exec-710
 Java stack information for the threads listed above:
 ===

[jira] [Updated] (HBASE-7259) Deadlock in HBaseClient when KeeperException occured

2012-12-03 Thread liwei (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HBASE-7259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

liwei updated HBASE-7259:
-

Attachment: (was: ZookeeperNodeTracker.patch)

 Deadlock in HBaseClient when KeeperException occured
 

 Key: HBASE-7259
 URL: https://issues.apache.org/jira/browse/HBASE-7259
 Project: HBase
  Issue Type: Bug
  Components: Zookeeper
Affects Versions: 0.94.0, 0.94.1, 0.94.2
Reporter: liwei
Priority: Critical

 HBaseClient was running after a period of time, all of get operation became 
 too slow.
 From the client logs I could see the following:
 1. Unable to get data of znode /hbase/root-region-server
 java.lang.InterruptedException
 at java.lang.Object.wait(Native Method)
 at java.lang.Object.wait(Object.java:485)
 at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1253)
 at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1129)
 at 
 org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:264)
 at 
 org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataInternal(ZKUtil.java:522)
 at 
 org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataAndWatch(ZKUtil.java:498)
 at 
 org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.getData(ZooKeeperNodeTracker.java:156)
 at 
 org.apache.hadoop.hbase.zookeeper.RootRegionTracker.getRootRegionLocation(RootRegionTracker.java:62)
 at 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:821)
 at 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
 at 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:933)
 at 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:832)
 at 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
 at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:234)
 at org.apache.hadoop.hbase.client.HTable.init(HTable.java:174)
 at 
 org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:150)
 at 
 org.apache.hadoop.hbase.client.MetaScanner.access$000(MetaScanner.java:48)
 at 
 org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:126)
 at 
 org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:123)
 at 
 org.apache.hadoop.hbase.client.HConnectionManager.execute(HConnectionManager.java:359)
 at 
 org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:123)
 at 
 org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:99)
 at 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.prefetchRegionCache(HConnectionManager.java:894)
 at 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:948)
 at 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:836)
 at 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
 at 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionLocation(HConnectionManager.java:725)
 at 
 org.apache.hadoop.hbase.client.ServerCallable.connect(ServerCallable.java:82)
 at 
 org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:162)
 at org.apache.hadoop.hbase.client.HTable.get(HTable.java:685)
 at 
 org.apache.hadoop.hbase.client.HTablePool$PooledHTable.get(HTablePool.java:366)
 2. Catalina.out found one Java-level deadlock:
 =
 catalina-exec-800:
   waiting to lock monitor 0x5f1f6530 (object 0x000731902200, a 
 java.lang.Object),
   which is held by catalina-exec-710
 catalina-exec-710:
   waiting to lock monitor 0x2aaab9a05bd0 (object 0x0007321f8708, a 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation),
   which is held by catalina-exec-29-EventThread
 catalina-exec-29-EventThread:
   waiting to lock monitor 0x5f9f0af0 (object 0x000732a9c7e0, a 
 org.apache.hadoop.hbase.zookeeper.RootRegionTracker),
   which is held by catalina-exec-710
 Java stack information for the threads listed above:
 ===
 catalina-exec-800:
 at

[jira] [Updated] (HBASE-7259) Deadlock in HBaseClient when KeeperException occured

2012-12-03 Thread liwei (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HBASE-7259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

liwei updated HBASE-7259:
-

Description: 
HBaseClient was running after a period of time, all of get operation became too 
slow.

From the client logs I could see the following:

1. Unable to get data of znode /hbase/root-region-server
java.lang.InterruptedException
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1253)
at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1129)
at 
org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:264)
at 
org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataInternal(ZKUtil.java:522)
at 
org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataAndWatch(ZKUtil.java:498)
at 
org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.getData(ZooKeeperNodeTracker.java:156)
at 
org.apache.hadoop.hbase.zookeeper.RootRegionTracker.getRootRegionLocation(RootRegionTracker.java:62)
at 
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:821)
at 
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
at 
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:933)
at 
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:832)
at 
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:234)
at org.apache.hadoop.hbase.client.HTable.init(HTable.java:174)
at 
org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:150)
at 
org.apache.hadoop.hbase.client.MetaScanner.access$000(MetaScanner.java:48)
at 
org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:126)
at 
org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:123)
at 
org.apache.hadoop.hbase.client.HConnectionManager.execute(HConnectionManager.java:359)
at 
org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:123)
at 
org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:99)
at 
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.prefetchRegionCache(HConnectionManager.java:894)
at 
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:948)
at 
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:836)
at 
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
at 
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionLocation(HConnectionManager.java:725)
at 
org.apache.hadoop.hbase.client.ServerCallable.connect(ServerCallable.java:82)
at 
org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:162)
at org.apache.hadoop.hbase.client.HTable.get(HTable.java:685)
at 
org.apache.hadoop.hbase.client.HTablePool$PooledHTable.get(HTablePool.java:366)

2. Catalina.out found one Java-level deadlock:

=

catalina-exec-800:
  waiting to lock monitor 0x5f1f6530 (object 0x000731902200, a 
java.lang.Object),
  which is held by catalina-exec-710
catalina-exec-710:
  waiting to lock monitor 0x2aaab9a05bd0 (object 0x0007321f8708, a 
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation),
  which is held by catalina-exec-29-EventThread
catalina-exec-29-EventThread:
  waiting to lock monitor 0x5f9f0af0 (object 0x000732a9c7e0, a 
org.apache.hadoop.hbase.zookeeper.RootRegionTracker),
  which is held by catalina-exec-710
Java stack information for the threads listed above:

===

catalina-exec-800:
at 
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:943)
- waiting to lock 0x000731902200 (a java.lang.Object)
at 
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:836)
at 
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.relocateRegion(HConnectionManager.java:807)
at 
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionLocation(HConnectionManager.java:725)
at

[jira] [Updated] (HBASE-7259) Deadlock in HBaseClient when KeeperException occured

2012-12-03 Thread liwei (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HBASE-7259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

liwei updated HBASE-7259:
-

Attachment: HConnectionManager.patch

 Deadlock in HBaseClient when KeeperException occured
 

 Key: HBASE-7259
 URL: https://issues.apache.org/jira/browse/HBASE-7259
 Project: HBase
  Issue Type: Bug
  Components: Zookeeper
Affects Versions: 0.94.0, 0.94.1, 0.94.2
Reporter: liwei
Priority: Critical
 Attachments: HConnectionManager.patch


 HBaseClient was running after a period of time, all of get operation became 
 too slow.
 From the client logs I could see the following:
 1. Unable to get data of znode /hbase/root-region-server
 java.lang.InterruptedException
 at java.lang.Object.wait(Native Method)
 at java.lang.Object.wait(Object.java:485)
 at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1253)
 at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1129)
 at 
 org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:264)
 at 
 org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataInternal(ZKUtil.java:522)
 at 
 org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataAndWatch(ZKUtil.java:498)
 at 
 org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.getData(ZooKeeperNodeTracker.java:156)
 at 
 org.apache.hadoop.hbase.zookeeper.RootRegionTracker.getRootRegionLocation(RootRegionTracker.java:62)
 at 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:821)
 at 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
 at 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:933)
 at 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:832)
 at 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
 at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:234)
 at org.apache.hadoop.hbase.client.HTable.init(HTable.java:174)
 at 
 org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:150)
 at 
 org.apache.hadoop.hbase.client.MetaScanner.access$000(MetaScanner.java:48)
 at 
 org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:126)
 at 
 org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:123)
 at 
 org.apache.hadoop.hbase.client.HConnectionManager.execute(HConnectionManager.java:359)
 at 
 org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:123)
 at 
 org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:99)
 at 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.prefetchRegionCache(HConnectionManager.java:894)
 at 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:948)
 at 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:836)
 at 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
 at 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionLocation(HConnectionManager.java:725)
 at 
 org.apache.hadoop.hbase.client.ServerCallable.connect(ServerCallable.java:82)
 at 
 org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:162)
 at org.apache.hadoop.hbase.client.HTable.get(HTable.java:685)
 at 
 org.apache.hadoop.hbase.client.HTablePool$PooledHTable.get(HTablePool.java:366)
 2. Catalina.out found one Java-level deadlock:
 =
 catalina-exec-800:
   waiting to lock monitor 0x5f1f6530 (object 0x000731902200, a 
 java.lang.Object),
   which is held by catalina-exec-710
 catalina-exec-710:
   waiting to lock monitor 0x2aaab9a05bd0 (object 0x0007321f8708, a 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation),
   which is held by catalina-exec-29-EventThread
 catalina-exec-29-EventThread:
   waiting to lock monitor 0x5f9f0af0 (object 0x000732a9c7e0, a 
 org.apache.hadoop.hbase.zookeeper.RootRegionTracker),
   which is held by catalina-exec-710
 Java stack information for the threads listed above:
 ===
 catalina-exec-800:
 at

[jira] [Updated] (HBASE-7259) Deadlock in HBaseClient when KeeperException occured

2012-12-03 Thread liwei (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HBASE-7259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

liwei updated HBASE-7259:
-

Attachment: (was: HConnectionManager.patch)

 Deadlock in HBaseClient when KeeperException occured
 

 Key: HBASE-7259
 URL: https://issues.apache.org/jira/browse/HBASE-7259
 Project: HBase
  Issue Type: Bug
  Components: Zookeeper
Affects Versions: 0.94.0, 0.94.1, 0.94.2
Reporter: liwei
Priority: Critical

 HBaseClient was running after a period of time, all of get operation became 
 too slow.
 From the client logs I could see the following:
 1. Unable to get data of znode /hbase/root-region-server
 java.lang.InterruptedException
 at java.lang.Object.wait(Native Method)
 at java.lang.Object.wait(Object.java:485)
 at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1253)
 at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1129)
 at 
 org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:264)
 at 
 org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataInternal(ZKUtil.java:522)
 at 
 org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataAndWatch(ZKUtil.java:498)
 at 
 org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.getData(ZooKeeperNodeTracker.java:156)
 at 
 org.apache.hadoop.hbase.zookeeper.RootRegionTracker.getRootRegionLocation(RootRegionTracker.java:62)
 at 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:821)
 at 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
 at 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:933)
 at 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:832)
 at 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
 at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:234)
 at org.apache.hadoop.hbase.client.HTable.init(HTable.java:174)
 at 
 org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:150)
 at 
 org.apache.hadoop.hbase.client.MetaScanner.access$000(MetaScanner.java:48)
 at 
 org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:126)
 at 
 org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:123)
 at 
 org.apache.hadoop.hbase.client.HConnectionManager.execute(HConnectionManager.java:359)
 at 
 org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:123)
 at 
 org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:99)
 at 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.prefetchRegionCache(HConnectionManager.java:894)
 at 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:948)
 at 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:836)
 at 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
 at 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionLocation(HConnectionManager.java:725)
 at 
 org.apache.hadoop.hbase.client.ServerCallable.connect(ServerCallable.java:82)
 at 
 org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:162)
 at org.apache.hadoop.hbase.client.HTable.get(HTable.java:685)
 at 
 org.apache.hadoop.hbase.client.HTablePool$PooledHTable.get(HTablePool.java:366)
 2. Catalina.out found one Java-level deadlock:
 =
 catalina-exec-800:
   waiting to lock monitor 0x5f1f6530 (object 0x000731902200, a 
 java.lang.Object),
   which is held by catalina-exec-710
 catalina-exec-710:
   waiting to lock monitor 0x2aaab9a05bd0 (object 0x0007321f8708, a 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation),
   which is held by catalina-exec-29-EventThread
 catalina-exec-29-EventThread:
   waiting to lock monitor 0x5f9f0af0 (object 0x000732a9c7e0, a 
 org.apache.hadoop.hbase.zookeeper.RootRegionTracker),
   which is held by catalina-exec-710
 Java stack information for the threads listed above:
 ===
 catalina-exec-800:
 at

[jira] [Updated] (HBASE-7259) Deadlock in HBaseClient when KeeperException occured

2012-12-03 Thread liwei (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HBASE-7259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

liwei updated HBASE-7259:
-

Attachment: HConnectionManager.patch

 Deadlock in HBaseClient when KeeperException occured
 

 Key: HBASE-7259
 URL: https://issues.apache.org/jira/browse/HBASE-7259
 Project: HBase
  Issue Type: Bug
  Components: Zookeeper
Affects Versions: 0.94.0, 0.94.1, 0.94.2
Reporter: liwei
Priority: Critical
 Attachments: HConnectionManager.patch


 HBaseClient was running after a period of time, all of get operation became 
 too slow.
 From the client logs I could see the following:
 1. Unable to get data of znode /hbase/root-region-server
 java.lang.InterruptedException
 at java.lang.Object.wait(Native Method)
 at java.lang.Object.wait(Object.java:485)
 at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1253)
 at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1129)
 at 
 org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:264)
 at 
 org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataInternal(ZKUtil.java:522)
 at 
 org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataAndWatch(ZKUtil.java:498)
 at 
 org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.getData(ZooKeeperNodeTracker.java:156)
 at 
 org.apache.hadoop.hbase.zookeeper.RootRegionTracker.getRootRegionLocation(RootRegionTracker.java:62)
 at 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:821)
 at 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
 at 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:933)
 at 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:832)
 at 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
 at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:234)
 at org.apache.hadoop.hbase.client.HTable.init(HTable.java:174)
 at 
 org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:150)
 at 
 org.apache.hadoop.hbase.client.MetaScanner.access$000(MetaScanner.java:48)
 at 
 org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:126)
 at 
 org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:123)
 at 
 org.apache.hadoop.hbase.client.HConnectionManager.execute(HConnectionManager.java:359)
 at 
 org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:123)
 at 
 org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:99)
 at 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.prefetchRegionCache(HConnectionManager.java:894)
 at 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:948)
 at 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:836)
 at 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
 at 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionLocation(HConnectionManager.java:725)
 at 
 org.apache.hadoop.hbase.client.ServerCallable.connect(ServerCallable.java:82)
 at 
 org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:162)
 at org.apache.hadoop.hbase.client.HTable.get(HTable.java:685)
 at 
 org.apache.hadoop.hbase.client.HTablePool$PooledHTable.get(HTablePool.java:366)
 2. Catalina.out found one Java-level deadlock:
 =
 catalina-exec-800:
   waiting to lock monitor 0x5f1f6530 (object 0x000731902200, a 
 java.lang.Object),
   which is held by catalina-exec-710
 catalina-exec-710:
   waiting to lock monitor 0x2aaab9a05bd0 (object 0x0007321f8708, a 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation),
   which is held by catalina-exec-29-EventThread
 catalina-exec-29-EventThread:
   waiting to lock monitor 0x5f9f0af0 (object 0x000732a9c7e0, a 
 org.apache.hadoop.hbase.zookeeper.RootRegionTracker),
   which is held by catalina-exec-710
 Java stack information for the threads listed above:
 ===
 catalina-exec-800:
 at

[jira] [Commented] (HBASE-7259) Deadlock in HBaseClient when KeeperException occured

2012-12-03 Thread liwei (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-7259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13508863#comment-13508863
 ] 

liwei commented on HBASE-7259:
--

I don't have privilege to commit.
How to do it? 

 Deadlock in HBaseClient when KeeperException occured
 

 Key: HBASE-7259
 URL: https://issues.apache.org/jira/browse/HBASE-7259
 Project: HBase
  Issue Type: Bug
  Components: Zookeeper
Affects Versions: 0.94.0, 0.94.1, 0.94.2
Reporter: liwei
Priority: Critical
 Attachments: HConnectionManager.patch


 HBaseClient was running after a period of time, all of get operation became 
 too slow.
 From the client logs I could see the following:
 1. Unable to get data of znode /hbase/root-region-server
 java.lang.InterruptedException
 at java.lang.Object.wait(Native Method)
 at java.lang.Object.wait(Object.java:485)
 at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1253)
 at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1129)
 at 
 org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:264)
 at 
 org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataInternal(ZKUtil.java:522)
 at 
 org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataAndWatch(ZKUtil.java:498)
 at 
 org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.getData(ZooKeeperNodeTracker.java:156)
 at 
 org.apache.hadoop.hbase.zookeeper.RootRegionTracker.getRootRegionLocation(RootRegionTracker.java:62)
 at 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:821)
 at 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
 at 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:933)
 at 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:832)
 at 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
 at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:234)
 at org.apache.hadoop.hbase.client.HTable.init(HTable.java:174)
 at 
 org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:150)
 at 
 org.apache.hadoop.hbase.client.MetaScanner.access$000(MetaScanner.java:48)
 at 
 org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:126)
 at 
 org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:123)
 at 
 org.apache.hadoop.hbase.client.HConnectionManager.execute(HConnectionManager.java:359)
 at 
 org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:123)
 at 
 org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:99)
 at 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.prefetchRegionCache(HConnectionManager.java:894)
 at 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:948)
 at 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:836)
 at 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
 at 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionLocation(HConnectionManager.java:725)
 at 
 org.apache.hadoop.hbase.client.ServerCallable.connect(ServerCallable.java:82)
 at 
 org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:162)
 at org.apache.hadoop.hbase.client.HTable.get(HTable.java:685)
 at 
 org.apache.hadoop.hbase.client.HTablePool$PooledHTable.get(HTablePool.java:366)
 2. Catalina.out found one Java-level deadlock:
 =
 catalina-exec-800:
   waiting to lock monitor 0x5f1f6530 (object 0x000731902200, a 
 java.lang.Object),
   which is held by catalina-exec-710
 catalina-exec-710:
   waiting to lock monitor 0x2aaab9a05bd0 (object 0x0007321f8708, a 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation),
   which is held by catalina-exec-29-EventThread
 catalina-exec-29-EventThread:
   waiting to lock monitor 0x5f9f0af0 (object 0x000732a9c7e0, a 
 org.apache.hadoop.hbase.zookeeper.RootRegionTracker),
   which is held by catalina-exec-710
 Java stack information for the threads listed above:
 ===

[jira] [Updated] (HBASE-7259) Deadlock in HBaseClient when KeeperException occured

2012-12-03 Thread liwei (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HBASE-7259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

liwei updated HBASE-7259:
-

Attachment: HBASE-7259-0.94.2.txt

from 0.94.2

 Deadlock in HBaseClient when KeeperException occured
 

 Key: HBASE-7259
 URL: https://issues.apache.org/jira/browse/HBASE-7259
 Project: HBase
  Issue Type: Bug
  Components: Zookeeper
Affects Versions: 0.94.0, 0.94.1, 0.94.2
Reporter: liwei
Priority: Critical
 Fix For: 0.94.4

 Attachments: HBASE-7259-0.94.2.txt, HConnectionManager.patch


 HBaseClient was running after a period of time, all of get operation became 
 too slow.
 From the client logs I could see the following:
 1. Unable to get data of znode /hbase/root-region-server
 {code}
 java.lang.InterruptedException
 at java.lang.Object.wait(Native Method)
 at java.lang.Object.wait(Object.java:485)
 at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1253)
 at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1129)
 at 
 org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:264)
 at 
 org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataInternal(ZKUtil.java:522)
 at 
 org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataAndWatch(ZKUtil.java:498)
 at 
 org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.getData(ZooKeeperNodeTracker.java:156)
 at 
 org.apache.hadoop.hbase.zookeeper.RootRegionTracker.getRootRegionLocation(RootRegionTracker.java:62)
 at 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:821)
 at 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
 at 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:933)
 at 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:832)
 at 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
 at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:234)
 at org.apache.hadoop.hbase.client.HTable.init(HTable.java:174)
 at 
 org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:150)
 at 
 org.apache.hadoop.hbase.client.MetaScanner.access$000(MetaScanner.java:48)
 at 
 org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:126)
 at 
 org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:123)
 at 
 org.apache.hadoop.hbase.client.HConnectionManager.execute(HConnectionManager.java:359)
 at 
 org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:123)
 at 
 org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:99)
 at 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.prefetchRegionCache(HConnectionManager.java:894)
 at 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:948)
 at 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:836)
 at 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
 at 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionLocation(HConnectionManager.java:725)
 at 
 org.apache.hadoop.hbase.client.ServerCallable.connect(ServerCallable.java:82)
 at 
 org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:162)
 at org.apache.hadoop.hbase.client.HTable.get(HTable.java:685)
 at 
 org.apache.hadoop.hbase.client.HTablePool$PooledHTable.get(HTablePool.java:366)
 {code}
 2. Catalina.out found one Java-level deadlock:
 {code}
 =
 catalina-exec-800:
   waiting to lock monitor 0x5f1f6530 (object 0x000731902200, a 
 java.lang.Object),
   which is held by catalina-exec-710
 catalina-exec-710:
   waiting to lock monitor 0x2aaab9a05bd0 (object 0x0007321f8708, a 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation),
   which is held by catalina-exec-29-EventThread
 catalina-exec-29-EventThread:
   waiting to lock monitor 0x5f9f0af0 (object 0x000732a9c7e0, a 
 org.apache.hadoop.hbase.zookeeper.RootRegionTracker),
   which is held by catalina-exec-710
 Java stack information for the threads listed above:

[jira] [Updated] (HBASE-7259) Deadlock in HBaseClient when KeeperException occured

2012-12-03 Thread liwei (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HBASE-7259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

liwei updated HBASE-7259:
-

Attachment: (was: HConnectionManager.patch)

 Deadlock in HBaseClient when KeeperException occured
 

 Key: HBASE-7259
 URL: https://issues.apache.org/jira/browse/HBASE-7259
 Project: HBase
  Issue Type: Bug
  Components: Zookeeper
Affects Versions: 0.94.0, 0.94.1, 0.94.2
Reporter: liwei
Priority: Critical
 Fix For: 0.94.4

 Attachments: HBASE-7259-0.94.2.txt


 HBaseClient was running after a period of time, all of get operation became 
 too slow.
 From the client logs I could see the following:
 1. Unable to get data of znode /hbase/root-region-server
 {code}
 java.lang.InterruptedException
 at java.lang.Object.wait(Native Method)
 at java.lang.Object.wait(Object.java:485)
 at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1253)
 at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1129)
 at 
 org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:264)
 at 
 org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataInternal(ZKUtil.java:522)
 at 
 org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataAndWatch(ZKUtil.java:498)
 at 
 org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.getData(ZooKeeperNodeTracker.java:156)
 at 
 org.apache.hadoop.hbase.zookeeper.RootRegionTracker.getRootRegionLocation(RootRegionTracker.java:62)
 at 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:821)
 at 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
 at 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:933)
 at 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:832)
 at 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
 at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:234)
 at org.apache.hadoop.hbase.client.HTable.init(HTable.java:174)
 at 
 org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:150)
 at 
 org.apache.hadoop.hbase.client.MetaScanner.access$000(MetaScanner.java:48)
 at 
 org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:126)
 at 
 org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:123)
 at 
 org.apache.hadoop.hbase.client.HConnectionManager.execute(HConnectionManager.java:359)
 at 
 org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:123)
 at 
 org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:99)
 at 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.prefetchRegionCache(HConnectionManager.java:894)
 at 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:948)
 at 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:836)
 at 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
 at 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionLocation(HConnectionManager.java:725)
 at 
 org.apache.hadoop.hbase.client.ServerCallable.connect(ServerCallable.java:82)
 at 
 org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:162)
 at org.apache.hadoop.hbase.client.HTable.get(HTable.java:685)
 at 
 org.apache.hadoop.hbase.client.HTablePool$PooledHTable.get(HTablePool.java:366)
 {code}
 2. Catalina.out found one Java-level deadlock:
 {code}
 =
 catalina-exec-800:
   waiting to lock monitor 0x5f1f6530 (object 0x000731902200, a 
 java.lang.Object),
   which is held by catalina-exec-710
 catalina-exec-710:
   waiting to lock monitor 0x2aaab9a05bd0 (object 0x0007321f8708, a 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation),
   which is held by catalina-exec-29-EventThread
 catalina-exec-29-EventThread:
   waiting to lock monitor 0x5f9f0af0 (object 0x000732a9c7e0, a 
 org.apache.hadoop.hbase.zookeeper.RootRegionTracker),
   which is held by catalina-exec-710
 Java stack information for the threads listed above:
 ===
 catalina-exec-800:

[jira] [Created] (HBASE-7259) Deadlock in HBaseClient when KeeperException occured

2012-12-02 Thread liwei (JIRA) 
liwei created HBASE-7259:


 Summary: Deadlock in HBaseClient when KeeperException occured
 Key: HBASE-7259
 URL: https://issues.apache.org/jira/browse/HBASE-7259
 Project: HBase
  Issue Type: Bug
  Components: Zookeeper
Affects Versions: 0.94.2, 0.94.1, 0.94.0
Reporter: liwei
Priority: Critical


HBaseClient was running after a period of time, all of get operation became too 
slow.

From the client logs I could see the following:

1. Unable to get data of znode /hbase/root-region-server
java.lang.InterruptedException
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1253)
at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1129)
at 
org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:264)
at 
org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataInternal(ZKUtil.java:522)
at 
org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataAndWatch(ZKUtil.java:498)
at 
org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.getData(ZooKeeperNodeTracker.java:156)
at 
org.apache.hadoop.hbase.zookeeper.RootRegionTracker.getRootRegionLocation(RootRegionTracker.java:62)
at 
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:821)
at 
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
at 
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:933)
at 
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:832)
at 
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:234)
at org.apache.hadoop.hbase.client.HTable.init(HTable.java:174)
at 
org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:150)
at 
org.apache.hadoop.hbase.client.MetaScanner.access$000(MetaScanner.java:48)
at 
org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:126)
at 
org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:123)
at 
org.apache.hadoop.hbase.client.HConnectionManager.execute(HConnectionManager.java:359)
at 
org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:123)
at 
org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:99)
at 
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.prefetchRegionCache(HConnectionManager.java:894)
at 
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:948)
at 
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:836)
at 
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
at 
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionLocation(HConnectionManager.java:725)
at 
org.apache.hadoop.hbase.client.ServerCallable.connect(ServerCallable.java:82)
at 
org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:162)
at org.apache.hadoop.hbase.client.HTable.get(HTable.java:685)
at 
org.apache.hadoop.hbase.client.HTablePool$PooledHTable.get(HTablePool.java:366)

2. jstack traces found one Java-level deadlock:

=

catalina-exec-800:
  waiting to lock monitor 0x5f1f6530 (object 0x000731902200, a 
java.lang.Object),
  which is held by catalina-exec-710
catalina-exec-710:
  waiting to lock monitor 0x2aaab9a05bd0 (object 0x0007321f8708, a 
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation),
  which is held by catalina-exec-29-EventThread
catalina-exec-29-EventThread:
  waiting to lock monitor 0x5f9f0af0 (object 0x000732a9c7e0, a 
org.apache.hadoop.hbase.zookeeper.RootRegionTracker),
  which is held by catalina-exec-710
Java stack information for the threads listed above:

===

catalina-exec-800:
at 
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:943)
- waiting to lock 0x000731902200 (a java.lang.Object)
at 
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:836)
at

[jira] [Updated] (HBASE-7259) Deadlock in HBaseClient when KeeperException occured

2012-12-02 Thread liwei (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HBASE-7259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

liwei updated HBASE-7259:
-

Attachment: ZookeeperNodeTracker.patch

 Deadlock in HBaseClient when KeeperException occured
 

 Key: HBASE-7259
 URL: https://issues.apache.org/jira/browse/HBASE-7259
 Project: HBase
  Issue Type: Bug
  Components: Zookeeper
Affects Versions: 0.94.0, 0.94.1, 0.94.2
Reporter: liwei
Priority: Critical
 Attachments: ZookeeperNodeTracker.patch


 HBaseClient was running after a period of time, all of get operation became 
 too slow.
 From the client logs I could see the following:
 1. Unable to get data of znode /hbase/root-region-server
 java.lang.InterruptedException
 at java.lang.Object.wait(Native Method)
 at java.lang.Object.wait(Object.java:485)
 at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1253)
 at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1129)
 at 
 org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:264)
 at 
 org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataInternal(ZKUtil.java:522)
 at 
 org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataAndWatch(ZKUtil.java:498)
 at 
 org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.getData(ZooKeeperNodeTracker.java:156)
 at 
 org.apache.hadoop.hbase.zookeeper.RootRegionTracker.getRootRegionLocation(RootRegionTracker.java:62)
 at 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:821)
 at 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
 at 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:933)
 at 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:832)
 at 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
 at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:234)
 at org.apache.hadoop.hbase.client.HTable.init(HTable.java:174)
 at 
 org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:150)
 at 
 org.apache.hadoop.hbase.client.MetaScanner.access$000(MetaScanner.java:48)
 at 
 org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:126)
 at 
 org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:123)
 at 
 org.apache.hadoop.hbase.client.HConnectionManager.execute(HConnectionManager.java:359)
 at 
 org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:123)
 at 
 org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:99)
 at 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.prefetchRegionCache(HConnectionManager.java:894)
 at 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:948)
 at 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:836)
 at 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
 at 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionLocation(HConnectionManager.java:725)
 at 
 org.apache.hadoop.hbase.client.ServerCallable.connect(ServerCallable.java:82)
 at 
 org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:162)
 at org.apache.hadoop.hbase.client.HTable.get(HTable.java:685)
 at 
 org.apache.hadoop.hbase.client.HTablePool$PooledHTable.get(HTablePool.java:366)
 2. jstack traces found one Java-level deadlock:
 =
 catalina-exec-800:
   waiting to lock monitor 0x5f1f6530 (object 0x000731902200, a 
 java.lang.Object),
   which is held by catalina-exec-710
 catalina-exec-710:
   waiting to lock monitor 0x2aaab9a05bd0 (object 0x0007321f8708, a 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation),
   which is held by catalina-exec-29-EventThread
 catalina-exec-29-EventThread:
   waiting to lock monitor 0x5f9f0af0 (object 0x000732a9c7e0, a 
 org.apache.hadoop.hbase.zookeeper.RootRegionTracker),
   which is held by catalina-exec-710
 Java stack information for the threads listed above:
 ===
 catalina-exec-800:
 at

[jira] [Updated] (HBASE-7259) Deadlock in HBaseClient when KeeperException occured

2012-12-02 Thread liwei (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HBASE-7259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

liwei updated HBASE-7259:
-

Description: 
HBaseClient was running after a period of time, all of get operation became too 
slow.

From the client logs I could see the following:

1. Unable to get data of znode /hbase/root-region-server
java.lang.InterruptedException
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1253)
at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1129)
at 
org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:264)
at 
org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataInternal(ZKUtil.java:522)
at 
org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataAndWatch(ZKUtil.java:498)
at 
org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.getData(ZooKeeperNodeTracker.java:156)
at 
org.apache.hadoop.hbase.zookeeper.RootRegionTracker.getRootRegionLocation(RootRegionTracker.java:62)
at 
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:821)
at 
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
at 
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:933)
at 
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:832)
at 
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:234)
at org.apache.hadoop.hbase.client.HTable.init(HTable.java:174)
at 
org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:150)
at 
org.apache.hadoop.hbase.client.MetaScanner.access$000(MetaScanner.java:48)
at 
org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:126)
at 
org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:123)
at 
org.apache.hadoop.hbase.client.HConnectionManager.execute(HConnectionManager.java:359)
at 
org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:123)
at 
org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:99)
at 
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.prefetchRegionCache(HConnectionManager.java:894)
at 
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:948)
at 
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:836)
at 
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
at 
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionLocation(HConnectionManager.java:725)
at 
org.apache.hadoop.hbase.client.ServerCallable.connect(ServerCallable.java:82)
at 
org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:162)
at org.apache.hadoop.hbase.client.HTable.get(HTable.java:685)
at 
org.apache.hadoop.hbase.client.HTablePool$PooledHTable.get(HTablePool.java:366)

2. jstack traces found one Java-level deadlock:

=

catalina-exec-800:
  waiting to lock monitor 0x5f1f6530 (object 0x000731902200, a 
java.lang.Object),
  which is held by catalina-exec-710
catalina-exec-710:
  waiting to lock monitor 0x2aaab9a05bd0 (object 0x0007321f8708, a 
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation),
  which is held by catalina-exec-29-EventThread
catalina-exec-29-EventThread:
  waiting to lock monitor 0x5f9f0af0 (object 0x000732a9c7e0, a 
org.apache.hadoop.hbase.zookeeper.RootRegionTracker),
  which is held by catalina-exec-710
Java stack information for the threads listed above:

===

catalina-exec-800:
at 
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:943)
- waiting to lock 0x000731902200 (a java.lang.Object)
at 
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:836)
at 
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.relocateRegion(HConnectionManager.java:807)
at 
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionLocation(HConnectionManager.java:725)
at

[jira] [Updated] (HBASE-7259) Deadlock in HBaseClient when KeeperException occured

2012-12-02 Thread liwei (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HBASE-7259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

liwei updated HBASE-7259:
-

Description: 
HBaseClient was running after a period of time, all of get operation became too 
slow.

From the client logs I could see the following:

1. Unable to get data of znode /hbase/root-region-server
java.lang.InterruptedException
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1253)
at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1129)
at 
org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:264)
at 
org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataInternal(ZKUtil.java:522)
at 
org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataAndWatch(ZKUtil.java:498)
at 
org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.getData(ZooKeeperNodeTracker.java:156)
at 
org.apache.hadoop.hbase.zookeeper.RootRegionTracker.getRootRegionLocation(RootRegionTracker.java:62)
at 
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:821)
at 
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
at 
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:933)
at 
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:832)
at 
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:234)
at org.apache.hadoop.hbase.client.HTable.init(HTable.java:174)
at 
org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:150)
at 
org.apache.hadoop.hbase.client.MetaScanner.access$000(MetaScanner.java:48)
at 
org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:126)
at 
org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:123)
at 
org.apache.hadoop.hbase.client.HConnectionManager.execute(HConnectionManager.java:359)
at 
org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:123)
at 
org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:99)
at 
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.prefetchRegionCache(HConnectionManager.java:894)
at 
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:948)
at 
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:836)
at 
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
at 
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionLocation(HConnectionManager.java:725)
at 
org.apache.hadoop.hbase.client.ServerCallable.connect(ServerCallable.java:82)
at 
org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:162)
at org.apache.hadoop.hbase.client.HTable.get(HTable.java:685)
at 
org.apache.hadoop.hbase.client.HTablePool$PooledHTable.get(HTablePool.java:366)

2. Catalina.out found one Java-level deadlock:

=

catalina-exec-800:
  waiting to lock monitor 0x5f1f6530 (object 0x000731902200, a 
java.lang.Object),
  which is held by catalina-exec-710
catalina-exec-710:
  waiting to lock monitor 0x2aaab9a05bd0 (object 0x0007321f8708, a 
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation),
  which is held by catalina-exec-29-EventThread
catalina-exec-29-EventThread:
  waiting to lock monitor 0x5f9f0af0 (object 0x000732a9c7e0, a 
org.apache.hadoop.hbase.zookeeper.RootRegionTracker),
  which is held by catalina-exec-710
Java stack information for the threads listed above:

===

catalina-exec-800:
at 
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:943)
- waiting to lock 0x000731902200 (a java.lang.Object)
at 
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:836)
at 
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.relocateRegion(HConnectionManager.java:807)
at 
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionLocation(HConnectionManager.java:725)
at

[jira] [Updated] (HBASE-7259) Deadlock in HBaseClient when KeeperException occured

2012-12-02 Thread liwei (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HBASE-7259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

liwei updated HBASE-7259:
-

Description: 
HBaseClient was running after a period of time, all of get operation became too 
slow.

From the client logs I could see the following:

1. Unable to get data of znode /hbase/root-region-server
java.lang.InterruptedException
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1253)
at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1129)
at 
org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:264)
at 
org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataInternal(ZKUtil.java:522)
at 
org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataAndWatch(ZKUtil.java:498)
at 
org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.getData(ZooKeeperNodeTracker.java:156)
at 
org.apache.hadoop.hbase.zookeeper.RootRegionTracker.getRootRegionLocation(RootRegionTracker.java:62)
at 
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:821)
at 
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
at 
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:933)
at 
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:832)
at 
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:234)
at org.apache.hadoop.hbase.client.HTable.init(HTable.java:174)
at 
org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:150)
at 
org.apache.hadoop.hbase.client.MetaScanner.access$000(MetaScanner.java:48)
at 
org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:126)
at 
org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:123)
at 
org.apache.hadoop.hbase.client.HConnectionManager.execute(HConnectionManager.java:359)
at 
org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:123)
at 
org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:99)
at 
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.prefetchRegionCache(HConnectionManager.java:894)
at 
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:948)
at 
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:836)
at 
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
at 
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionLocation(HConnectionManager.java:725)
at 
org.apache.hadoop.hbase.client.ServerCallable.connect(ServerCallable.java:82)
at 
org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:162)
at org.apache.hadoop.hbase.client.HTable.get(HTable.java:685)
at 
org.apache.hadoop.hbase.client.HTablePool$PooledHTable.get(HTablePool.java:366)

2. Catalina.out found one Java-level deadlock:

=

catalina-exec-800:
  waiting to lock monitor 0x5f1f6530 (object 0x000731902200, a 
java.lang.Object),
  which is held by catalina-exec-710
catalina-exec-710:
  waiting to lock monitor 0x2aaab9a05bd0 (object 0x0007321f8708, a 
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation),
  which is held by catalina-exec-29-EventThread
catalina-exec-29-EventThread:
  waiting to lock monitor 0x5f9f0af0 (object 0x000732a9c7e0, a 
org.apache.hadoop.hbase.zookeeper.RootRegionTracker),
  which is held by catalina-exec-710
Java stack information for the threads listed above:

===

catalina-exec-800:
at 
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:943)
- waiting to lock 0x000731902200 (a java.lang.Object)
at 
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:836)
at 
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.relocateRegion(HConnectionManager.java:807)
at 
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionLocation(HConnectionManager.java:725)
at