stack created HBASE-9187:
----------------------------
Summary: HBaseAdmin#testTableExists can go zombie
Key: HBASE-9187
URL: https://issues.apache.org/jira/browse/HBASE-9187
Project: HBase
Issue Type: Sub-task
Reporter: stack
See it here as a zombie in hadoopqa:
https://builds.apache.org/job/PreCommit-HBASE-Build/6687/consoleText
Looking at it, we seem stuck in here:
{code}
"RpcServer.handler=1,port=51776" daemon prio=10 tid=0x72001400 nid=0x17ea
waiting on condition [0x71cd4000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:150)
- locked <0x81042070> (a
org.apache.hadoop.hbase.client.RpcRetryingCaller)
at org.apache.hadoop.hbase.client.HTable.get(HTable.java:732)
at
org.apache.hadoop.hbase.master.TableNamespaceManager.get(TableNamespaceManager.java:111)
- locked <0x7f71ba70> (a
org.apache.hadoop.hbase.master.TableNamespaceManager)
at
org.apache.hadoop.hbase.master.HMaster.getNamespaceDescriptor(HMaster.java:3076)
at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:1779)
at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:1820)
at
org.apache.hadoop.hbase.protobuf.generated.MasterAdminProtos$MasterAdminService$2.callBlockingMethod(MasterAdminProtos.java:26698)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2068)
at
org.apache.hadoop.hbase.ipc.RpcServer$CallRunner.run(RpcServer.java:1807)
at
org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:165)
at
org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$000(SimpleRpcScheduler.java:41)
at
org.apache.hadoop.hbase.ipc.SimpleRpcScheduler$1.run(SimpleRpcScheduler.java:113)
at java.lang.Thread.run(Thread.java:662)
{code}
This lock is held: 0x7f71ba70 We are doing retries against the new ns table.
A bunch other threads are trying to get in here while we are retrying:
{code}
"RpcServer.handler=0,port=51776" daemon prio=10 tid=0x72000400 nid=0x17e9
waiting for monitor entry [0x71d25000]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.hadoop.hbase.master.TableNamespaceManager.get(TableNamespaceManager.java:111)
- waiting to lock <0x7f71ba70> (a
org.apache.hadoop.hbase.master.TableNamespaceManager)
at
org.apache.hadoop.hbase.master.HMaster.getNamespaceDescriptor(HMaster.java:3076)
at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:1779)
at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:1820)
at
org.apache.hadoop.hbase.protobuf.generated.MasterAdminProtos$MasterAdminService$2.callBlockingMethod(MasterAdminProtos.java:26698)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2068)
at
org.apache.hadoop.hbase.ipc.RpcServer$CallRunner.run(RpcServer.java:1807)
at
org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:165)
at
org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$000(SimpleRpcScheduler.java:41)
at
org.apache.hadoop.hbase.ipc.SimpleRpcScheduler$1.run(SimpleRpcScheduler.java:113)
at java.lang.Thread.run(Thread.java:662)
...
"RpcServer.handler=4,port=51776" daemon prio=10 tid=0x72cc9000 nid=0x17ed
waiting for monitor entry [0x71be1000]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.hadoop.hbase.master.TableNamespaceManager.get(TableNamespaceManager.java:111)
- waiting to lock <0x7f71ba70> (a
org.apache.hadoop.hbase.master.TableNamespaceManager)
at
org.apache.hadoop.hbase.master.HMaster.getNamespaceDescriptor(HMaster.java:3076)
at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:1779)
at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:1820)
at
org.apache.hadoop.hbase.protobuf.generated.MasterAdminProtos$MasterAdminService$2.callBlockingMethod(MasterAdminProtos.java:26698)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2068)
at
org.apache.hadoop.hbase.ipc.RpcServer$CallRunner.run(RpcServer.java:1807)
at
org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:165)
at
org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$000(SimpleRpcScheduler.java:41)
at
org.apache.hadoop.hbase.ipc.SimpleRpcScheduler$1.run(SimpleRpcScheduler.java:113)
at java.lang.Thread.run(Thread.java:662)
"RpcServer.handler=3,port=51776" daemon prio=10 tid=0x72cc7800 nid=0x17ec
waiting for monitor entry [0x71c32000]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.hadoop.hbase.master.TableNamespaceManager.get(TableNamespaceManager.java:111)
- waiting to lock <0x7f71ba70> (a
org.apache.hadoop.hbase.master.TableNamespaceManager)
at
org.apache.hadoop.hbase.master.HMaster.getNamespaceDescriptor(HMaster.java:3076)
at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:1779)
at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:1820)
at
org.apache.hadoop.hbase.protobuf.generated.MasterAdminProtos$MasterAdminService$2.callBlockingMethod(MasterAdminProtos.java:26698)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2068)
at
org.apache.hadoop.hbase.ipc.RpcServer$CallRunner.run(RpcServer.java:1807)
at
org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:165)
at
org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$000(SimpleRpcScheduler.java:41)
at
org.apache.hadoop.hbase.ipc.SimpleRpcScheduler$1.run(SimpleRpcScheduler.java:113)
at java.lang.Thread.run(Thread.java:662)
"RpcServer.handler=2,port=51776" daemon prio=10 tid=0x72002c00 nid=0x17eb
waiting for monitor entry [0x71c83000]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.hadoop.hbase.master.TableNamespaceManager.get(TableNamespaceManager.java:111)
- waiting to lock <0x7f71ba70> (a
org.apache.hadoop.hbase.master.TableNamespaceManager)
at
org.apache.hadoop.hbase.master.HMaster.getNamespaceDescriptor(HMaster.java:3076)
at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:1779)
at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:1820)
at
org.apache.hadoop.hbase.protobuf.generated.MasterAdminProtos$MasterAdminService$2.callBlockingMethod(MasterAdminProtos.java:26698)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2068)
at
org.apache.hadoop.hbase.ipc.RpcServer$CallRunner.run(RpcServer.java:1807)
at
org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:165)
at
org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$000(SimpleRpcScheduler.java:41)
at
org.apache.hadoop.hbase.ipc.SimpleRpcScheduler$1.run(SimpleRpcScheduler.java:113)
at java.lang.Thread.run(Thread.java:662)
{code}
I'd guess no one is getting in here till we finish our 35 retryings (almost ten
minutes, which makes us look like a zombie).
Seems like we need to be able to interrupt in here when done or at least add
logging why we are in here having trouble trying to get from the ns table?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
