Enis Soztutar created HBASE-14474:
-------------------------------------
Summary: DeadLock in RpcClientImpl.Connection.close()
Key: HBASE-14474
URL: https://issues.apache.org/jira/browse/HBASE-14474
Project: HBase
Issue Type: Bug
Components: rpc
Reporter: Enis Soztutar
Assignee: Enis Soztutar
Priority: Blocker
Fix For: 2.0.0, 1.2.0, 1.0.3, 1.1.3
>From a code base that contains 1.1.2 + HBASE-14449 + HBASE-14241 and
>HBASE-14313 we can reproduce a dead lock with serverKilling CM easily:
{code}
Found one Java-level deadlock: =============================
"htable-pool1-t63": waiting to lock monitor 0x0000000001cb1688 (object
0x00000000806ef150, a org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection),
which is held by "IPC Client (1403704789) connection to
enis-hbase-sep-21-6.novalocal/172.22.107.106:16020 from root" "IPC Client
(1403704789) connection to enis-hbase-sep-21-6.novalocal/172.22.107.106:16020
from root": waiting to lock monitor 0x0000000001cb1738 (object
0x00000000806f0c60, a java.lang.Object), which is held by "htable-pool1-t63"
Java stack information for the threads listed above:
=================================================== "htable-pool1-t63":
at
org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.close(RpcClientImpl.java:819)
- waiting to lock <0x00000000806ef150> (a
org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection) at
org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.writeRequest(RpcClientImpl.java:906)
- locked <0x00000000806f0c60> (a java.lang.Object) at
org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.tracedWriteRequest(RpcClientImpl.java:856)
at
org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1192)
at
org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:213)
at
org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:287)
at
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.multi(ClientProtos.java:32699)
at
org.apache.hadoop.hbase.client.MultiServerCallable.call(MultiServerCallable.java:129)
at
org.apache.hadoop.hbase.client.MultiServerCallable.call(MultiServerCallable.java:54)
at
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200)
at
org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncProcess.java:708)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266) at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745) "IPC Client (1403704789)
connection to enis-hbase-sep-21-6.novalocal/172.22.107.106:16020 from root":
at
org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.close(RpcClientImpl.java:832)
- waiting to lock <0x00000000806f0c60> (a java.lang.Object) -
locked <0x00000000806ef150> (a
org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection) at
org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.run(RpcClientImpl.java:574)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
