[
https://issues.apache.org/jira/browse/ZOOKEEPER-1715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13679271#comment-13679271
]
Sean Bridges commented on ZOOKEEPER-1715:
-----------------------------------------
Doing some more debugging, the main test thread is blocked in
NettyServerCnxnFactory#shutdown at the call to,
bootstrap.releaseExternalResources();
That call is blocked trying to shut down an executor with 1 thread blocked
trying to acquire the lock on a NettyServerCnxn, with a stack trace like,
{code}
Name: New I/O worker #4
State: BLOCKED on org.apache.zookeeper.server.NettyServerCnxn@6827a180 owned
by: CommitProcWorkThread-1
Total blocked: 2 Total waited: 0
Stack trace:
org.apache.zookeeper.server.NettyServerCnxnFactory$CnxnChannelHandler.messageReceived(NettyServerCnxnFactory.java:141)
org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88)
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:558)
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:553)
org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
org.jboss.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(AbstractNioWorker.java:441)
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:328)
org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:36)
org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
java.lang.Thread.run(Thread.java:722)
{code}
The lock is owned by another thread which is blocked trying to disableRcv with
a stack trace as shown,
{code}
Name: CommitProcWorkThread-1
State: WAITING on org.jboss.netty.channel.DefaultChannelFuture@405a6d04
Total blocked: 14 Total waited: 77
Stack trace:
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:503)
org.jboss.netty.channel.DefaultChannelFuture.awaitUninterruptibly(DefaultChannelFuture.java:258)
org.apache.zookeeper.server.NettyServerCnxn.disableRecv(NettyServerCnxn.java:810)
org.apache.zookeeper.server.NettyServerCnxn.receiveMessage(NettyServerCnxn.java:742)
org.apache.zookeeper.server.NettyServerCnxnFactory$CnxnChannelHandler.processMessage(NettyServerCnxnFactory.java:164)
org.apache.zookeeper.server.NettyServerCnxnFactory$CnxnChannelHandler.messageReceived(NettyServerCnxnFactory.java:141)
- locked org.apache.zookeeper.server.NettyServerCnxn@6827a180
org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88)
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:558)
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:553)
org.apache.zookeeper.server.NettyServerCnxn.enableRecv(NettyServerCnxn.java:207)
org.apache.zookeeper.server.NettyServerCnxn.sendResponse(NettyServerCnxn.java:190)
org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:416)
org.apache.zookeeper.server.quorum.CommitProcessor$CommitWorkRequest.doWork(CommitProcessor.java:271)
org.apache.zookeeper.server.WorkerService$ScheduledWorkRequest.run(WorkerService.java:152)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
java.lang.Thread.run(Thread.java:722)
{code}
The attached patch fixes the test by adding a non blocking version of
disableRecv, but I am not sure that is a good approach.
> Upgrade netty version
> ---------------------
>
> Key: ZOOKEEPER-1715
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1715
> Project: ZooKeeper
> Issue Type: Improvement
> Affects Versions: 3.4.5
> Environment: zookeeper 3.4.5 uses netty 3.2.2, which was released in
> August 2010. The latest version of netty is 3.6.6 released May 2013.
> Zookeeper should consider upgrading.
> Reporter: Sean Bridges
> Fix For: 3.5.0
>
> Attachments: ZOOKEEPER-1715-2.patch, ZOOKEEPER-1715.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
