Sergey Shelukhin created HBASE-22353:
----------------------------------------
Summary: update non-shaded netty for Hadoop 2 to a more recent
version of 3.6
Key: HBASE-22353
URL: https://issues.apache.org/jira/browse/HBASE-22353
Project: HBase
Issue Type: Bug
Reporter: Sergey Shelukhin
When using Netty socket for ZK, we got this deadlock.
Appears to be https://github.com/netty/netty/issues/1181 (or one of similar
tickets before that).
We are using Netty 3.6.2 for Hadoop 2, seems like it should be safe to upgrade
to 3.6.10, assuming it's purely a bugfix release for 3.6.2 and they are
compatible?
{noformat}
Java stack information for the threads listed above:
===================================================
"main-SendThread(...)":
at org.jboss.netty.handler.ssl.SslHandler.wrap(SslHandler.java:958)
- waiting to lock <0x00000000c91d8848> (a java.lang.Object)
- locked <0x00000000cdcc7740> (a java.util.LinkedList)
at
org.jboss.netty.handler.ssl.SslHandler.handleDownstream(SslHandler.java:627)
at
org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:587)
at
org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:578)
at org.jboss.netty.channel.Channels.write(Channels.java:704)
at org.jboss.netty.channel.Channels.write(Channels.java:671)
at
org.jboss.netty.channel.AbstractChannel.write(AbstractChannel.java:248)
at
org.apache.zookeeper.ClientCnxnSocketNetty.sendPkt(ClientCnxnSocketNetty.java:268)
at
org.apache.zookeeper.ClientCnxnSocketNetty.doWrite(ClientCnxnSocketNetty.java:291)
at
org.apache.zookeeper.ClientCnxnSocketNetty.doTransport(ClientCnxnSocketNetty.java:249)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1146)
"New I/O worker #3":
at
org.jboss.netty.handler.ssl.SslHandler.channelClosed(SslHandler.java:1554)
- waiting to lock <0x00000000cdcc7740> (a java.util.LinkedList)
at
org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:88)
at
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
at
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:555)
at org.jboss.netty.channel.Channels.fireChannelClosed(Channels.java:468)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.close(AbstractNioWorker.java:351)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.write0(AbstractNioWorker.java:254)
- locked <0x00000000c91d8770> (a java.lang.Object)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.writeFromUserCode(AbstractNioWorker.java:145)
at
org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:83)
at
org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:775)
at org.jboss.netty.channel.Channels.write(Channels.java:725)
at org.jboss.netty.channel.Channels.write(Channels.java:686)
at
org.jboss.netty.handler.ssl.SslHandler.wrapNonAppData(SslHandler.java:1140)
at org.jboss.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1229)
- locked <0x00000000c91d8848> (a java.lang.Object)
at org.jboss.netty.handler.ssl.SslHandler.decode(SslHandler.java:910)
at
org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:425)
at
org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
at
org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
at
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
at
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:555)
at
org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
at
org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:107)
at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:312)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at
org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Found 1 deadlock.
{noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
