I'm always seeing these "channel closed " exceptions , with low
probability, i.e. about every 10 hours under heavy load.
I'm not sure if it's the server that got the channel closed or the
client, so I included the exception stack from both sides.
anybody has an idea how to debug this?
also, let's say it does have a valid reason for closing this, what is
my strategy of coping with this? I originally have many
senders, due to the channel close exception, many of them died, after
this, only 2 application threads remain, but they
all seem blocked on trying to grab a connection from Netty's pool, so
even if I create new sender threads, it seems they would still
block. so how can I tell netty to "reset/replenish " its connections?
Thanks a lot
Yang
client side:
WARN 16:51:02,079 Unexpected exception from downstream.
java.nio.channels.ClosedChannelException
at
org.jboss.netty.channel.socket.nio.NioWorker.cleanUpWriteBuffer(NioWorker.java:636)
at
org.jboss.netty.channel.socket.nio.NioWorker.writeFromUserCode(NioWorker.java:369)
at
org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:117)
at org.jboss.netty.channel.Channels.write(Channels.java:632)
at
org.jboss.netty.handler.codec.oneone.OneToOneEncoder.handleDownstream(OneToOneEncoder.java:70)
at org.jboss.netty.channel.Channels.write(Channels.java:611)
at org.jboss.netty.channel.Channels.write(Channels.java:578)
at
org.jboss.netty.channel.AbstractChannel.write(AbstractChannel.java:259)
at
org.apache.avro.ipc.NettyTransceiver.transceive(NettyTransceiver.java:131)
at org.apache.avro.ipc.Requestor.request(Requestor.java:134)
at
org.apache.avro.ipc.specific.SpecificRequestor.invoke(SpecificRequestor.java:55)
at $Proxy0.collect_ad_click(Unknown Source)
server side:
WARN 16:51:01,939 Unexpected exception from downstream.
java.io.IOException: Broken pipe
at sun.nio.ch.FileDispatcher.write0(Native Method)
at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:122)
at sun.nio.ch.IOUtil.write(IOUtil.java:78)
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:352)
at
org.jboss.netty.channel.socket.nio.SocketSendBufferPool$PooledSendBuffer.transferTo(SocketSendBufferPool.java:239)
at
org.jboss.netty.channel.socket.nio.NioWorker.write0(NioWorker.java:469)
at
org.jboss.netty.channel.socket.nio.NioWorker.writeFromUserCode(NioWorker.java:387)
at
org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink.handleAcceptedSocket(NioServerSocketPipelineSink.java:137)
at
org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink.eventSunk(NioServerSocketPipelineSink.java:76)
at org.jboss.netty.channel.Channels.write(Channels.java:632)
at
org.jboss.netty.handler.codec.oneone.OneToOneEncoder.handleDownstream(OneToOneEncoder.java:70)
at org.jboss.netty.channel.Channels.write(Channels.java:611)
at org.jboss.netty.channel.Channels.write(Channels.java:578)
at
org.jboss.netty.channel.AbstractChannel.write(AbstractChannel.java:259)
at
org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.messageReceived(NettyServer.java:137)
at
org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:120)
at
org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:302)
at
org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:317)
at
org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:299)
at
org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:216)
at
org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:274)
at
org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:261)
at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:349)
at
org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:280)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:200)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
after exception, the client sender's state, blocked :
"EventQueue-Sender5" prio=10 tid=0x00007f5a0c2f5000 nid=0x6e3b waiting
on condition [0x00007f5a19519000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000007510e00c0> (a
java.util.concurrent.Semaphore$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:838)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:998)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
at java.util.concurrent.Semaphore.acquire(Semaphore.java:313)
at
org.apache.avro.ipc.NettyTransceiver$CallFuture.get(NettyTransceiver.java:203)
at
org.apache.avro.ipc.NettyTransceiver.transceive(NettyTransceiver.java:133)
at org.apache.avro.ipc.Requestor.request(Requestor.java:134)
- locked <0x0000000757144220> (a
org.apache.avro.ipc.specific.SpecificRequestor)
at
org.apache.avro.ipc.specific.SpecificRequestor.invoke(SpecificRequestor.java:55)
at $Proxy0.collect_ad_click(Unknown Source)
at
com.cgm.whisky.emitter.ConnectionPool$EventsCollectorWithSerial.collect_ad_click(ConnectionPool.java:60)
