Hi Yang, >From the stack trace you posted it appears that you are using a version of Avro prior to 1.5.2. Which version are you using? There have been a number of significant changes recently to the RPC framework and the Netty implementation in particular. Could you please try to reproduce the problem using Avro 1.5.2 or newer? The problem may be resolved with an upgrade. If the problem still exists in the newer versions, it will be a lot easier to diagnose/fix it if we can see stack traces from a post-1.5.2 version.
Thanks, James On Wed, Sep 14, 2011 at 1:08 PM, Yang <[email protected]> wrote: > I'm always seeing these "channel closed " exceptions , with low > probability, i.e. about every 10 hours under heavy load. > > I'm not sure if it's the server that got the channel closed or the > client, so I included the exception stack from both sides. > anybody has an idea how to debug this? > > also, let's say it does have a valid reason for closing this, what is > my strategy of coping with this? I originally have many > senders, due to the channel close exception, many of them died, after > this, only 2 application threads remain, but they > all seem blocked on trying to grab a connection from Netty's pool, so > even if I create new sender threads, it seems they would still > block. so how can I tell netty to "reset/replenish " its connections? > > > Thanks a lot > Yang > > > client side: > > > > WARN 16:51:02,079 Unexpected exception from downstream. > java.nio.channels.ClosedChannelException > at > org.jboss.netty.channel.socket.nio.NioWorker.cleanUpWriteBuffer(NioWorker.java:636) > at > org.jboss.netty.channel.socket.nio.NioWorker.writeFromUserCode(NioWorker.java:369) > at > org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:117) > at org.jboss.netty.channel.Channels.write(Channels.java:632) > at > org.jboss.netty.handler.codec.oneone.OneToOneEncoder.handleDownstream(OneToOneEncoder.java:70) > at org.jboss.netty.channel.Channels.write(Channels.java:611) > at org.jboss.netty.channel.Channels.write(Channels.java:578) > at > org.jboss.netty.channel.AbstractChannel.write(AbstractChannel.java:259) > at > org.apache.avro.ipc.NettyTransceiver.transceive(NettyTransceiver.java:131) > at org.apache.avro.ipc.Requestor.request(Requestor.java:134) > at > org.apache.avro.ipc.specific.SpecificRequestor.invoke(SpecificRequestor.java:55) > at $Proxy0.collect_ad_click(Unknown Source) > > > > server side: > > > WARN 16:51:01,939 Unexpected exception from downstream. > java.io.IOException: Broken pipe > at sun.nio.ch.FileDispatcher.write0(Native Method) > at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47) > at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:122) > at sun.nio.ch.IOUtil.write(IOUtil.java:78) > at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:352) > at > org.jboss.netty.channel.socket.nio.SocketSendBufferPool$PooledSendBuffer.transferTo(SocketSendBufferPool.java:239) > at > org.jboss.netty.channel.socket.nio.NioWorker.write0(NioWorker.java:469) > at > org.jboss.netty.channel.socket.nio.NioWorker.writeFromUserCode(NioWorker.java:387) > at > org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink.handleAcceptedSocket(NioServerSocketPipelineSink.java:137) > at > org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink.eventSunk(NioServerSocketPipelineSink.java:76) > at org.jboss.netty.channel.Channels.write(Channels.java:632) > at > org.jboss.netty.handler.codec.oneone.OneToOneEncoder.handleDownstream(OneToOneEncoder.java:70) > at org.jboss.netty.channel.Channels.write(Channels.java:611) > at org.jboss.netty.channel.Channels.write(Channels.java:578) > at > org.jboss.netty.channel.AbstractChannel.write(AbstractChannel.java:259) > at > org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.messageReceived(NettyServer.java:137) > at > org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:120) > at > org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:302) > at > org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:317) > at > org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:299) > at > org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:216) > at > org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:274) > at > org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:261) > at > org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:349) > at > org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:280) > at > org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:200) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) > at java.lang.Thread.run(Thread.java:679) > > > > > > after exception, the client sender's state, blocked : > > > "EventQueue-Sender5" prio=10 tid=0x00007f5a0c2f5000 nid=0x6e3b waiting > on condition [0x00007f5a19519000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00000007510e00c0> (a > java.util.concurrent.Semaphore$NonfairSync) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:838) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:998) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) > at java.util.concurrent.Semaphore.acquire(Semaphore.java:313) > at > org.apache.avro.ipc.NettyTransceiver$CallFuture.get(NettyTransceiver.java:203) > at > org.apache.avro.ipc.NettyTransceiver.transceive(NettyTransceiver.java:133) > at org.apache.avro.ipc.Requestor.request(Requestor.java:134) > - locked <0x0000000757144220> (a > org.apache.avro.ipc.specific.SpecificRequestor) > at > org.apache.avro.ipc.specific.SpecificRequestor.invoke(SpecificRequestor.java:55) > at $Proxy0.collect_ad_click(Unknown Source) > at > com.cgm.whisky.emitter.ConnectionPool$EventsCollectorWithSerial.collect_ad_click(ConnectionPool.java:60) >
