Hi Manu, What version do you use? And could you provide more logs around this part? Seems like thrift connection stayed open too long and eventually disconnected from other side, and Nimbus didn't discard that connection. (Do you think it's related to TCP keepalive? Just guessing that you state more than 2 hrs.)
Thanks, Jungtaek Lim (HeartSaVioR) 2016년 8월 30일 (화) 오전 10:09, Manu Zhang <[email protected]>님이 작성: > Hi all, > > I have a topology running for more than 2 hours and then I cannot kill the > topology from dashboard. A dialog with "Error communicating with Nimbus" > pops out. The nimbus log shows the following exceptions. > > 2016-08-30 09:01:35.758 o.a.s.t.s.AbstractNonblockingServer$FrameBuffer > [WARN] Got an IOException during write! > java.io.IOException: Connection reset by peer > at sun.nio.ch.FileDispatcherImpl.write0(Native Method) > at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47) > at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93) > at sun.nio.ch.IOUtil.write(IOUtil.java:65) > at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:471) > at > org.apache.storm.thrift.transport.TNonblockingSocket.write(TNonblockingSocket.java:165) > at > org.apache.storm.thrift.server.AbstractNonblockingServer$FrameBuffer.write(AbstractNonblockingServer.java:414) > at > org.apache.storm.thrift.server.AbstractNonblockingServer$AbstractSelectThread.handleWrite(AbstractNonblockingServer.java:221) > at > org.apache.storm.thrift.server.TNonblockingServer$SelectAcceptThread.select(TNonblockingServer.java:206) > at > org.apache.storm.thrift.server.TNonblockingServer$SelectAcceptThread.run(TNonblockingServer.java:154) > 2016-08-30 09:01:54.321 o.a.s.t.s.AbstractNonblockingServer$FrameBuffer > [WARN] Got an IOException during write! > java.io.IOException: Connection reset by peer > at sun.nio.ch.FileDispatcherImpl.write0(Native Method) > at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47) > at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93) > at sun.nio.ch.IOUtil.write(IOUtil.java:65) > at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:471) > at > org.apache.storm.thrift.transport.TNonblockingSocket.write(TNonblockingSocket.java:165) > at > org.apache.storm.thrift.server.AbstractNonblockingServer$FrameBuffer.write(AbstractNonblockingServer.java:414) > at > org.apache.storm.thrift.server.AbstractNonblockingServer$AbstractSelectThread.handleWrite(AbstractNonblockingServer.java:221) > at > org.apache.storm.thrift.server.TNonblockingServer$SelectAcceptThread.select(TNonblockingServer.java:206) > at > org.apache.storm.thrift.server.TNonblockingServer$SelectAcceptThread.run(TNonblockingServer.java:154) > > I can still kill the topology from command line. > > Any thoughts ? > > > Thanks, > Manu Zhang > >
