The exception you're seeing is an issue that was fixed in the latest storm release (0.9.2-incubating): https://issues.apache.org/jira/browse/STORM-187
If you are required to use an older version, one thing you can do is replace netty with 0mq as discussed here: http://mail-archives.apache.org/mod_mbox/storm-user/201404.mbox/%3CCALFqTqTS5t8=igkpcjs+b96vwvcdsfkesv++ym6osa6bph4...@mail.gmail.com%3E On Wed, Sep 10, 2014 at 3:21 PM, Snehal <snehal.ku...@gmail.com> wrote: > We found the following exception stacktrace during the startup of the > worker: > > 2014-09-09 01:55:29 STDIO [ERROR] Sep 09, 2014 1:55:29 AM > org.jboss.netty.channel.DefaultChannelPipeline > WARNING: An exception was thrown by a user handler while handling an > exception event ([id: 0x4cb6a9a5] EXCEPTION: java.net.ConnectException: > Connection refused) > java.lang.IllegalArgumentException: timeout value is negative > at java.lang.Thread.sleep(Native Method) > at backtype.storm.messaging.netty.Client.reconnect(Client.java:94) > at > backtype.storm.messaging.netty.StormClientHandler.exceptionCaught(StormClientHandler.java:118) > at > org.jboss.netty.handler.codec.frame.FrameDecoder.exceptionCaught(FrameDecoder.java:377) > at > org.jboss.netty.channel.Channels.fireExceptionCaught(Channels.java:525) > at > org.jboss.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:109) > at > org.jboss.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:78) > at > org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:312) > at > org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:41) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > 2014-09-09 01:55:30 b.s.m.n.Client [INFO] Reconnect ... [24] > 2014-09-09 01:55:31 b.s.m.n.Client [INFO] Reconnect ... [25] > 2014-09-09 01:55:31 STDIO [ERROR] Sep 09, 2014 1:55:31 AM > org.jboss.netty.channel.DefaultChannelPipeline > WARNING: An exception was thrown by a user handler while handling an > exception event ([id: 0xa80acab1] EXCEPTION: java.net.ConnectException: > Connection refused) > java.lang.IllegalArgumentException: timeout value is negative > at java.lang.Thread.sleep(Native Method) > at backtype.storm.messaging.netty.Client.reconnect(Client.java:94) > at > backtype.storm.messaging.netty.StormClientHandler.exceptionCaught(StormClientHandler.java:118) > at > org.jboss.netty.handler.codec.frame.FrameDecoder.exceptionCaught(FrameDecoder.java:377) > at > org.jboss.netty.channel.Channels.fireExceptionCaught(Channels.java:525) > at > org.jboss.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:109) > at > org.jboss.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:78) > at > org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:312) > at > org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:41) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > > On Tue, Sep 9, 2014 at 4:22 PM, Snehal <snehal.ku...@gmail.com> wrote: > >> Hello, >> >> We have a multi-node storm cluster and we are using *Guaranteed >> processing* of Storm for making sure that events are processed exactly >> once. >> >> Our topology consists of a spout and multiple bolts (50). We have set the >> number of acker equivalent to the number of workers. From our logs we can >> identify that bolts are acking but the spout may not receive them. We do >> notice that the spout is receiving the ack from bolts which are on the same >> node. We believe the issue to be either network connectivity or ack timing >> out. >> >> Any idea what is cause of why the remote bolts are unable to ack and how >> do we debug this? Any suggestions are appreciated. >> >> Thanks, >> Snehal >> > >
