Hi everybody,
I've setted up a cluster 0.9.0.1 version.
dedicated nimbus, zookeeper, two supervisors.
Have to say that everything works just fine with one supervisor. But
when I add another one, task from one machine can't communicate with
task on another machine, and because of that, messages are getting timed
out.
I'm running everything on AWS using CentOS instances. I'v opend all
ports for supervisor security group. Stoped iptables. But still no luck.
when I do |netstat -a| I can see |ESTABLISHED| connections between
servers. I can also successfully |ping| each other.
No need to say that zookeeper works fine, and I can run |echo ruok | nc
<ip> 2181| and I'm getting result.
But when Im checking worker log, I see those errors:
|2014-03-13 13:53:28 STDIO [ERROR] Mar 13, 2014 1:53:28 PM
org.jboss.netty.channel.DefaultChannelPipeline
WARNING: An exception was thrown by a user handler while handling an exception
event ([id: 0x4bb68a13] EXCEPTION: java.net.NoRouteToHostException: No route to
host)
java.lang.IllegalArgumentException: timeout value is negative
at java.lang.Thread.sleep(Native Method)
at backtype.storm.messaging.netty.Client.reconnect(Client.java:94)
at
backtype.storm.messaging.netty.StormClientHandler.exceptionCaught(StormClientHandler.java:118)
at
org.jboss.netty.handler.codec.frame.FrameDecoder.exceptionCaught(FrameDecoder.java:377)
at org.jboss.netty.channel.Channels.fireExceptionCaught(Channels.java:525)
at
org.jboss.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:109)
at
org.jboss.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:78)
at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:312)
at
org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:41)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
2014-03-13 13:53:29 b.s.m.n.StormClientHandler [INFO] Connection failed:
java.net.NoRouteToHostException: No route to host
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[na:1.7.0_51]
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
~[na:1.7.0_51]
at
org.jboss.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:148)
~[netty-3.6.3.Final.jar:na]
at
org.jboss.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:104)
~[netty-3.6.3.Final.jar:na]
at
org.jboss.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:78)
~[netty-3.6.3.Final.jar:na]
at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:312)
~[netty-3.6.3.Final.jar:na]
at
org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:41)
~[netty-3.6.3.Final.jar:na]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[na:1.7.0_51]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[na:1.7.0_51]
at java.lang.Thread.run(Thread.java:744) [na:1.7.0_51]
|
