Hi everyone, I’m currently setting up a storm cluster on AWS, but I have some problems with how storm supervisor connects to the peer nodes. On AWS I’m running storm in an auto scaling group, meaning that if one of the nodes go down, it automatically re-spawns another node with the same image. On AWS any node can disappear at any time, therefore it’s important to run a setup similar to this. All nodes have a hostname like ip-XX-XX-XX-XX representing their IP. However, they are not able to connect to the others hostnames, e.g. hostname ip-10-0-0-1 will not know about the hostname ip-10-0-0-2, although it could access 10.0.0.2. This is resulting in errors like this on the supervisors:
2014-07-08 11:43:05 b.s.d.executor [INFO] Loaded executor tasks count-bolt:[3 3] 2014-07-08 11:43:05 b.s.d.executor [INFO] Finished loading executor count-bolt:[3 3] 2014-07-08 11:43:05 b.s.d.executor [INFO] Loading executor word-spout:[5 5] 2014-07-08 11:43:05 b.s.d.executor [INFO] Loaded executor tasks word-spout:[5 5] 2014-07-08 11:43:05 b.s.d.executor [INFO] Preparing bolt count-bolt:(3) 2014-07-08 11:43:05 b.s.d.executor [INFO] Finished loading executor word-spout:[5 5] 2014-07-08 11:43:05 b.s.d.executor [INFO] Opening spout word-spout:(5) 2014-07-08 11:43:05 b.s.d.executor [INFO] Loading executor __system:[-1 -1] 2014-07-08 11:43:05 b.s.d.executor [INFO] Loaded executor tasks __system:[-1 -1] 2014-07-08 11:43:05 b.s.d.executor [INFO] Finished loading executor __system:[-1 -1] 2014-07-08 11:43:05 b.s.u.ShellProcess [INFO] Storm multilang serializer: backtype.storm.multilang.JsonSerializer 2014-07-08 11:43:05 b.s.u.ShellProcess [INFO] Storm multilang serializer: backtype.storm.multilang.JsonSerializer 2014-07-08 11:43:05 b.s.d.executor [INFO] Preparing bolt __system:(-1) 2014-07-08 11:43:05 b.s.m.n.Client [INFO] Reconnect started for Netty-Client-ip-10-0-11-36:6703... [4] 2014-07-08 11:43:05 b.s.m.n.StormClientErrorHandler [INFO] Connection failed Netty-Client-ip-10-0-11-36:6703 java.nio.channels.UnresolvedAddressException: null at sun.nio.ch.Net.checkAddress(Net.java:127) ~[na:1.7.0_55] at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:644) ~[na:1.7.0_55] at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.connect(NioClientSocketPipelineSink.java:140) ~[netty-3.2.2.Final.jar:na] at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:103) ~[netty-3.2.2.Final.jar:na] at org.jboss.netty.handler.codec.oneone.OneToOneEncoder.handleDownstream(OneToOneEncoder.java:60) ~[netty-3.2.2.Final.jar:na] at org.jboss.netty.channel.Channels.connect(Channels.java:541) [netty-3.2.2.Final.jar:na] at org.jboss.netty.channel.AbstractChannel.connect(AbstractChannel.java:218) [netty-3.2.2.Final.jar:na] at org.jboss.netty.bootstrap.ClientBootstrap.connect(ClientBootstrap.java:227) [netty-3.2.2.Final.jar:na] at org.jboss.netty.bootstrap.ClientBootstrap.connect(ClientBootstrap.java:188) [netty-3.2.2.Final.jar:na] at backtype.storm.messaging.netty.Client.connect(Client.java:147) [storm-core-0.9.2-incubating.jar:0.9.2-incubating] at backtype.storm.messaging.netty.Client.access$000(Client.java:42) [storm-core-0.9.2-incubating.jar:0.9.2-incubating] at backtype.storm.messaging.netty.Client$1.run(Client.java:104) [storm-core-0.9.2-incubating.jar:0.9.2-incubating] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [na:1.7.0_55] at java.util.concurrent.FutureTask.run(FutureTask.java:262) [na:1.7.0_55] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178) [na:1.7.0_55] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292) [na:1.7.0_55] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_55] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_55] at java.lang.Thread.run(Thread.java:744) [na:1.7.0_55] I’m a bit unsure how to proceed. Is it possible to form to register the IP instead of the hostname somehow? Or is there another way to do this? Kind regards, Kristian
