Hello all,
I have recently ported my 5-node cluster on Amazon EC2. In detail, I have
one node running Nimbus and ZooKeeper, and 4 nodes running supervisors. All
of my instances are t2.micro type (1Ghz vCPU and 1GB RAM) and I am using
Storm 0.9.4, ZooKeeper 3.4.6, Open JDK v1.7, and all images run on Ubuntu
v14 64-bit.
Even though I have tested my topologies in my local cluster, and they work
without any problems (my local cluster runs on Storm 0.9.2), when I deploy
my topology on the EC2 nodes, I receive the following error on one specific
Bolt, every time:
2015-05-21T20:45:09.867+0000 b.s.d.worker [INFO] Worker has topology config
{"storm.id" "operator-serial-top-22-1432241101", "dev.zookeeper.path"
"/tmp/dev-storm-zookeeper", "topology.tick.tuple.freq.secs" nil,
"topology.builtin.metrics.bucket.size.secs" 60,
"topology.fall.back.on.java.serialization" true,
"topology.max.error.report.per.interval" 5, "zmq.linger.millis" 5000,
"topology.skip.missing.kryo.registrations" false,
"storm.messaging.netty.client_worker_threads" 1, "ui.childopts" "-Xmx768m",
"storm.zookeeper.session.timeout" 20000, "nimbus.reassign" true,
"topology.trident.batch.emit.interval.millis" 500, "
storm.messaging.netty.flush.check.interval.ms" 10,
"nimbus.monitor.freq.secs" 10, "logviewer.childopts" "-Xmx128m",
"java.library.path" "/usr/local/lib:/opt/local/lib:/usr/lib", "storm.home"
"/opt/storm-0.9.4", "topology.executor.send.buffer.size" 1024,
"storm.local.dir" "/mnt/storm", "storm.messaging.netty.buffer_size"
5242880, "supervisor.worker.start.timeout.secs" 120,
"topology.enable.message.timeouts" true, "nimbus.cleanup.inbox.freq.secs"
600, "nimbus.inbox.jar.expiration.secs" 3600, "drpc.worker.threads" 64,
"storm.meta.serialization.delegate"
"backtype.storm.serialization.DefaultSerializationDelegate",
"topology.worker.shared.thread.pool.size" 4, "nimbus.host" "52.7.165.232",
"storm.messaging.netty.min_wait_ms" 100, "storm.zookeeper.port" 2181,
"transactional.zookeeper.port" nil, "topology.executor.receive.buffer.size"
1024, "transactional.zookeeper.servers" nil, "storm.zookeeper.root"
"/storm", "storm.zookeeper.retry.intervalceiling.millis" 30000,
"supervisor.enable" true, "storm.messaging.netty.server_worker_threads" 1,
"storm.zookeeper.servers" ["52.7.165.232"], "transactional.zookeeper.root"
"/transactional", "topology.acker.executors" nil,
"topology.kryo.decorators" (), "topology.name" "operator-serial-top",
"topology.transfer.buffer.size" 1024, "topology.worker.childopts" nil,
"drpc.queue.size" 128, "worker.childopts" "-Xmx768m",
"supervisor.heartbeat.frequency.secs" 5,
"topology.error.throttle.interval.secs" 10, "zmq.hwm" 0, "drpc.port" 3772,
"supervisor.monitor.frequency.secs" 3, "drpc.childopts" "-Xmx768m",
"topology.receiver.buffer.size" 8, "task.heartbeat.frequency.secs" 3,
"topology.tasks" nil, "storm.messaging.netty.max_retries" 300,
"topology.spout.wait.strategy"
"backtype.storm.spout.SleepSpoutWaitStrategy",
"nimbus.thrift.max_buffer_size" 1048576, "topology.max.spout.pending" nil,
"storm.zookeeper.retry.interval" 1000, "
topology.sleep.spout.wait.strategy.time.ms" 1, "nimbus.topology.validator"
"backtype.storm.nimbus.DefaultTopologyValidator", "supervisor.slots.ports"
[6700 6701 6702 6703], "topology.environment" nil, "topology.debug" false,
"nimbus.task.launch.secs" 120, "nimbus.supervisor.timeout.secs" 60,
"topology.kryo.register" nil, "topology.message.timeout.secs" 30,
"task.refresh.poll.secs" 10, "topology.workers" 4, "supervisor.childopts"
"-Xmx256m", "nimbus.thrift.port" 6627, "topology.stats.sample.rate" 0.05,
"worker.heartbeat.frequency.secs" 1, "topology.tuple.serializer"
"backtype.storm.serialization.types.ListDelegateSerializer",
"topology.disruptor.wait.strategy"
"com.lmax.disruptor.BlockingWaitStrategy", "topology.multilang.serializer"
"backtype.storm.multilang.JsonSerializer", "nimbus.task.timeout.secs" 30,
"storm.zookeeper.connection.timeout" 15000, "topology.kryo.factory"
"backtype.storm.serialization.DefaultKryoFactory", "drpc.invocations.port"
3773, "logviewer.port" 8000, "zmq.threads" 1, "storm.zookeeper.retry.times"
5, "topology.worker.receiver.thread.count" 1, "storm.thrift.transport"
"backtype.storm.security.auth.SimpleTransportPlugin",
"topology.state.synchronization.timeout.secs" 60,
"supervisor.worker.timeout.secs" 30, "nimbus.file.copy.expiration.secs"
600, "storm.messaging.transport" "backtype.storm.messaging.netty.Context", "
logviewer.appender.name" "A1", "storm.messaging.netty.max_wait_ms" 1000,
"drpc.request.timeout.secs" 600, "storm.local.mode.zmq" false, "ui.port"
8080, "nimbus.childopts" "-Xmx1024m", "storm.cluster.mode" "distributed",
"topology.max.task.parallelism" nil,
"storm.messaging.netty.transfer.batch.size" 262144, "topology.classpath"
nil}
2015-05-21T20:45:09.867+0000 b.s.d.worker [INFO] Worker
6572b1be-a3ef-4115-bc60-13ab66c0771c for storm
operator-serial-top-22-1432241101 on
8406ba95-4c0a-4af7-8f65-5feba9eb7022:6703 has finished loading
2015-05-21T20:45:10.188+0000 b.s.d.worker [INFO] All connections are ready
for worker 8406ba95-4c0a-4af7-8f65-5feba9eb7022:6703 with id
6572b1be-a3ef-4115-bc60-13ab66c0771c
2015-05-21T20:45:10.245+0000 b.s.d.executor [INFO] Preparing bolt join_1:(6)
2015-05-21T20:45:10.252+0000 g.k.s.s.a.OperatorBolt [INFO] +OPERATOR-BOLT (
join_1:[email protected]) in retrieveDownstreamTasks().
2015-05-21T20:45:10.252+0000 b.s.d.executor [INFO] Preparing bolt
__system:(-1)
2015-05-21T20:45:10.263+0000 b.s.d.executor [INFO] Prepared bolt
__system:(-1)
2015-05-21T20:45:10.286+0000 b.s.d.executor [INFO] Preparing bolt
__acker:(2)
2015-05-21T20:45:10.288+0000 b.s.d.executor [INFO] Prepared bolt __acker:(2)
2015-05-21T20:45:10.768+0000 b.s.d.executor [INFO] Prepared bolt join_1:(6)
2015-05-21T20:45:10.985+0000 o.a.z.ZooKeeper [INFO] Client
environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT
2015-05-21T20:45:10.985+0000 o.a.z.ZooKeeper [INFO] Client environment:
host.name=ip-172-31-54-111.ec2.internal
2015-05-21T20:45:10.985+0000 o.a.z.ZooKeeper [INFO] Client
environment:java.version=1.7.0_79
2015-05-21T20:45:10.985+0000 o.a.z.ZooKeeper [INFO] Client
environment:java.vendor=Oracle Corporation
2015-05-21T20:45:10.985+0000 o.a.z.ZooKeeper [INFO] Client
environment:java.home=/usr/lib/jvm/java-7-openjdk-amd64/jre
2015-05-21T20:45:10.985+0000 o.a.z.ZooKeeper [INFO] Client
environment:java.class.path=/opt/apache-storm-0.9.4/lib/joda-time-2.0.jar:/opt/apache-storm-0.9.4/lib/commons-lang-2.5.jar:/opt/apache-storm-0.9.4/lib/commons-exec-1.1.jar:/opt/apache-storm-0.9.4/lib/commons-logging-1.1.3.jar:/opt/apache-storm-0.9.4/lib/clj-time-0.4.1.jar:/opt/apache-storm-0.9.4/lib/jetty-6.1.26.jar:/opt/apache-storm-0.9.4/lib/logback-classic-1.0.13.jar:/opt/apache-storm-0.9.4/lib/clojure-1.5.1.jar:/opt/apache-storm-0.9.4/lib/json-simple-1.1.jar:/opt/apache-storm-0.9.4/lib/logback-core-1.0.13.jar:/opt/apache-storm-0.9.4/lib/servlet-api-2.5.jar:/opt/apache-storm-0.9.4/lib/commons-fileupload-1.2.1.jar:/opt/apache-storm-0.9.4/lib/minlog-1.2.jar:/opt/apache-storm-0.9.4/lib/tools.logging-0.2.3.jar:/opt/apache-storm-0.9.4/lib/kryo-2.21.jar:/opt/apache-storm-0.9.4/lib/jetty-util-6.1.26.jar:/opt/apache-storm-0.9.4/lib/chill-java-0.3.5.jar:/opt/apache-storm-0.9.4/lib/math.numeric-tower-0.0.1.jar:/opt/apache-storm-0.9.4/lib/log4j-over-slf4j-1.6.6.jar:/opt/apache-storm-0.9.4/lib/tools.cli-0.2.4.jar:/opt/apache-storm-0.9.4/lib/asm-4.0.jar:/opt/apache-storm-0.9.4/lib/ring-jetty-adapter-0.3.11.jar:/opt/apache-storm-0.9.4/lib/objenesis-1.2.jar:/opt/apache-storm-0.9.4/lib/jgrapht-core-0.9.0.jar:/opt/apache-storm-0.9.4/lib/commons-io-2.4.jar:/opt/apache-storm-0.9.4/lib/ring-devel-0.3.11.jar:/opt/apache-storm-0.9.4/lib/ring-core-1.1.5.jar:/opt/apache-storm-0.9.4/lib/tools.macro-0.1.0.jar:/opt/apache-storm-0.9.4/lib/carbonite-1.4.0.jar:/opt/apache-storm-0.9.4/lib/hiccup-0.3.6.jar:/opt/apache-storm-0.9.4/lib/core.incubator-0.1.0.jar:/opt/apache-storm-0.9.4/lib/compojure-1.1.3.jar:/opt/apache-storm-0.9.4/lib/commons-codec-1.6.jar:/opt/apache-storm-0.9.4/lib/storm-core-0.9.4.jar:/opt/apache-storm-0.9.4/lib/disruptor-2.10.1.jar:/opt/apache-storm-0.9.4/lib/jline-2.11.jar:/opt/apache-storm-0.9.4/lib/slf4j-api-1.7.5.jar:/opt/apache-storm-0.9.4/lib/clj-stacktrace-0.2.2.jar:/opt/apache-storm-0.9.4/lib/ring-servlet-0.3.11.jar:/opt/apache-storm-0.9.4/lib/reflectasm-1.07-shaded.jar:/opt/apache-storm-0.9.4/lib/snakeyaml-1.11.jar:/opt/apache-storm-0.9.4/lib/clout-1.0.1.jar:/opt/apache-storm-0.9.4/conf:/mnt/storm/supervisor/stormdist/operator-serial-top-22-1432241101/stormjar.jar
2015-05-21T20:45:10.985+0000 o.a.z.ZooKeeper [INFO] Client
environment:java.library.path=/mnt/storm/supervisor/stormdist/operator-serial-top-22-1432241101/resources/Linux-amd64:/mnt/storm/supervisor/stormdist/operator-serial-top-22-1432241101/resources:/usr/local/lib:/opt/local/lib:/usr/lib
2015-05-21T20:45:10.986+0000 o.a.z.ZooKeeper [INFO] Client
environment:java.io.tmpdir=/tmp
2015-05-21T20:45:10.986+0000 o.a.z.ZooKeeper [INFO] Client
environment:java.compiler=<NA>
2015-05-21T20:45:10.986+0000 o.a.z.ZooKeeper [INFO] Client environment:
os.name=Linux
2015-05-21T20:45:10.986+0000 o.a.z.ZooKeeper [INFO] Client
environment:os.arch=amd64
2015-05-21T20:45:10.987+0000 o.a.z.ZooKeeper [INFO] Client
environment:os.version=3.13.0-48-generic
2015-05-21T20:45:10.987+0000 o.a.z.ZooKeeper [INFO] Client environment:
user.name=ubuntu
2015-05-21T20:45:10.987+0000 o.a.z.ZooKeeper [INFO] Client
environment:user.home=/home/ubuntu
2015-05-21T20:45:10.987+0000 o.a.z.ZooKeeper [INFO] Client
environment:user.dir=/
2015-05-21T20:45:10.990+0000 o.a.z.ZooKeeper [INFO] Initiating client
connection, connectString=52.7.165.232:2181 sessionTimeout=100000
watcher=gr.katsip.synefo.storm.operators.crypstream.DataCollector$1@4f91912f
2015-05-21T20:45:11.024+0000 o.a.z.ClientCnxn [INFO] Opening socket
connection to server 52.7.165.232/52.7.165.232:2181. Will not attempt to
authenticate using SASL (unknown error)
2015-05-21T20:45:11.030+0000 o.a.z.ClientCnxn [INFO] Socket connection
established to 52.7.165.232/52.7.165.232:2181, initiating session
2015-05-21T20:45:11.035+0000 o.a.z.ClientCnxn [INFO] Session establishment
complete on server 52.7.165.232/52.7.165.232:2181, sessionid =
0x14d72688cda0222, negotiated timeout = 40000
2015-05-21T20:49:06.916+0000 b.s.m.n.StormClientErrorHandler [INFO]
Connection failed Netty-Client-ip-172-31-10-201.ec2.internal/
172.31.10.201:6703
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.read0(Native Method) ~[na:1.7.0_79]
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
~[na:1.7.0_79]
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
~[na:1.7.0_79]
at sun.nio.ch.IOUtil.read(IOUtil.java:192) ~[na:1.7.0_79]
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379)
~[na:1.7.0_79]
at
org.apache.storm.netty.channel.socket.nio.NioWorker.read(NioWorker.java:64)
[storm-core-0.9.4.jar:0.9.4]
at
org.apache.storm.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
[storm-core-0.9.4.jar:0.9.4]
at
org.apache.storm.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318)
[storm-core-0.9.4.jar:0.9.4]
at
org.apache.storm.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
[storm-core-0.9.4.jar:0.9.4]
at
org.apache.storm.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
[storm-core-0.9.4.jar:0.9.4]
at
org.apache.storm.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
[storm-core-0.9.4.jar:0.9.4]
at
org.apache.storm.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
[storm-core-0.9.4.jar:0.9.4]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[na:1.7.0_79]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[na:1.7.0_79]
at java.lang.Thread.run(Thread.java:745) [na:1.7.0_79]
2015-05-21T20:49:06.918+0000 b.s.m.n.Client [ERROR] failed to send 91
messages to Netty-Client-ip-172-31-10-201.ec2.internal/172.31.10.201:6703:
java.nio.channels.ClosedChannelException
2015-05-21T20:49:06.919+0000 b.s.m.n.Client [INFO] connection attempt 1 to
Netty-Client-ip-172-31-10-201.ec2.internal/172.31.10.201:6703 scheduled to
run in 0 ms
2015-05-21T20:49:06.920+0000 b.s.m.n.Client [ERROR] failed to send 56
messages to Netty-Client-ip-172-31-10-201.ec2.internal/172.31.10.201:6703:
java.nio.channels.ClosedChannelException
2015-05-21T20:49:06.920+0000 b.s.m.n.Client [ERROR] failed to send 7
messages to Netty-Client-ip-172-31-10-201.ec2.internal/172.31.10.201:6703:
java.nio.channels.ClosedChannelException
2015-05-21T20:49:06.920+0000 b.s.m.n.Client [ERROR] failed to send 1
messages to Netty-Client-ip-172-31-10-201.ec2.internal/172.31.10.201:6703:
java.nio.channels.ClosedChannelException
2015-05-21T20:49:06.921+0000 b.s.m.n.Client [ERROR] connection attempt 1 to
Netty-Client-ip-172-31-10-201.ec2.internal/172.31.10.201:6703 failed:
java.lang.RuntimeException: Returned channel was actually not established
2015-05-21T20:49:06.920+0000 b.s.m.n.Client [ERROR] connection to
Netty-Client-ip-172-31-10-201.ec2.internal/172.31.10.201:6703 is unavailable
2015-05-21T20:49:06.921+0000 b.s.m.n.Client [ERROR] dropping 1 message(s)
destined for Netty-Client-ip-172-31-10-201.ec2.internal/172.31.10.201:6703
2015-05-21T20:49:06.921+0000 b.s.m.n.Client [ERROR] connection to
Netty-Client-ip-172-31-10-201.ec2.internal/172.31.10.201:6703 is unavailable
2015-05-21T20:49:06.921+0000 b.s.m.n.Client [ERROR] dropping 5 message(s)
destined for Netty-Client-ip-172-31-10-201.ec2.internal/172.31.10.201:6703
2015-05-21T20:49:06.921+0000 b.s.m.n.Client [INFO] connection attempt 2 to
Netty-Client-ip-172-31-10-201.ec2.internal/172.31.10.201:6703 scheduled to
run in 103 ms
2015-05-21T20:49:06.922+0000 b.s.m.n.Client [ERROR] connection to
Netty-Client-ip-172-31-10-201.ec2.internal/172.31.10.201:6703 is unavailable
2015-05-21T20:49:06.922+0000 b.s.m.n.Client [ERROR] dropping 1 message(s)
destined for Netty-Client-ip-172-31-10-201.ec2.internal/172.31.10.201:6703
2015-05-21T20:49:06.922+0000 b.s.m.n.Client [ERROR] connection to
Netty-Client-ip-172-31-10-201.ec2.internal/172.31.10.201:6703 is unavailable
2015-05-21T20:49:06.922+0000 b.s.m.n.Client [ERROR] dropping 1 message(s)
destined for Netty-Client-ip-172-31-10-201.ec2.internal/172.31.10.201:6703
2015-05-21T20:49:06.922+0000 b.s.m.n.Client [ERROR] connection to
Netty-Client-ip-172-31-10-201.ec2.internal/172.31.10.201:6703 is unavailable
2015-05-21T20:49:06.922+0000 b.s.m.n.Client [ERROR] dropping 1 message(s)
destined for Netty-Client-ip-172-31-10-201.ec2.internal/172.31.10.201:6703
2015-05-21T20:49:06.921+0000 b.s.m.n.Client [ERROR] failed to send 1
messages to Netty-Client-ip-172-31-10-201.ec2.internal/172.31.10.201:6703:
java.nio.channels.ClosedChannelException
2015-05-21T20:49:06.922+0000 b.s.m.n.Client [ERROR] failed to send 1
messages to Netty-Client-ip-172-31-10-201.ec2.internal/172.31.10.201:6703:
java.nio.channels.ClosedChannelException
2015-05-21T20:49:06.922+0000 b.s.m.n.Client [ERROR] connection to
Netty-Client-ip-172-31-10-201.ec2.internal/172.31.10.201:6703 is unavailable
2015-05-21T20:49:06.923+0000 b.s.m.n.Client [ERROR] dropping 1 message(s)
destined for Netty-Client-ip-172-31-10-201.ec2.internal/172.31.10.201:6703
2015-05-21T20:49:06.923+0000 b.s.m.n.Client [ERROR] connection to
Netty-Client-ip-172-31-10-201.ec2.internal/172.31.10.201:6703 is unavailable
2015-05-21T20:49:06.923+0000 b.s.m.n.Client [ERROR] dropping 42 message(s)
destined for Netty-Client-ip-172-31-10-201.ec2.internal/172.31.10.201:6703
2015-05-21T20:49:06.923+0000 b.s.m.n.Client [ERROR] failed to send 1
messages to Netty-Client-ip-172-31-10-201.ec2.internal/172.31.10.201:6703:
java.nio.channels.ClosedChannelException
2015-05-21T20:49:06.923+0000 b.s.m.n.Client [ERROR] failed to send 1
messages to Netty-Client-ip-172-31-10-201.ec2.internal/172.31.10.201:6703:
java.nio.channels.ClosedChannelException
2015-05-21T20:49:06.923+0000 b.s.m.n.Client [ERROR] connection to
Netty-Client-ip-172-31-10-201.ec2.internal/172.31.10.201:6703 is unavailable
2015-05-21T20:49:06.923+0000 b.s.m.n.Client [ERROR] dropping 1 message(s)
destined for Netty-Client-ip-172-31-10-201.ec2.internal/172.31.10.201:6703
2015-05-21T20:49:06.923+0000 b.s.m.n.Client [ERROR] connection to
Netty-Client-ip-172-31-10-201.ec2.internal/172.31.10.201:6703 is unavailable
2015-05-21T20:49:06.923+0000 b.s.m.n.Client [ERROR] dropping 1 message(s)
destined for Netty-Client-ip-172-31-10-201.ec2.internal/172.31.10.201:6703
2015-05-21T20:49:06.923+0000 b.s.m.n.Client [ERROR] connection to
Netty-Client-ip-172-31-10-201.ec2.internal/172.31.10.201:6703 is unavailable
2015-05-21T20:49:06.923+0000 b.s.m.n.Client [ERROR] dropping 1 message(s)
destined for Netty-Client-ip-172-31-10-201.ec2.internal/172.31.10.201:6703
2015-05-21T20:49:06.924+0000 b.s.m.n.Client [ERROR] connection to
Netty-Client-ip-172-31-10-201.ec2.internal/172.31.10.201:6703 is unavailable
2015-05-21T20:49:06.924+0000 b.s.m.n.Client [ERROR] dropping 1 message(s)
destined for Netty-Client-ip-172-31-10-201.ec2.internal/172.31.10.201:6703
2015-05-21T20:49:06.924+0000 b.s.m.n.Client [ERROR] connection to
Netty-Client-ip-172-31-10-201.ec2.internal/172.31.10.201:6703 is unavailable
2015-05-21T20:49:06.924+0000 b.s.m.n.Client [ERROR] dropping 1 message(s)
destined for Netty-Client-ip-172-31-10-201.ec2.internal/172.31.10.201:6703
2015-05-21T20:49:06.924+0000 b.s.m.n.Client [ERROR] connection to
Netty-Client-ip-172-31-10-201.ec2.internal/172.31.10.201:6703 is unavailable
2015-05-21T20:49:06.924+0000 b.s.m.n.Client [ERROR] dropping 1 message(s)
destined for Netty-Client-ip-172-31-10-201.ec2.internal/172.31.10.201:6703
2015-05-21T20:49:06.924+0000 b.s.m.n.Client [ERROR] connection to
Netty-Client-ip-172-31-10-201.ec2.internal/172.31.10.201:6703 is unavailable
2015-05-21T20:49:06.924+0000 b.s.m.n.Client [ERROR] dropping 1 message(s)
destined for Netty-Client-ip-172-31-10-201.ec2.internal/172.31.10.201:6703
2015-05-21T20:49:06.924+0000 b.s.m.n.Client [ERROR] connection to
Netty-Client-ip-172-31-10-201.ec2.internal/172.31.10.201:6703 is unavailable
2015-05-21T20:49:06.924+0000 b.s.m.n.Client [ERROR] dropping 1 message(s)
destined for Netty-Client-ip-172-31-10-201.ec2.internal/172.31.10.201:6703
2015-05-21T20:49:06.924+0000 b.s.m.n.Client [ERROR] connection to
Netty-Client-ip-172-31-10-201.ec2.internal/172.31.10.201:6703 is unavailable
2015-05-21T20:49:06.924+0000 b.s.m.n.Client [ERROR] dropping 1 message(s)
destined for Netty-Client-ip-172-31-10-201.ec2.internal/172.31.10.201:6703
2015-05-21T20:49:06.924+0000 b.s.m.n.Client [ERROR] connection to
Netty-Client-ip-172-31-10-201.ec2.internal/172.31.10.201:6703 is unavailable
2015-05-21T20:49:06.924+0000 b.s.m.n.Client [ERROR] dropping 1 message(s)
destined for Netty-Client-ip-172-31-10-201.ec2.internal/172.31.10.201:6703
2015-05-21T20:49:06.924+0000 b.s.m.n.Client [ERROR] connection to
Netty-Client-ip-172-31-10-201.ec2.internal/172.31.10.201:6703 is unavailable
2015-05-21T20:49:06.925+0000 b.s.m.n.Client [ERROR] dropping 1 message(s)
destined for Netty-Client-ip-172-31-10-201.ec2.internal/172.31.10.201:6703
2015-05-21T20:49:06.925+0000 b.s.m.n.Client [ERROR] connection to
Netty-Client-ip-172-31-10-201.ec2.internal/172.31.10.201:6703 is unavailable
2015-05-21T20:49:06.925+0000 b.s.m.n.Client [ERROR] dropping 1 message(s)
destined for Netty-Client-ip-172-31-10-201.ec2.internal/172.31.10.201:6703
2015-05-21T20:49:06.925+0000 b.s.m.n.Client [ERROR] connection to
Netty-Client-ip-172-31-10-201.ec2.internal/172.31.10.201:6703 is unavailable
2015-05-21T20:49:06.925+0000 b.s.m.n.Client [ERROR] dropping 1 message(s)
destined for Netty-Client-ip-172-31-10-201.ec2.internal/172.31.10.201:6703
And the above message keeps getting printed in my log files until the bolt
dies and comes back again.
It is strange to me, because this error happens after the topology has been
executing for quite a while and I do not get any indication on what I might
be doing wrong (at least in my code). Any suggestions/ideas/thoughts? Do
you think I might have my network packets dropped because I am using small
instances? I am asking the previous, because I do not see any other reason
why the worker might be unavailable (Netty-Client closed channel).
By the way, my topologies are built using the following dependencies:
<dependency>
<groupId>org.apache.storm</groupId>
<artifactId>storm-core</artifactId>
<version>0.9.3</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>3.8.1</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>commons-codec</groupId>
<artifactId>commons-codec</artifactId>
<version>1.10</version>
</dependency>
<dependency>
<groupId>org.apache.zookeeper</groupId>
<artifactId>zookeeper</artifactId>
<version>3.4.6</version>
</dependency>
<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-core</artifactId>
<version>2.1</version>
</dependency>
<dependency>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.2</version>
</dependency>
<dependency>
<groupId>mysql</groupId>
<artifactId>mysql-connector-java</artifactId>
<version>5.1.35</version>
</dependency>
Sorry for the long email. I hope someone has an idea of what is going wrong.
Thank you,
Nick
--
Nikolaos Romanos Katsipoulakis,
University of Pittsburgh, PhD candidate
