Hi everyone,
We are having some issues with the Storm topology. The problem is that some
tuples are being lost somewhere in the topology. Just after the topology is
deployed, it goes pretty well, but after several hours it starts to loose a
significant amount of tuples.
>From what we've found out from the logs, the thing is that the tuples exit
one bolt/spout, and never enter the next bolt.
Here is some info about the topology:
- The version is 0.9.1, and netty is used as transport
- The spout is extending BaseRichSpout, and the bolts extend
BaseBasicBolt
- The spout is using Kestrel message queue
- The cluster consists of 2 nodes: zookeeper, nimbus and ui are running
on one node, and the workers run on another node. I am attaching the
content of the config files below. We have also tried running the workers
on another node (the same where nimbus and zookeeper are), and also on both
nodes, but the behavior is the same.
According to the Storm UI there are no Failed tuples. Can anybody give any
idea of what might be the reason of the tuples getting lost?
Thanks.
*Storm config (storm.yaml)*
(In case both nodes have workers running, the configuration is the same on
both nodes, just the "storm.local.hostname" parameter changes)
storm.zookeeper.servers:
- "zkserver1"
nimbus.host: "nimbusserver"
storm.local.dir: "/mnt/storm"
supervisor.slots.ports:
- 6700
- 6701
- 6702
- 6703
storm.local.hostname: "storm1server"
nimbus.childopts: "-Xmx1024m -Djava.net.preferIPv4Stack=true"
ui.childopts: "-Xmx768m -Djava.net.preferIPv4Stack=true"
supervisor.childopts: "-Xmx1024m -Djava.net.preferIPv4Stack=true"
worker.childopts: "-Xmx3548m -Djava.net.preferIPv4Stack=true"
storm.cluster.mode: "distributed"
storm.local.mode.zmq: false
storm.thrift.transport: "backtype.storm.security.auth.SimpleTransportPlugin"
storm.messaging.transport: "backtype.storm.messaging.netty.Context"
storm.messaging.netty.server_worker_threads: 1
storm.messaging.netty.client_worker_threads: 1
storm.messaging.netty.buffer_size: 5242880 #5MB buffer
storm.messaging.netty.max_retries: 30
storm.messaging.netty.max_wait_ms: 1000
storm.messaging.netty.min_wait_ms: 100
*Zookeeper config (zoo.cfg):*
tickTime=2000
initLimit=10
syncLimit=5
dataDir=/var/zookeeper
clientPort=2181
autopurge.purgeInterval=24
autopurge.snapRetainCount=5
server.1=localhost:2888:3888
*Topology configuration* passed to the StormSubmitter:
Config conf = new Config();
conf.setNumAckers(6);
conf.setNumWorkers(4);
conf.setMaxSpoutPending(100);
Best regards,
Daria Mayorova
