I am running into the same issue. Where do the lost tuples gone ? If they were queueing in the transport layer, the memory usage should keep increasing, but I didn't see any noticeable memory leaks.
Does storm have the guarantee all tuples sent from task A to task B will be received by task B ? Moreover, are they in order ? Can anybody give any idea on this issue 2014-04-02 20:56 GMT+08:00 Daria Mayorova <[email protected]>: > Hi everyone, > > We are having some issues with the Storm topology. The problem is that > some tuples are being lost somewhere in the topology. Just after the > topology is deployed, it goes pretty well, but after several hours it > starts to loose a significant amount of tuples. > > From what we've found out from the logs, the thing is that the tuples exit > one bolt/spout, and never enter the next bolt. > > Here is some info about the topology: > > - The version is 0.9.1, and netty is used as transport > - The spout is extending BaseRichSpout, and the bolts extend > BaseBasicBolt > - The spout is using Kestrel message queue > - The cluster consists of 2 nodes: zookeeper, nimbus and ui are > running on one node, and the workers run on another node. I am attaching > the content of the config files below. We have also tried running the > workers on another node (the same where nimbus and zookeeper are), and also > on both nodes, but the behavior is the same. > > According to the Storm UI there are no Failed tuples. Can anybody give any > idea of what might be the reason of the tuples getting lost? > > Thanks. > > *Storm config (storm.yaml)* > (In case both nodes have workers running, the configuration is the same on > both nodes, just the "storm.local.hostname" parameter changes) > > storm.zookeeper.servers: > - "zkserver1" > nimbus.host: "nimbusserver" > storm.local.dir: "/mnt/storm" > supervisor.slots.ports: > - 6700 > - 6701 > - 6702 > - 6703 > storm.local.hostname: "storm1server" > > nimbus.childopts: "-Xmx1024m -Djava.net.preferIPv4Stack=true" > ui.childopts: "-Xmx768m -Djava.net.preferIPv4Stack=true" > supervisor.childopts: "-Xmx1024m -Djava.net.preferIPv4Stack=true" > worker.childopts: "-Xmx3548m -Djava.net.preferIPv4Stack=true" > > storm.cluster.mode: "distributed" > storm.local.mode.zmq: false > storm.thrift.transport: > "backtype.storm.security.auth.SimpleTransportPlugin" > > storm.messaging.transport: "backtype.storm.messaging.netty.Context" > > storm.messaging.netty.server_worker_threads: 1 > storm.messaging.netty.client_worker_threads: 1 > storm.messaging.netty.buffer_size: 5242880 #5MB buffer > storm.messaging.netty.max_retries: 30 > storm.messaging.netty.max_wait_ms: 1000 > storm.messaging.netty.min_wait_ms: 100 > > *Zookeeper config (zoo.cfg):* > tickTime=2000 > initLimit=10 > syncLimit=5 > dataDir=/var/zookeeper > clientPort=2181 > autopurge.purgeInterval=24 > autopurge.snapRetainCount=5 > server.1=localhost:2888:3888 > > *Topology configuration* passed to the StormSubmitter: > Config conf = new Config(); > conf.setNumAckers(6); > conf.setNumWorkers(4); > conf.setMaxSpoutPending(100); > > > Best regards, > Daria Mayorova > -- ====================================================== Gvain Email: [email protected]
