I have exactly the same question. I am using kafka spout from https://github.com/wurstmeister/storm-kafka-0.8-plus.git with kafka 0.8 release and ordinary (non-trident) storm topology.
How can I guarantee processing of messages sent while topology was down or while e.g. storm cluster was down for maintenance? ---------- Andrey Yegorov On Wed, Feb 12, 2014 at 8:05 AM, Danijel Schiavuzzi <[email protected]>wrote: > Hi Chitra, > > Which Kafka spout version are you exactly using, and what spout type -- > Trident or the ordinary Storm spout? > > I ask that because, unfortunately, there are multiple Kafka spout versions > around the web. According to my research, your best bet is the one in > storm-contrib in case you use Kafka version 0.7, and storm-kafka-0.8-plus > in case you use Kafka 0.8. > > Best regards, > > Danijel Schiavuzzi > www.schiavuzzi.com > > > On Wed, Feb 12, 2014 at 8:42 AM, Chitra Raveendran < > [email protected]> wrote: > >> Hi >> >> I have a topology in production which uses the default kafka spout, I >> have set this parameter >> *spoutConfig.forceStartOffsetTime(-1);* >> >> This parameter -1 helps me in such a way that, it consumes from the >> latest message, and doesn't start reading data from kafka right from the >> beginning (That would be unnecessary and redundant in my usecase). >> >> But in production, whenever a new release goes in, I stop and start the >> topology which would take a few seconds to minutes. I have been loosing out >> on some data during the time that the topology is down. >> >> How can I avoid this. I have tried running without the ForcedOffsetTime >> parameter, but that did not work. What am I doing wrong, how can I continue >> reading from the last offest ? >> >> Thanks, >> Chitra >> >> > > > -- > Danijel Schiavuzzi >
