If you don't want to replay all the messages from the beginning then using the timestamp is the ideal choice. As far my understanding goes there is no exact way to pass the value of your last topology's undeployment time to your new topology. So to make sure you dont loose any messages specify a time stamp thats a bit earlier to the time which you undeployed your last topology.
On Sun, Feb 16, 2014 at 6:35 PM, Chitra Raveendran < [email protected]> wrote: > Hi > > So according to this logic I should set the timestamp parameter to the > value when the topology was stopped ? > > But how do we identify the exact instance when the topology went down, so > that storm could start consuming from then ? Is it based on approximation ? > Or is there some concrete way to find the exact instance when the topology > was down? > > Is there any other parameter which is based on last offset and not time? > > Regards > Chitra > On Feb 16, 2014 11:00 PM, "Vinoth Kumar Kannan" <[email protected]> > wrote: > >> forceStartOffsetTime value can be -2, -1, or a time stamp in >> milliseconds >> >> >> - -1 to read the latest offset of the topic >> - -2 to read from the beginning. >> - timestamp to read from a specific time >> >> >> >> On Sun, Feb 16, 2014 at 6:15 PM, Chitra Raveendran < >> [email protected]> wrote: >> >>> Any body ?? Any answers !!! >>> >>> I'm sure someone would have a work around ! >>> >>> Please help :) >>> On Feb 15, 2014 12:11 AM, "Andrey Yegorov" <[email protected]> >>> wrote: >>> >>>> I have exactly the same question. >>>> I am using kafka spout from >>>> https://github.com/wurstmeister/storm-kafka-0.8-plus.git with kafka >>>> 0.8 release and ordinary (non-trident) storm topology. >>>> >>>> How can I guarantee processing of messages sent while topology was down >>>> or while e.g. storm cluster was down for maintenance? >>>> >>>> ---------- >>>> Andrey Yegorov >>>> >>>> >>>> On Wed, Feb 12, 2014 at 8:05 AM, Danijel Schiavuzzi <[email protected] >>>> > wrote: >>>> >>>>> Hi Chitra, >>>>> >>>>> Which Kafka spout version are you exactly using, and what spout type >>>>> -- Trident or the ordinary Storm spout? >>>>> >>>>> I ask that because, unfortunately, there are multiple Kafka spout >>>>> versions around the web. According to my research, your best bet is the >>>>> one >>>>> in storm-contrib in case you use Kafka version 0.7, and >>>>> storm-kafka-0.8-plus in case you use Kafka 0.8. >>>>> >>>>> Best regards, >>>>> >>>>> Danijel Schiavuzzi >>>>> www.schiavuzzi.com >>>>> >>>>> >>>>> On Wed, Feb 12, 2014 at 8:42 AM, Chitra Raveendran < >>>>> [email protected]> wrote: >>>>> >>>>>> Hi >>>>>> >>>>>> I have a topology in production which uses the default kafka spout, I >>>>>> have set this parameter >>>>>> *spoutConfig.forceStartOffsetTime(-1);* >>>>>> >>>>>> This parameter -1 helps me in such a way that, it consumes from the >>>>>> latest message, and doesn't start reading data from kafka right from the >>>>>> beginning (That would be unnecessary and redundant in my usecase). >>>>>> >>>>>> But in production, whenever a new release goes in, I stop and start >>>>>> the topology which would take a few seconds to minutes. I have been >>>>>> loosing >>>>>> out on some data during the time that the topology is down. >>>>>> >>>>>> How can I avoid this. I have tried running without the >>>>>> ForcedOffsetTime parameter, but that did not work. What am I doing wrong, >>>>>> how can I continue reading from the last offest ? >>>>>> >>>>>> Thanks, >>>>>> Chitra >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> Danijel Schiavuzzi >>>>> >>>> >>>> >> >> >> -- >> With Regards, >> Vinoth Kumar K >> > -- With Regards, Vinoth Kumar K
