Hi Replaying messages would be very inefficient for our use case as our application computes realtime counts. Right now the value is set at -1.
The timestamp option would lead to recounting if set at an earlier time. This still would not be very efficient where counts are critical. I wanted to know if there was any another option. Thanks a lot for the help. On Feb 16, 2014 11:32 PM, "Vinoth Kumar Kannan" <[email protected]> wrote: > If you don't want to replay all the messages from the beginning then using > the timestamp is the ideal choice. As far my understanding goes there is no > exact way to pass the value of your last topology's undeployment time to > your new topology. So to make sure you dont loose any messages specify a > time stamp thats a bit earlier to the time which you undeployed your last > topology. > > > > On Sun, Feb 16, 2014 at 6:35 PM, Chitra Raveendran < > [email protected]> wrote: > >> Hi >> >> So according to this logic I should set the timestamp parameter to the >> value when the topology was stopped ? >> >> But how do we identify the exact instance when the topology went down, so >> that storm could start consuming from then ? Is it based on approximation ? >> Or is there some concrete way to find the exact instance when the topology >> was down? >> >> Is there any other parameter which is based on last offset and not time? >> >> Regards >> Chitra >> On Feb 16, 2014 11:00 PM, "Vinoth Kumar Kannan" <[email protected]> >> wrote: >> >>> forceStartOffsetTime value can be -2, -1, or a time stamp in >>> milliseconds >>> >>> >>> - -1 to read the latest offset of the topic >>> - -2 to read from the beginning. >>> - timestamp to read from a specific time >>> >>> >>> >>> On Sun, Feb 16, 2014 at 6:15 PM, Chitra Raveendran < >>> [email protected]> wrote: >>> >>>> Any body ?? Any answers !!! >>>> >>>> I'm sure someone would have a work around ! >>>> >>>> Please help :) >>>> On Feb 15, 2014 12:11 AM, "Andrey Yegorov" <[email protected]> >>>> wrote: >>>> >>>>> I have exactly the same question. >>>>> I am using kafka spout from >>>>> https://github.com/wurstmeister/storm-kafka-0.8-plus.git with kafka >>>>> 0.8 release and ordinary (non-trident) storm topology. >>>>> >>>>> How can I guarantee processing of messages sent while topology was >>>>> down or while e.g. storm cluster was down for maintenance? >>>>> >>>>> ---------- >>>>> Andrey Yegorov >>>>> >>>>> >>>>> On Wed, Feb 12, 2014 at 8:05 AM, Danijel Schiavuzzi < >>>>> [email protected]> wrote: >>>>> >>>>>> Hi Chitra, >>>>>> >>>>>> Which Kafka spout version are you exactly using, and what spout type >>>>>> -- Trident or the ordinary Storm spout? >>>>>> >>>>>> I ask that because, unfortunately, there are multiple Kafka spout >>>>>> versions around the web. According to my research, your best bet is the >>>>>> one >>>>>> in storm-contrib in case you use Kafka version 0.7, and >>>>>> storm-kafka-0.8-plus in case you use Kafka 0.8. >>>>>> >>>>>> Best regards, >>>>>> >>>>>> Danijel Schiavuzzi >>>>>> www.schiavuzzi.com >>>>>> >>>>>> >>>>>> On Wed, Feb 12, 2014 at 8:42 AM, Chitra Raveendran < >>>>>> [email protected]> wrote: >>>>>> >>>>>>> Hi >>>>>>> >>>>>>> I have a topology in production which uses the default kafka spout, >>>>>>> I have set this parameter >>>>>>> *spoutConfig.forceStartOffsetTime(-1);* >>>>>>> >>>>>>> This parameter -1 helps me in such a way that, it consumes from the >>>>>>> latest message, and doesn't start reading data from kafka right from the >>>>>>> beginning (That would be unnecessary and redundant in my usecase). >>>>>>> >>>>>>> But in production, whenever a new release goes in, I stop and start >>>>>>> the topology which would take a few seconds to minutes. I have been >>>>>>> loosing >>>>>>> out on some data during the time that the topology is down. >>>>>>> >>>>>>> How can I avoid this. I have tried running without the >>>>>>> ForcedOffsetTime parameter, but that did not work. What am I doing >>>>>>> wrong, >>>>>>> how can I continue reading from the last offest ? >>>>>>> >>>>>>> Thanks, >>>>>>> Chitra >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Danijel Schiavuzzi >>>>>> >>>>> >>>>> >>> >>> >>> -- >>> With Regards, >>> Vinoth Kumar K >>> >> > > > -- > With Regards, > Vinoth Kumar K >
