Just adding on to my observation , forceStartOffsetTime accepts a timestamp value in milliseconds, but it seems to be working just like how it would have worked if I passed parameter -1. It is reading only from current offset and not from the input timestamp!
I guess the kafka spout I'm using only handles 2 conditions as input,ie. either start from beginning or start from next offset! It would be great if someone could help Thanks Chitra On Thu, Mar 6, 2014 at 3:34 PM, Chitra Raveendran < [email protected]> wrote: > Hi > > Sorry for the late reply, I just got time to experiment today and, > realized forceStartOffsetTime is not accepting timestamp(milli seconds) > value as a parameter. > > This doesn't seem to work. I'm using the kafka spout from storm-contrib, > and it is a normal storm topology not a trident topology!! > > Regards > Chitra > > > On Mon, Feb 17, 2014 at 1:26 AM, Chitra Raveendran < > [email protected]> wrote: > >> Yes I tried without the forced offset time parameter, but the topology >> stopped consuming messages. >> >> Regards >> Chitra >> On Feb 17, 2014 1:24 AM, "P. Taylor Goetz" <[email protected]> wrote: >> >>> If turn off forceOffsetTime, it should resume from the last offset >>> stored in zookeeper. >>> >>> - Taylor >>> >>> On Feb 16, 2014, at 12:35 PM, Chitra Raveendran < >>> [email protected]> wrote: >>> >>> Hi >>> >>> So according to this logic I should set the timestamp parameter to the >>> value when the topology was stopped ? >>> >>> But how do we identify the exact instance when the topology went down, >>> so that storm could start consuming from then ? Is it based on >>> approximation ? Or is there some concrete way to find the exact instance >>> when the topology was down? >>> >>> Is there any other parameter which is based on last offset and not time? >>> >>> Regards >>> Chitra >>> On Feb 16, 2014 11:00 PM, "Vinoth Kumar Kannan" <[email protected]> >>> wrote: >>> >>>> forceStartOffsetTime value can be -2, -1, or a time stamp in >>>> milliseconds >>>> >>>> >>>> - -1 to read the latest offset of the topic >>>> - -2 to read from the beginning. >>>> - timestamp to read from a specific time >>>> >>>> >>>> >>>> On Sun, Feb 16, 2014 at 6:15 PM, Chitra Raveendran < >>>> [email protected]> wrote: >>>> >>>>> Any body ?? Any answers !!! >>>>> >>>>> I'm sure someone would have a work around ! >>>>> >>>>> Please help :) >>>>> On Feb 15, 2014 12:11 AM, "Andrey Yegorov" <[email protected]> >>>>> wrote: >>>>> >>>>>> I have exactly the same question. >>>>>> I am using kafka spout from >>>>>> https://github.com/wurstmeister/storm-kafka-0.8-plus.git with kafka >>>>>> 0.8 release and ordinary (non-trident) storm topology. >>>>>> >>>>>> How can I guarantee processing of messages sent while topology was >>>>>> down or while e.g. storm cluster was down for maintenance? >>>>>> >>>>>> ---------- >>>>>> Andrey Yegorov >>>>>> >>>>>> >>>>>> On Wed, Feb 12, 2014 at 8:05 AM, Danijel Schiavuzzi < >>>>>> [email protected]> wrote: >>>>>> >>>>>>> Hi Chitra, >>>>>>> >>>>>>> Which Kafka spout version are you exactly using, and what spout type >>>>>>> -- Trident or the ordinary Storm spout? >>>>>>> >>>>>>> I ask that because, unfortunately, there are multiple Kafka spout >>>>>>> versions around the web. According to my research, your best bet is the >>>>>>> one >>>>>>> in storm-contrib in case you use Kafka version 0.7, and >>>>>>> storm-kafka-0.8-plus in case you use Kafka 0.8. >>>>>>> >>>>>>> Best regards, >>>>>>> >>>>>>> Danijel Schiavuzzi >>>>>>> www.schiavuzzi.com >>>>>>> >>>>>>> >>>>>>> On Wed, Feb 12, 2014 at 8:42 AM, Chitra Raveendran < >>>>>>> [email protected]> wrote: >>>>>>> >>>>>>>> Hi >>>>>>>> >>>>>>>> I have a topology in production which uses the default kafka spout, >>>>>>>> I have set this parameter >>>>>>>> *spoutConfig.forceStartOffsetTime(-1);* >>>>>>>> >>>>>>>> This parameter -1 helps me in such a way that, it consumes from the >>>>>>>> latest message, and doesn't start reading data from kafka right from >>>>>>>> the >>>>>>>> beginning (That would be unnecessary and redundant in my usecase). >>>>>>>> >>>>>>>> But in production, whenever a new release goes in, I stop and start >>>>>>>> the topology which would take a few seconds to minutes. I have been >>>>>>>> loosing >>>>>>>> out on some data during the time that the topology is down. >>>>>>>> >>>>>>>> How can I avoid this. I have tried running without the >>>>>>>> ForcedOffsetTime parameter, but that did not work. What am I doing >>>>>>>> wrong, >>>>>>>> how can I continue reading from the last offest ? >>>>>>>> >>>>>>>> Thanks, >>>>>>>> Chitra >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Danijel Schiavuzzi >>>>>>> >>>>>> >>>>>> >>>> >>>> >>>> -- >>>> With Regards, >>>> Vinoth Kumar K >>>> >>> >>> > > > -- Regards, *Chitra Raveendran* *Data Scientist* Mobile: +91 819753660│*Email:* [email protected] *Flutura Business Solutions Private Limited – “A Decision Sciences & Analytics Company”*│ #693, 2nd Floor, Geetanjali, 15th Cross, J.P Nagar 2nd Phase, Bangalore – 560078│
