In storm documentation ( https://github.com/nathanmarz/storm/wiki/Running-topologies-on-a-production-cluster) under the topic "Updating a running topology" there is a proposed "storm swap" command. But its still in pending for quiet some time now.
Is there any timeline for its implementation in the community? On Sun, Feb 16, 2014 at 7:24 PM, Chitra Raveendran < [email protected]> wrote: > Hi > > Replaying messages would be very inefficient for our use case as our > application computes realtime counts. Right now the value is set at -1. > > The timestamp option would lead to recounting if set at an earlier time. > This still would not be very efficient where counts are critical. > > I wanted to know if there was any another option. > > Thanks a lot for the help. > On Feb 16, 2014 11:32 PM, "Vinoth Kumar Kannan" <[email protected]> > wrote: > >> If you don't want to replay all the messages from the beginning then >> using the timestamp is the ideal choice. As far my understanding goes there >> is no exact way to pass the value of your last topology's undeployment time >> to your new topology. So to make sure you dont loose any messages specify a >> time stamp thats a bit earlier to the time which you undeployed your last >> topology. >> >> >> >> On Sun, Feb 16, 2014 at 6:35 PM, Chitra Raveendran < >> [email protected]> wrote: >> >>> Hi >>> >>> So according to this logic I should set the timestamp parameter to the >>> value when the topology was stopped ? >>> >>> But how do we identify the exact instance when the topology went down, >>> so that storm could start consuming from then ? Is it based on >>> approximation ? Or is there some concrete way to find the exact instance >>> when the topology was down? >>> >>> Is there any other parameter which is based on last offset and not time? >>> >>> Regards >>> Chitra >>> On Feb 16, 2014 11:00 PM, "Vinoth Kumar Kannan" <[email protected]> >>> wrote: >>> >>>> forceStartOffsetTime value can be -2, -1, or a time stamp in >>>> milliseconds >>>> >>>> >>>> - -1 to read the latest offset of the topic >>>> - -2 to read from the beginning. >>>> - timestamp to read from a specific time >>>> >>>> >>>> >>>> On Sun, Feb 16, 2014 at 6:15 PM, Chitra Raveendran < >>>> [email protected]> wrote: >>>> >>>>> Any body ?? Any answers !!! >>>>> >>>>> I'm sure someone would have a work around ! >>>>> >>>>> Please help :) >>>>> On Feb 15, 2014 12:11 AM, "Andrey Yegorov" <[email protected]> >>>>> wrote: >>>>> >>>>>> I have exactly the same question. >>>>>> I am using kafka spout from >>>>>> https://github.com/wurstmeister/storm-kafka-0.8-plus.git with kafka >>>>>> 0.8 release and ordinary (non-trident) storm topology. >>>>>> >>>>>> How can I guarantee processing of messages sent while topology was >>>>>> down or while e.g. storm cluster was down for maintenance? >>>>>> >>>>>> ---------- >>>>>> Andrey Yegorov >>>>>> >>>>>> >>>>>> On Wed, Feb 12, 2014 at 8:05 AM, Danijel Schiavuzzi < >>>>>> [email protected]> wrote: >>>>>> >>>>>>> Hi Chitra, >>>>>>> >>>>>>> Which Kafka spout version are you exactly using, and what spout type >>>>>>> -- Trident or the ordinary Storm spout? >>>>>>> >>>>>>> I ask that because, unfortunately, there are multiple Kafka spout >>>>>>> versions around the web. According to my research, your best bet is the >>>>>>> one >>>>>>> in storm-contrib in case you use Kafka version 0.7, and >>>>>>> storm-kafka-0.8-plus in case you use Kafka 0.8. >>>>>>> >>>>>>> Best regards, >>>>>>> >>>>>>> Danijel Schiavuzzi >>>>>>> www.schiavuzzi.com >>>>>>> >>>>>>> >>>>>>> On Wed, Feb 12, 2014 at 8:42 AM, Chitra Raveendran < >>>>>>> [email protected]> wrote: >>>>>>> >>>>>>>> Hi >>>>>>>> >>>>>>>> I have a topology in production which uses the default kafka spout, >>>>>>>> I have set this parameter >>>>>>>> *spoutConfig.forceStartOffsetTime(-1);* >>>>>>>> >>>>>>>> This parameter -1 helps me in such a way that, it consumes from the >>>>>>>> latest message, and doesn't start reading data from kafka right from >>>>>>>> the >>>>>>>> beginning (That would be unnecessary and redundant in my usecase). >>>>>>>> >>>>>>>> But in production, whenever a new release goes in, I stop and start >>>>>>>> the topology which would take a few seconds to minutes. I have been >>>>>>>> loosing >>>>>>>> out on some data during the time that the topology is down. >>>>>>>> >>>>>>>> How can I avoid this. I have tried running without the >>>>>>>> ForcedOffsetTime parameter, but that did not work. What am I doing >>>>>>>> wrong, >>>>>>>> how can I continue reading from the last offest ? >>>>>>>> >>>>>>>> Thanks, >>>>>>>> Chitra >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Danijel Schiavuzzi >>>>>>> >>>>>> >>>>>> >>>> >>>> >>>> -- >>>> With Regards, >>>> Vinoth Kumar K >>>> >>> >> >> >> -- >> With Regards, >> Vinoth Kumar K >> > -- With Regards, Vinoth Kumar K
