By setting forceFromStart to true, aren't I telling it to start from the beginning or earliest time then ?
On Thu, May 29, 2014 at 12:59 AM, Danijel Schiavuzzi <[email protected] > wrote: > You must set both forceFromStart to true and startOffsetTime to -1 or -2. > > > On Thu, May 29, 2014 at 12:23 AM, Raphael Hsieh <[email protected]> > wrote: > >> I'm doing both tridentKafkaConfig.forceFromStart = false; as well as >> tridentKafkaConfig.startOffsetTime = -1; >> >> Neither are working for me. Looking at my nimbus UI, I still get a large >> spike in processed data, before it levels off and seems to not process >> anything. >> >> >> >> On Wed, May 28, 2014 at 3:17 PM, Shaikh Riyaz <[email protected]> >> wrote: >> >>> I think you can use kafkaConfig.forceFromStart = *false*; >>> >>> We have implemented this and its working fine. >>> >>> Regards, >>> Riyaz >>> >>> >>> >>> On Thu, May 29, 2014 at 1:02 AM, Raphael Hsieh <[email protected]> >>> wrote: >>> >>>> This is still not working for me. I've set the offset to -1 and it is >>>> still backfilling data. >>>> Is there any documentation on the start offsets that I could take a >>>> look at ? >>>> Or even documentation on kafka.api.OffsetRequest.LatestTime() ? >>>> >>>> >>>> On Wed, May 28, 2014 at 1:01 PM, Raphael Hsieh <[email protected]> >>>> wrote: >>>> >>>>> would the Trident version of this be >>>>> tridentKafkaConfig.startOffsetTime ? >>>>> >>>>> >>>>> On Wed, May 28, 2014 at 12:23 PM, Danijel Schiavuzzi < >>>>> [email protected]> wrote: >>>>> >>>>>> By default, the Kafka spout resumes consuming where it last left off. >>>>>> That offset is stored in ZooKeeper. >>>>>> >>>>>> You can set forceStartOffset to -2 to start consuming from the >>>>>> earliest available offset, or -1 to start consuming from the latest >>>>>> available offset. >>>>>> >>>>>> >>>>>> On Wednesday, May 28, 2014, Raphael Hsieh <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> If I don't tell trident to start consuming data from the beginning >>>>>>> of the Kafka stream, where does it start from? >>>>>>> If I were to do: >>>>>>> tridentKafkaConfig.forceFromStart = true; >>>>>>> Then it will tell the spout to start consuming from the start of the >>>>>>> stream. If that is not set, then where does it start consuming from? and >>>>>>> How might I go about telling it to start consuming from the very end of >>>>>>> the >>>>>>> stream? >>>>>>> >>>>>>> If a disaster were to happen and all my hosts died, when I start my >>>>>>> cluster back up, it might start consuming from where it left off. I >>>>>>> would >>>>>>> rather manually process that old data, and have my storm system start >>>>>>> processing the live data. >>>>>>> >>>>>>> Thanks >>>>>>> -- >>>>>>> Raphael Hsieh >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Danijel Schiavuzzi >>>>>> >>>>>> E: [email protected] >>>>>> W: www.schiavuzzi.com >>>>>> T: +385989035562 >>>>>> Skype: danijels7 >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> Raphael Hsieh >>>>> Amazon.com >>>>> Software Development Engineer I >>>>> (978) 764-9014 >>>>> >>>>> >>>>> >>>>> >>>> >>>> >>>> >>>> -- >>>> Raphael Hsieh >>>> Amazon.com >>>> Software Development Engineer I >>>> (978) 764-9014 >>>> >>>> >>>> >>>> >>> >>> >>> >>> -- >>> Regards, >>> >>> Riyaz >>> >>> >> >> >> -- >> Raphael Hsieh >> >> >> > > > > -- > Danijel Schiavuzzi > > E: [email protected] > W: www.schiavuzzi.com > T: +385989035562 > Skype: danijels7 > -- Raphael Hsieh Amazon.com Software Development Engineer I (978) 764-9014
