Well I will have a look into it.
I know it is kind of conflict of interest to use storm to batch data. But S3 does not support individual message appending so needs to be batched, persisted locally and then bulk uploaded. I am just trying to explore if it is possible as we have pretty stable kafka-storm set up. On Wed, May 11, 2016 at 12:17 PM, saiprasad mishra < [email protected]> wrote: > I think you are looking for MessageMetadataSchemeAsMultiScheme to be used > by the kafka spout config which got added in 1.0.0 i guess > Regards > Sai > > On Wed, May 11, 2016 at 11:10 AM, Nathan Leung <[email protected]> wrote: > >> Why not just ack the tuple once it's been written to a file. If your >> topology fails then the data will be re-read from Kafka. Kafka spout >> already does this for you. Then uploading files to S3 is the >> responsibility of another job. For example, a storm topology that monitors >> the output folder. >> >> Monitoring the data from Kafka all the way out to S3 seems unnecessary. >> >> On Wed, May 11, 2016 at 1:50 PM, Milind Vaidya <[email protected]> wrote: >> >>> It does not matter, in the sense I am ready to upgrade if this thing is >>> in the roadmap. >>> >>> None the less >>> >>> kafka_2.9.2-0.8.1.1 apache-storm-0.9.4 >>> >>> >>> >>> >>> On Wed, May 11, 2016 at 5:53 AM, Abhishek Agarwal <[email protected]> >>> wrote: >>> >>>> which version of storm-kafka, are you using? >>>> >>>> On Wed, May 11, 2016 at 12:29 AM, Milind Vaidya <[email protected]> >>>> wrote: >>>> >>>>> Anybody ? Anything about this ? >>>>> >>>>> On Wed, May 4, 2016 at 11:31 AM, Milind Vaidya <[email protected]> >>>>> wrote: >>>>> >>>>>> Is there any way I can know what Kafka offset corresponds to current >>>>>> tuple I am processing in a bolt ? >>>>>> >>>>>> Use case : Need to batch events from Kafka, persists them to a local >>>>>> file and eventually upload it to the S3. To manager failure cases, need >>>>>> to >>>>>> know the Kafka offset for a message, so that it can be persisted to >>>>>> Zookeeper and will be used to write / upload file. >>>>>> >>>>>> >>>>> >>>> >>>> >>>> -- >>>> Regards, >>>> Abhishek Agarwal >>>> >>>> >>> >> >
