What would be needed to do this? Just thinking off the top of my head:
1. create a zookeeper store to keep track of the last message offset persisted to s3, and which messages each consumer is processing. 2. pull messages off and group in whatever grouping you want (per message, 10 messages, etc.), and spin off a executorservice to push to s3, update the zookeeper offset. I'm new to kafka, but I would have to investigate on how multiple consumers can pull messages and push to s3, while not having the consumers pull the same messages. Setting up a zookeeper store to track progress specifically for what has been pushed to s3. On Wed, May 23, 2012 at 1:35 AM, Russell Jurney <russell.jur...@gmail.com>wrote: > Yeah, no kidding. I keep waiting on one :) > > Russell Jurney http://datasyndrome.com > > On May 22, 2012, at 10:31 PM, Jay Kreps <jay.kr...@gmail.com> wrote: > > > No. Patches accepted. > > > > -Jay > > > > On Tue, May 22, 2012 at 10:23 PM, Russell Jurney > > <russell.jur...@gmail.com>wrote: > > > >> Is there a simple way to dump Kafka events to S3 yet? > >> > >> Russell Jurney http://datasyndrome.com > >> >
