Russell, One way to do this, is use the ConsoleConsumer and plug in your formatter. You can write your formatter to optionally deserialize the message into an object, and either write it to S3 immediately, or buffer and then write to S3. If you go with this approach, the console consumer consumes data as fast as you can write to S3.
Thanks, Neha On Thu, Dec 15, 2011 at 6:04 PM, Russell Jurney <russell.jur...@gmail.com> wrote: > Jun, thanks! Would you say that writing a Kafka consumer that pipes > to S3 is a little code, or a lot of code? > > Russell Jurney > twitter.com/rjurney > russell.jur...@gmail.com > datasyndrome.com > > On Dec 15, 2011, at 5:39 PM, Jun Rao <jun...@gmail.com> wrote: > >> Russell, >> >> We do have Ruby/Python clients under contrib (may not be well supported). >> You will need to write a Kafka consumer that pipes to S3. >> >> Thanks, >> >> Jun >> >> On Thu, Dec 15, 2011 at 5:23 PM, Russell Jurney >> <russell.jur...@gmail.com>wrote: >> >>> Can Kafka do this, without writing a lot of code? >>> >>> Russell Jurney >>> twitter.com/rjurney >>> russell.jur...@gmail.com >>> datasyndrome.com >>>
