So what would you like the S3 files to actually look like? One Kafka message body per line? Should the message topic be tossed in there too?
A tricky aspect is that the Kafka message body is an opaque byte array. For my own case I'm using JSON for the payload so it makes my requirements simpler. - Niek On Tue, Mar 20, 2012 at 10:07 PM, Russell Jurney <russell.jur...@gmail.com> wrote: > I want events in S3 to process them in Hadoop. I'd like to emit them in my > app, and have them magically show up in 64MB chunks on S3. Like most everyone > else. > > Russell Jurney http://datasyndrome.com >
