I'm going to use thrift, avro or protobuf for serialization. Russell Jurney http://datasyndrome.com
On Mar 21, 2012, at 11:59 AM, Vaibhav Puranik <vpura...@gmail.com> wrote: > I would use the payload. I want the message to be exactly as it is. We want > to name the files as per topic. > (That's how we differentiate right now). > > Regards, > Vaibhav > > On Wed, Mar 21, 2012 at 11:01 AM, Niek Sanders <niek.sand...@gmail.com>wrote: > >> So what would you like the S3 files to actually look like? >> >> One Kafka message body per line? Should the message topic be tossed >> in there too? >> >> A tricky aspect is that the Kafka message body is an opaque byte >> array. For my own case I'm using JSON for the payload so it makes my >> requirements simpler. >> >> - Niek >> >> >> >> On Tue, Mar 20, 2012 at 10:07 PM, Russell Jurney >> <russell.jur...@gmail.com> wrote: >>> I want events in S3 to process them in Hadoop. I'd like to emit them in >> my app, and have them magically show up in 64MB chunks on S3. Like most >> everyone else. >>> >>> Russell Jurney http://datasyndrome.com >>> >>
