another good option would be messagepack -- flexible & schemaless like json, but binary.
Sent from my iPhone On 21 Mar 2012, at 20:46, Russell Jurney <russell.jur...@gmail.com> wrote: > I'm going to use thrift, avro or protobuf for serialization. > > Russell Jurney http://datasyndrome.com > > On Mar 21, 2012, at 11:59 AM, Vaibhav Puranik <vpura...@gmail.com> wrote: > >> I would use the payload. I want the message to be exactly as it is. We want >> to name the files as per topic. >> (That's how we differentiate right now). >> >> Regards, >> Vaibhav >> >> On Wed, Mar 21, 2012 at 11:01 AM, Niek Sanders <niek.sand...@gmail.com>wrote: >> >>> So what would you like the S3 files to actually look like? >>> >>> One Kafka message body per line? Should the message topic be tossed >>> in there too? >>> >>> A tricky aspect is that the Kafka message body is an opaque byte >>> array. For my own case I'm using JSON for the payload so it makes my >>> requirements simpler. >>> >>> - Niek >>> >>> >>> >>> On Tue, Mar 20, 2012 at 10:07 PM, Russell Jurney >>> <russell.jur...@gmail.com> wrote: >>>> I want events in S3 to process them in Hadoop. I'd like to emit them in >>> my app, and have them magically show up in 64MB chunks on S3. Like most >>> everyone else. >>>> >>>> Russell Jurney http://datasyndrome.com >>>> >>>
