Hi Murtaza, >> - Is there any sample code around this since this is probably a common >> use-case. I meant is there a CustomAvroEncoder which we can use out of the >> box or any chance this can also be open-sourced?
The encoding/decoding using Avro is pretty simple. We just use the BinaryEncoder with the Specific/Generic DatumWriter to write IndexedRecord objects. >> - In terms of internals - are we converting avro into byte stream and >> creating a Message Object and then writing to the queue, does this incur any >> overhead in your opinion? The overhead of serialization I've seen in production is ~0.05 ms per record. Thanks, Neha On Tue, Jun 26, 2012 at 10:07 PM, Murtaza Doctor <murt...@richrelevance.com> wrote: > Hello Folks, > > We are currently evaluating Kafka and had a few questions around the > Encoder functionality. > Our data is in avro format and we wish to send the data to the broker in > this format as well eventually write to HDFS. As documented, we do realize > that we need a Custom Encoder to achieve creation of the Message object. > > Questions we had: > - Is there any sample code around this since this is probably a common > use-case. I meant is there a CustomAvroEncoder which we can use out of the > box or any chance this can also be open-sourced? > - In terms of internals - are we converting avro into byte stream and > creating a Message Object and then writing to the queue, does this incur > any overhead in your opinion? > - Any best practices around this or how others would approach this problem? > > If there is any value we would definitely like to see this added to the > FAQs or even part of some sample code. > > Thanks, > murtaza > >
