> > But what is the suggested way of transferring encoded bytes over the > wire. Avro does not seem to expose the encoded bytes anywhere. >
I am using a custom method for transporting the encoding bytes, which is why I had to get at the bytes. Avro has an HTTP transport mechanisms defined, but I haven't used it and I don't know if it is fully implemented in the C++ project. For typical Avro usage you should not be accessing the raw bytes, which is why nobody bothered to expose them. So even if I am encoding a small dataset, I would end up > reading a min "Chunk_size" of memory block. > The "count" variable should be the exact number of bytes in the stream. Both for loops are checking "i<count", and "i" is only incremented in the inner loop. Once you reach the count it will break out of both loops. On Wed, May 16, 2012 at 4:45 AM, Gaurav Nanda <[email protected]> wrote: > Hi Matt, The approach you are using to read the written bytes would > end up reading all the bytes allocated but not the bytes actually > written. So even if I am encoding a small dataset, I would end up > reading a min "Chunk_size" of memory block. > > On Mon, May 14, 2012 at 9:03 PM, Gaurav Nanda <[email protected]> wrote: > > Thanks Matt ! > > > > But what is the suggested way of transferring encoded bytes over the > > wire. Avro does not seem to expose the encoded bytes anywhere. > > > > - Gaurav Nanda > > > > On Sat, May 12, 2012 at 6:29 AM, Matt Stevenson > > <[email protected]> wrote: > >> I made a copy of the MemoryOutputStream with a header, so you can get > at the > >> bytes. > >> > >> http://branchingworlds.com/avro/OpenMemoryOutputStream.h > >> http://branchingworlds.com/avro/OpenMemoryOutputStream.cpp > >> > >> Add those to your project. > >> include "OpenMemoryOutputStream.h" and call openMemoryOutputStream() > rather > >> than memoryOutputStream(). > >> > >> The memory is stored in chunks. Here's an example of iterating through > it: > >> > >>> auto_ptr<OpenMemoryOutputStream> os = openMemoryOutputStream(); > >>> EncoderPtr e = binaryEncoder(); > >>> e->init(*os); > >>> > >>> avro::encode(*e, *t); > >>> e->flush(); > >>> > >>> int count = os->byteCount(); > >>> char* data = new char[count]; > >>> int i=0; > >>> for (std::vector<uint8_t*>::const_iterator it = os->data_.begin(); it > != > >>> os->data_.end() && i<count; ++it) { > >>> uint8_t* chunk = *it; > >>> int size = os->chunkSize_; > >>> for(int j=0; j<size && i<count; j++, i++){ > >>> data[i] = chunk[j]; > >>> } > >>> } > >> > >> > >> On Thu, May 10, 2012 at 12:19 PM, Gaurav Nanda <[email protected]> > wrote: > >>> > >>> I want to extract byte array from memoryOutputStream to transfer the > >>> encoded data. How do I achieve that? > >> > >> > >> > >> > >> -- > >> Matt Stevenson. > -- Matt Stevenson.
