Got it Matt. Thanks ! On Thu, May 17, 2012 at 9:59 PM, Matt Stevenson <[email protected]>wrote:
> But what is the suggested way of transferring encoded bytes over the >> wire. Avro does not seem to expose the encoded bytes anywhere. >> > > I am using a custom method for transporting the encoding bytes, which is > why I had to get at the bytes. > Avro has an HTTP transport mechanisms defined, but I haven't used it and I > don't know if it is fully implemented in the C++ project. > For typical Avro usage you should not be accessing the raw bytes, which is > why nobody bothered to expose them. > > So even if I am encoding a small dataset, I would end up >> reading a min "Chunk_size" of memory block. >> > > The "count" variable should be the exact number of bytes in the stream. > Both for loops are checking "i<count", and "i" is only incremented in the > inner loop. Once you reach the count it will break out of both loops. > > > On Wed, May 16, 2012 at 4:45 AM, Gaurav Nanda <[email protected]> wrote: > >> Hi Matt, The approach you are using to read the written bytes would >> end up reading all the bytes allocated but not the bytes actually >> written. So even if I am encoding a small dataset, I would end up >> reading a min "Chunk_size" of memory block. >> >> On Mon, May 14, 2012 at 9:03 PM, Gaurav Nanda <[email protected]> >> wrote: >> > Thanks Matt ! >> > >> > But what is the suggested way of transferring encoded bytes over the >> > wire. Avro does not seem to expose the encoded bytes anywhere. >> > >> > - Gaurav Nanda >> > >> > On Sat, May 12, 2012 at 6:29 AM, Matt Stevenson >> > <[email protected]> wrote: >> >> I made a copy of the MemoryOutputStream with a header, so you can get >> at the >> >> bytes. >> >> >> >> http://branchingworlds.com/avro/OpenMemoryOutputStream.h >> >> http://branchingworlds.com/avro/OpenMemoryOutputStream.cpp >> >> >> >> Add those to your project. >> >> include "OpenMemoryOutputStream.h" and call openMemoryOutputStream() >> rather >> >> than memoryOutputStream(). >> >> >> >> The memory is stored in chunks. Here's an example of iterating >> through it: >> >> >> >>> auto_ptr<OpenMemoryOutputStream> os = openMemoryOutputStream(); >> >>> EncoderPtr e = binaryEncoder(); >> >>> e->init(*os); >> >>> >> >>> avro::encode(*e, *t); >> >>> e->flush(); >> >>> >> >>> int count = os->byteCount(); >> >>> char* data = new char[count]; >> >>> int i=0; >> >>> for (std::vector<uint8_t*>::const_iterator it = os->data_.begin(); it >> != >> >>> os->data_.end() && i<count; ++it) { >> >>> uint8_t* chunk = *it; >> >>> int size = os->chunkSize_; >> >>> for(int j=0; j<size && i<count; j++, i++){ >> >>> data[i] = chunk[j]; >> >>> } >> >>> } >> >> >> >> >> >> On Thu, May 10, 2012 at 12:19 PM, Gaurav Nanda <[email protected]> >> wrote: >> >>> >> >>> I want to extract byte array from memoryOutputStream to transfer the >> >>> encoded data. How do I achieve that? >> >> >> >> >> >> >> >> >> >> -- >> >> Matt Stevenson. >> > > > > -- > Matt Stevenson. >
