Our current implementation raises an exception saying the buffer is too small.
The reason for my suggested api change is purely to allow our client to allocate buffers, not to introduce extras semantics and complexity into the API or the underlying code. What use case would your suggestions help with? Are they corner cases? My company's use cases are encapsulated by my suggested changes. Extending this to fixed and strings would be sensible, but not something we've implement. b On Tue, 9 Mar 2021 at 15:41, Thiruvalluvan MG <[email protected]> wrote: > Makes sense. > But, I have a few concerns: > 1. When the client invokes decodeBytesData() and the the buffer is too > small, what should the implementation do. One answer is to throw an > exception. But it will be nice if the client can somehow be told of the > expected minimum length so that the client can reattempt.2. The client is > forced to make two calls in order to decode avro Bytes data type. As a > consequence one needs to maintain extra state in the decoder to remember > that decodeBytesSize() has been called but not is > corresponding decodeBytesData().3. The same set of problems exist for > strings and "fixed" types as well. It'll be nice we come up with some > uniform approach for all three Avro types. > Thanks > Thiru > On Tuesday, 9 March, 2021, 01:40:26 am IST, Bruno Nicoletti > <[email protected]> wrote: > > Thanks for the feedback. That would require a greater reworking of the > code > internals than my suggestion, which much more closely matches how the code > is currently structured. > > On Mon, 8 Mar 2021 at 19:46, Thiruvalluvan MG <[email protected]> > wrote: > > > I think it is good idea. Here is a suggestion for improvement. Instead > of > > having two functions, we can have just one: > > size_t Decode::decodeBytesData(uint8_t *buffer, size_t len); > > > > If the decoding can fit in len bytes, it decodes and return the number of > > bytes used. If it cannot, then, it will return the number of bytes > > required. It will help those clients who have some buffer space. They can > > try to decode into the their buffer, if it fails, they will have to > > allocate a bigger buffer and try again. Perhaps, we can call the > > function decodeBytesDataIfPossible to make the intention clear. > > Thanks > > Thiru On Monday, 8 March, 2021, 09:16:16 pm IST, Bruno Nicoletti > > <[email protected]> wrote: > > > > Hi All, > > > > We serialise some binary blobs as part of our data using the C++ > > implementation of Avro. Currently the C++ API has several calls for this > > that pass values back to the client via std::vector<uint8_t>, for > > example... > > > > std::vector<uint8_t> Decode::decodeBytes(); > > void Decode::decodeBytes(std::vector<uint8_t> &value); > > > > If the client isn't holding the data in a std::vector, or using some > other > > allocation method, this forces the client to do a separate allocation and > > memcpy the data out of the vector that the API returns the data in. We've > > implemented a new API on the decoder that allows the client to allocate > the > > data and pass the buffers into the decoder. The API is... > > > > /// Decodes the number of bytes that will be read in a subsequent > > /// \ref decodeBytesData call. > > size_t Decode::decodeBytesSize(); > > > > /// Decodes the bytes data into an allocated buffer of length len, > > which > > /// must contain at least as many bytes as was returned by the > previous > > /// \ref decodeBytesSize() call. > > void Decode::decodeBytesData(uint8_t *buffer, size_t len); > > > > So the client would do 2 calls to read the data blob, > > size_t nBytesToRead = decoder->decodeBytesSize(); > > void *buffer = myAllocationMethod(nBytesToRead); > > decoder->decodeBytes(buffer, nBytesToRead); > > > > Would this be a thing you'd want to integrate onto the mainline of avro? > If > > so we can tidy it up, make some tests and make a PR for it. > > > > thanks > > > > Bruno > > > > -- > > Bruno Nicoletti, CTO and Co-founder > > Filigree Technologies Ltd, > > > > > > -- > Bruno Nicoletti, CTO and Co-founder > Filigree Technologies Ltd, > Unit SB 140, China Works, 100 Black Prince Rd, London, SE1 7SJ, UK > Registered in England and Wales No: 10742657 > -- Bruno Nicoletti, CTO and Co-founder Filigree Technologies Ltd, Unit SB 140, China Works, 100 Black Prince Rd, London, SE1 7SJ, UK Registered in England and Wales No: 10742657
