eppingere opened a new issue, #13978: URL: https://github.com/apache/arrow/issues/13978
I'm trying to use arrow IPC in C++ with a custom IO layer. I'm wondering what is the correct way to reclaim memory that is allocated by `arrow::io::InputStream::Read(int64_t)`. This returns a `std::shared_ptr<arrow::Buffer>`. The arrow documentation is contradictory on whether `std::shared_ptr<arrow::Buffer>` owns its own memory. Here it says that it doesn't own its memory: https://arrow.apache.org/docs/cpp/api/memory.html#_CPPv4N5arrow6BufferE Here it say it does own its own memory: https://arrow.apache.org/docs/cpp/api/io.html#_CPPv4N5arrow2io8Writable5WriteERKNSt10shared_ptrI6BufferEE If `arrow::Buffer` doesn't own its own memory, how do you know that arrow has finished using the memory contained in the buffer to free it? This is particularly useful in streaming applications of large amount of arrow ipc. Does arrow ever build references/pointers into the buffer that last beyond the lifetime of the `std::shared_ptr<arrow::Buffer>`? It seems that it can if the `std::shared_ptr<arrow::Buffer>`'s deconstruction doesn't deallocate the memory underlying the buffer. This seems to be implied to be the case depending on how you read the lifetime requirement on the bytes passed into this consume function: https://arrow.apache.org/docs/cpp/api/ipc.html#_CPPv4N5arrow3ipc13StreamDecoder7ConsumeEPK7uint8_t7int64_t If you read it as "the bytes passed in must be valid for all record batch processing" as in until the stream of bytes is done. Or you could read it as "these bytes must be valid until the bytes fed pass `next_required_size()` many" Is there an example of how to do this? If not what is the recommended way of reclaiming memory for processed batches that no longer have references in arrow that was allocated in the IO layer? Thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
