roee88 opened a new issue #596: URL: https://github.com/apache/arrow-rs/issues/596
The FFI implementation deviates from the C Data Interface specification in [Member allocation](https://arrow.apache.org/docs/format/CDataInterface.html#member-allocation). > The ArrowSchema and ArrowArray structures follow the same conventions for memory management. The term “base structure” below refers to the ArrowSchema or ArrowArray that is passed between producer and consumer – not any child structure thereof. > > Member allocation > > It is intended for the base structure to be stack- or heap-allocated by the consumer. In this case, the producer API should take a pointer to the consumer-allocated structure. Taking an _example_ of a pyarrow consumer and arrow-rs producer, my understanding is that the intended behavior is: 1. pyarrow allocates empty ffi structure(s) 2. pyarrow pass the empty ffi structure(s) to arrow-rs 3. arrow-rs fills the empty ffi structure(s) from a rust Array/Schema/RecordBatch 4. when done with the data pyarrow calls the release callback on the ffi structure(s) that it allocated in step 1 This doesn't seem to be supported currently. The current implementation for exporting through the C Data Interface only allows converting an arrow array/schema to ffi array/schema, both allocated by arrow-rs. A solution to this must also take into account that the release callback must be called explicitly by the consumer, avoiding calling release unintentionally by drop. The opposite path is correctly implemented as demonstrated in https://github.com/apache/arrow-rs/blob/9f40f899e439d072fc859e0b4abf46776387e0d1/arrow-pyarrow-integration-testing/src/lib.rs#L174 (rust allocates empty, pass to the the a python exporter that fills those structs). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
