pitrou commented on pull request #8287: URL: https://github.com/apache/arrow/pull/8287#issuecomment-701297302
> If we slice a buffer from C++ and export it, does it knows that it cannot release the content? Yes, it does. I think you're overestimating the difficulty here, it's actually quite simple. In C++, Buffers (and slices, which are Buffers) are reference-counted (using the [shared_ptr](https://en.cppreference.com/w/cpp/memory/shared_ptr) class). To take your example: * A is reference-counted (at the start, the reference count is 1 since C++ holds a reference) * B is reference-counted, and also increments A's reference count (through the `parent` pointer) * the exported C data increments B's reference count (through the allocated `private_data`) * when Rust releases the C data, the `release` callback decrements B's reference count (by destroying the `private_data`) * if B's reference count has dropped to 0, it is destroyed, which also decrements A's reference count (through `parent`) * as long as C++ has a strong reference to A, its reference count is >= 1, so it isn't destroyed And anyway, Rust shouldn't worry about what happens on the C++ side. You're getting a `ArrowArray` struct, which may come from C++, but may also come from something else (e.g. [DuckDB](https://github.com/cwida/duckdb) has started supporting the C data interface). > I think that Rust's implementation of dictionaries is difficult to bridge with C Ah, that may be a problem. But you can start with non-dictionary arrays. I'd also recommend to start small (only primitive types, for example), check that everything works (especially lifetime handling), then implement more types. > How do we handle threads? We mutex the release? You don't have to. `release` should only be called when the consumer is done with the data, so by definition it cannot be called if other threads are accessing data. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
