lidavidm commented on issue #28: URL: https://github.com/apache/arrow-go/issues/28#issuecomment-3275257479
The ultimate issue is this: (1) I'm guessing Polars creates a array by taking over a Vec ([a Buffer is created from a SharedStorage](https://github.com/pola-rs/polars/blob/1c2ac1a25f5d65ccb4caff07d93aea5004dabc31/crates/polars-arrow/src/buffer/immutable.rs#L92), and [a SharedStorage can be created from a Vec](https://github.com/pola-rs/polars/blob/1c2ac1a25f5d65ccb4caff07d93aea5004dabc31/crates/polars-arrow/src/storage.rs#L159-L164)) via `as_mut_ptr` and `forget`. (2) An empty Vec won't allocate and you get whatever RawVec has. (3) A RawVec [is a RawVecInner](https://doc.rust-lang.org/src/alloc/raw_vec/mod.rs.html#87) which is a `Unique<u8>`. (4) [A freshly initialized Unique is a `NonNull::dangling`](https://doc.rust-lang.org/beta/src/core/ptr/unique.rs.html#74) (5) That just [creates an invalid pointer](https://doc.rust-lang.org/beta/src/core/ptr/non_null.rs.html#130-133) which in this case happens to be `0x1` (because it casts the numeric value of the alignment of the type to a pointer, and the alignment of u8 is 1). Why? Because that enables the compiler to [optimize `Option<Unique<T>>` to just a raw pointer](https://doc.rust-lang.org/beta/src/core/ptr/non_null.rs.html#18-21) (pretty neat trick, IMO) (6) So ultimately Polars takes over a pointer of value `0x1` to an allocation of length `0`, which is fine inside Rust code (7) But when exported across the FFI boundary, it trips up the Go garbage collector which instantly loses its mind when it glances at an invalid but non-nullptr pointer. I think Polars should update SharedStorage::from_vec to explicitly use a nullptr when the vec it tries to take over is empty, since that can safely be exported via FFI. Note that I haven't confirmed all this in a debugger, I'm figuring this out purely based on looking at docs, the pqarrow source, and what I'm seeing with polars in a REPL. But I think this is pretty plausible and could be confirmed by someone willing to build polars from source. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
