wsulais opened a new pull request, #10754: URL: https://github.com/apache/arrow-rs/pull/10754
# Which issue does this PR close? - Closes #10752. # Rationale for this change arrow-rs binds `ArrowArray`, `ArrowSchema` and `ArrowArrayStream`, but nothing from the [C Device Data Interface](https://arrow.apache.org/docs/format/CDeviceDataInterface.html). A crate holding a device-resident Arrow array has no shared struct to pass it through, so it defines its own `#[repr(C)]` copy of `ArrowDeviceArray`, and two crates that both do that cannot interoperate. Arrow C++, nanoarrow and pyarrow all implement the interface already. This makes arrow-rs able to speak the interface and carry a device array's metadata faithfully. It does not make arrow-rs device-aware: no vendor dependency, no allocation, no copying, no synchronisation, and `sync_event` is never dereferenced. That is the split described in #7618 — kernels outside arrow-rs, but arrow-rs "able to receive arrays stored in GPU memory and pass them to kernels that can process them in the GPU". It is also a prerequisite for the async device stream interface raised in #7228: `ArrowAsyncTask::extract_data` writes into a `struct ArrowDeviceArray*`, so that interface cannot be bound in Rust until this struct exists. Async additionally raises the `futures` dependency question from #7228; the sync structs do not. It is not in this PR. # What changes are included in this PR? `arrow-data::ffi`, next to `FFI_ArrowArray`: - `ArrowDeviceType`, a `#[repr(transparent)]` newtype over `i32` with the constants from `abi.h`. Not an enum: the values track dlpack's `DLDeviceType` upstream, so a producer may send a device type this version does not know, and materialising an unrecognised discriminant into a Rust enum would be UB. - `FFI_ArrowDeviceArray`, with `new_cpu`, an `unsafe new` for callers that own device memory, `from_raw`, `empty`, and accessors. It has no release callback of its own, per the spec — the embedded `FFI_ArrowArray` owns the data. `arrow-array::ffi`: - `to_device_ffi`, `from_device_ffi`, `from_device_ffi_and_data_type`. Import rejects any `device_type` other than `ARROW_DEVICE_CPU` with an `ArrowError::CDataInterface` naming the device, rather than reading a device pointer as host memory. - Both types re-exported, so they reach `arrow::ffi` as `FFI_ArrowArray` does. `arrow-array::ffi_stream`: - `FFI_ArrowDeviceArrayStream`, with `new` from a `RecordBatchReader` (declaring `ARROW_DEVICE_CPU`) and an `unsafe new_unchecked` for a producer supplying its own callbacks. - `ArrowDeviceArrayStreamReader`, a `RecordBatchReader` over one, refusing a non-CPU stream at construction. Export declares a `device_id` of -1, which apache/arrow#41101 added to the spec for device types with no intrinsic device identifier and which Arrow C++ and pyarrow both emit. Import accepts any `device_id` for CPU data, since nanoarrow uses 0. No new dependencies or features; all of it sits behind the existing `ffi` feature. # Are these changes tested? Yes, and no GPU is needed. - `size_of`, `align_of` and `offset_of!` for both structs, gated to 64-bit targets. This is the check that matters: a `#[repr(C)]` struct with its fields in the wrong order compiles cleanly and round-trips fine within Rust. - CPU round trips for the array and the stream, shaped like the existing `to_ffi` / `from_ffi` tests, plus a doctest on `to_device_ffi`. - Non-CPU rejection for both. The stream test's `get_schema` and `get_next` are `unreachable!()`, so it fails if import touches the producer before checking `device_type`, and it asserts the refused stream is released rather than leaked. - An unknown `device_type` of 99 round-trips through the struct and is refused by number. - Both `device_id` conventions import for CPU. - `from_raw` leaves the source released, and a device array can be moved across a thread boundary. Cross-checked locally against two other implementations, which CI cannot do: - apache/arrow's `abi.h`, compiled with gcc 15.3: every `sizeof` and `offsetof` asserted here agrees, for both structs, as do the `ARROW_DEVICE_*` constants. - pyarrow 25.0.1: an int32 array with nulls and a utf8 array exported here and imported with `Array._import_from_c_device`, and the reverse via `Array._export_to_c_device`. pyarrow also emits `device_id == -1` for CPU data. I can add the pyarrow direction to `arrow-pyarrow-integration-testing` if that is wanted; it would need a minimum pyarrow version for the device methods. # Are there any user-facing changes? New public API only, and no breaking changes. arrow-rs still reads only CPU buffers: a non-CPU array is refused with an error, never dereferenced. Not included, each separable: the async device stream interface, device-resident `Buffer`s or kernels that can read them, and `sync_event` semantics beyond passing the pointer through. Assisted-by: Claude Opus 5 (claude-opus-5[1m]) via Claude Code 2.1.233 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
