CurtHagenlocher opened a new pull request, #503: URL: https://github.com/apache/arrow-dotnet/pull/503
## What's Changed Adds producer and consumer support for the experimental [C Async Device Stream Interface](https://arrow.apache.org/docs/format/CDeviceDataInterface.html#async-device-stream-interface) (`ArrowAsyncDeviceStreamHandler`, `ArrowAsyncProducer`, `ArrowAsyncTask`). The new API is marked `[Experimental("ArrowAsyncDeviceStreamApi")]`. **Producer:** `CArrowAsyncDeviceStreamProducer.Produce(IArrowArrayStream, handler*)` - A single produce loop, driven by `request(n)`, so handler callbacks are never concurrent and the source stream is never read from two threads. A `request` made inside `on_schema` doesn't start delivery until `on_schema` has returned. - Follows the spec for errors and shutdown: - `request(n <= 0)` gets `on_error(EINVAL)` and then `release`. - `request` after `cancel` does nothing. - `cancel` ends with `release` and no `on_error`. - A failed `extract_data` is followed by `on_error`. - Nothing is called on the handler after `release`. - Frees its `ArrowAsyncProducer` after `release`, as the spec allows. `extract_data` works even after that, since some consumers extract later. - Like the C stream exporters (#483), it takes ownership of each batch it exports. **Consumer:** `CArrowAsyncDeviceStreamConsumer.CreateStream(out handler*, queueSize = 16)` - Returns an `IArrowArrayStream`. It requests `queueSize` batches up front, then asks for more in chunks as the reader drains them. - The consumer allocates the handler and frees it in its own `release` callback, so callers never have to work out when it's safe to free. - Disposing the stream cancels the producer. Batches that arrive afterwards are extracted and dropped. ## Testing - **In-process (16 tests, net8.0/net462/net472):** round trips, ordering at queue sizes 1/3/64 (this checks that the source is never read concurrently), cancellation, a cancelled read not losing a batch, a schema export failure, and `request(n <= 0)`. The last one uses a test consumer written directly against the C callbacks with its own delegate types, so on .NET Framework it also covers #491. - **Interop with arrow-go v18.8.0 (10 tests, net8.0):** run through a small c-shared Go shim in `test/arrow-go-interop`, in both directions: round trips, empty streams, errors, and cancellation from either side. Data enters and leaves the shim through the synchronous C stream interface, so the only async step is the one being tested. The shim works around apache/arrow-go#1572 (arrow-go's consumer calls `request()` on the producer after release) with a proxy producer; see `proxy.h`. ## CI - **New `go-interop` job** on Linux, macOS and Windows. It's separate from the integration job, which is already long. It builds the shim and runs only the interop tests, with `ARROW_GO_INTEROP_REQUIRED=1` so a missing shim fails instead of skipping. - **Elsewhere,** the interop tests skip, so the main `test` job doesn't need Go. This is a draft partly to find out whether `windows-latest` has a gcc that cgo can use. If it doesn't, the Windows leg of `go-interop` will fail at the shim build, and its toolchain step prints `go env CC` and the compiler version. 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
