CurtHagenlocher opened a new pull request, #503:
URL: https://github.com/apache/arrow-dotnet/pull/503

   ## What's Changed
   
   Adds producer and consumer support for the experimental [C Async Device 
Stream 
Interface](https://arrow.apache.org/docs/format/CDeviceDataInterface.html#async-device-stream-interface)
 (`ArrowAsyncDeviceStreamHandler`, `ArrowAsyncProducer`, `ArrowAsyncTask`). The 
new API is marked `[Experimental("ArrowAsyncDeviceStreamApi")]`.
   
   **Producer:** `CArrowAsyncDeviceStreamProducer.Produce(IArrowArrayStream, 
handler*)`
   - A single produce loop, driven by `request(n)`, so handler callbacks are 
never concurrent and the source stream is never read from two threads. A 
`request` made inside `on_schema` doesn't start delivery until `on_schema` has 
returned.
   - Follows the spec for errors and shutdown:
     - `request(n <= 0)` gets `on_error(EINVAL)` and then `release`.
     - `request` after `cancel` does nothing.
     - `cancel` ends with `release` and no `on_error`.
     - A failed `extract_data` is followed by `on_error`.
     - Nothing is called on the handler after `release`.
   - Frees its `ArrowAsyncProducer` after `release`, as the spec allows. 
`extract_data` works even after that, since some consumers extract later.
   - Like the C stream exporters (#483), it takes ownership of each batch it 
exports.
   
   **Consumer:** `CArrowAsyncDeviceStreamConsumer.CreateStream(out handler*, 
queueSize = 16)`
   - Returns an `IArrowArrayStream`. It requests `queueSize` batches up front, 
then asks for more in chunks as the reader drains them.
   - The consumer allocates the handler and frees it in its own `release` 
callback, so callers never have to work out when it's safe to free.
   - Disposing the stream cancels the producer. Batches that arrive afterwards 
are extracted and dropped.
   
   ## Testing
   
   - **In-process (16 tests, net8.0/net462/net472):** round trips, ordering at 
queue sizes 1/3/64 (this checks that the source is never read concurrently), 
cancellation, a cancelled read not losing a batch, a schema export failure, and 
`request(n <= 0)`. The last one uses a test consumer written directly against 
the C callbacks with its own delegate types, so on .NET Framework it also 
covers #491.
   - **Interop with arrow-go v18.8.0 (10 tests, net8.0):** run through a small 
c-shared Go shim in `test/arrow-go-interop`, in both directions: round trips, 
empty streams, errors, and cancellation from either side. Data enters and 
leaves the shim through the synchronous C stream interface, so the only async 
step is the one being tested. The shim works around apache/arrow-go#1572 
(arrow-go's consumer calls `request()` on the producer after release) with a 
proxy producer; see `proxy.h`.
   
   ## CI
   
   - **New `go-interop` job** on Linux, macOS and Windows. It's separate from 
the integration job, which is already long. It builds the shim and runs only 
the interop tests, with `ARROW_GO_INTEROP_REQUIRED=1` so a missing shim fails 
instead of skipping.
   - **Elsewhere,** the interop tests skip, so the main `test` job doesn't need 
Go.
   
   This is a draft partly to find out whether `windows-latest` has a gcc that 
cgo can use. If it doesn't, the Windows leg of `go-interop` will fail at the 
shim build, and its toolchain step prints `go env CC` and the compiler version.
   
   🤖 Generated with [Claude Code](https://claude.com/claude-code)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to