kylebarron opened a new pull request, #11161: URL: https://github.com/apache/arrow-rs/pull/11161
# Which issue does this PR close? - Closes https://github.com/apache/arrow-rs/issues/6586. # Rationale for this change Arrow-rs doesn't currently make it possible to describe sequences of _arrays_. It only really defines sequences of _Record batches_. This makes it hard to interface with libraries that use "chunked arrays" concepts or similar. This PR introduces code that I've used in [`pyo3-arrow`](https://crates.io/crates/pyo3-arrow) for a while. (It was necessary to support interop with a pyarrow `ChunkedArray`). # What changes are included in this PR? - Creates `ArrayReader` and `ArrayIterator` concepts: parallels to `RecordBatchReader` and `RecordBatchIterator` that yields `ArrayRef` instead of `RecordBatch`. - Defines `ArrayStreamReader` to describe an FFI stream of arrays, not of record batches > [!IMPORTANT] > Naming is hard here because the existing `ArrowArrayStreamReader` is a bit ambiguously named. It's the core struct for reading from the `ArrowArrayStream` FFI container. But it reads _`RecordBatch`es_, not arrays. > > So what do we name the struct that yields arrays? One struct can't implement both `ArrayReader` and `RecordBatchReader` because each trait requires implementation of `Iterator` with a different yield type. > > I'm not sure what the best path forward is here. ## Todo: - [ ] We don't yet support _exporting_ an `ArrayReader` through FFI. I wanted to wait and get feedback on naming and general approach before implementing that. # Are these changes tested? <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 2. Serve as another way to document the expected behavior of the code If tests are not included in your PR, please explain why (for example, are they covered by existing tests)? If this PR claims a performance improvement, please include evidence such as benchmark results. --> # Are there any user-facing changes? Yes, new APIs. <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. If there are any breaking changes to public APIs, please call them out. --> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
