jonasdedden opened a new pull request, #10492:
URL: https://github.com/apache/arrow-rs/pull/10492
# Which issue does this PR close?
- Closes #10491.
# Rationale for this change
PyO3's `experimental-inspect` feature records the Python type of every
conversion into the built binary, so that tools like `maturin generate-stubs`
can emit a real `.pyi`. The type comes from the `INPUT_TYPE` / `OUTPUT_TYPE`
associated constants on `FromPyObject` / `IntoPyObject`.
`PyArrowType` implements both traits but leaves those constants at their
default, `_typeshed.Incomplete`. Because `PyArrowType` is how bindings exchange
*every* Arrow value with Python, that one default erases the entire public API
of any such binding. A function like
```rust
#[pyfunction]
fn cast_record_batch(
record_batch: PyArrowType<RecordBatch>,
schema: PyArrowType<Schema>,
) -> PyResult<PyArrowType<RecordBatch>> { ... }
```
generates
```python
def cast_record_batch(record_batch: Incomplete, schema: Incomplete) ->
Incomplete: ...
```
# What changes are included in this PR?
- A new `experimental-inspect` feature on `arrow-pyarrow`, which just turns
on PyO3's. `arrow` gets a matching feature that implies `pyarrow`.
- `FromPyArrow::INPUT_TYPE`, `ToPyArrow::OUTPUT_TYPE` and
`IntoPyArrow::OUTPUT_TYPE`, defaulting to `_typeshed.Incomplete` so that
out-of-tree implementors are unaffected.
- Implementations for every type the crate converts: `DataType`, `Field`,
`Schema`, `ArrayData` (→ `pyarrow.Array`), `RecordBatch`, `Vec<T>`,
`ArrowArrayStreamReader`, `Box<dyn RecordBatchReader + Send>` and `Table`.
- `PyArrowType`'s `FromPyObject` / `IntoPyObject` impls forward them.
- Unit tests pinning the rendered hint for each type, and a note in the
crate docs.
Input and output are separate constants because they genuinely differ:
`Vec<T>` is built from anything iterable but handed back as a `list`, so it is
`collections.abc.Iterable[pyarrow.RecordBatch]` in and
`list[pyarrow.RecordBatch]` out.
# Are these changes tested?
Yes. `arrow-pyarrow` gains unit tests asserting the rendered hint for every
conversion, run with `cargo test -p arrow-pyarrow --features
experimental-inspect`.
Beyond that, the change was verified end to end against a real binding: with
`arrow` patched to this branch and the binding's own code left on plain
`PyArrowType`, `maturin generate-stubs` produces the signatures and the `from
pyarrow import ...` line shown above.
# Are there any user-facing changes?
No behaviour changes, and nothing is compiled unless `experimental-inspect`
is enabled. The traits gain associated constants, but they are defaulted and
feature-gated, so existing implementations keep compiling untouched.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
