rok commented on PR #55264:
URL: https://github.com/apache/arrow/pull/55264#issuecomment-6946304456
<details><summary>Detailed AI generated open questions</summary>
## Open questions
Summary of what is still open on this PR after going through all comments
and review threads (head `374093b`).
### Needs an answer or a decision
1. **Stubs and non-editable installs** (Joris, in the `development.rst`
thread). The docs say "With a non-editable install ... the stubs aren't
installed yet". This is because of #49831: the CMake block that installs the
stubs has been commented out since June, so no build installs them yet. Still
unexplained from the same thread: `mypy pyarrow/tests/test_table.py` finds the
same issues with or without `pyarrow-stubs` on the command line.
2. **IDE support with an editable install.** VS Code did not pick up the
stubs for Joris on 6 Nov. The layout has changed since
(`pyarrow-stubs/pyarrow/` became `pyarrow-stubs/`), which fixed PyCharm, but VS
Code has not been confirmed.
3. **Changes to `pyarrow/tests`.** The suggestion was to either type check
those tests in CI or leave the edits out of this PR. Most were removed, but 11
`# type: ignore` lines across six test files remain, and CI doesn't check them.
4. **`--follow-untyped-imports` for mypy.** Raised in review and tested
locally, but no decision is recorded and it isn't in `pyproject.toml`.
5. **`dictionary()` and `run_end_encoded()` reject a plain `DataType`**
(Copilot). `pa.dictionary(arr.type, pa.string())` works at runtime and fails in
mypy and pyright, because the parameter is typed with the stub-only integer
types. Should the "parameters take the base `Array`" rule also apply to
`DataType`?
6. **`Array` operators and `Array.from_dlpack` are missing from the stubs**
(Copilot). `arr + arr`, `-arr`, `arr & arr` and `pa.Array.from_dlpack(...)`
work on the nightly and are rejected by both checkers. Add them here or in a
follow-up?
### Deferred, tracked in an issue
- #55499: `pa.array()` can return a `ChunkedArray`; annotated as `Array` for
now.
- #55418: `pa.record_batch` raises for tuple data.
- #55450: remove the deprecated `ChunkedArray.data`.
- #55486: `test_asarray` never runs.
- #48970: remove the `__getattr__ -> Any` fallbacks before the stubs ship.
### Deferred, no issue yet
- **Squash `__lib/*.pyi` into `lib.pyi`**, and decide whether
`_stubs_typing.pyi` stays separate. Proposed for after the whole series is
merged.
- **`Literal` versus `str` for string options** (`unit` etc.). Left strict,
as in NumPy and pandas-stubs; to revisit once we see downstream impact.
- **`to_pandas` return type per class.** Left as a TODO.
- **`KeyValueMetadata.keys/values/items`** return generators at runtime
rather than mapping views. Worth a separate issue.
- **`Type_*` constants and private helpers** (`string_to_tzinfo`,
`tzinfo_to_string`, `ensure_metadata`, `table_to_blocks`) are not in the stubs.
Add them or allowlist them once stubtest stops skipping missing names.
- **Re-check the `Self` rule before merge** ("use `Self` when it is correct,
a concrete type otherwise").
- **`Array1D`** for parameters that only accept 1-d NumPy arrays.
- **`Iterable` / `Sequence` / `list` for array-like inputs.** Handled case
by case; no general rule written down.
- **`Buffer` placeholder is an empty class**, so `buffer.to_pybytes()` or
`len(buffer)` are rejected. Waiting for the I/O stubs PR.
### Additional findings from a local pass
Checked against the current head with mypy and pyright; not raised in review
so far.
- **Dict schemas:** `pa.schema({"a": pa.int8()})` and `pa.struct({...})` are
rejected. `Mapping[str | bytes, ...]` doesn't accept a `dict[str, ...]`,
because `Mapping` is invariant in its key type.
- **Metadata dicts:** for the same reason, a `dict[str, str]` variable is
rejected as `metadata=` in `pa.table`, `pa.record_batch`, `from_pydict` and
`from_pylist`.
- **`take` / `filter` with a `ChunkedArray`:** `Array.take` and
`Array.filter` are typed as returning `Array`, but the runtime returns a
`ChunkedArray`. `RecordBatch.take(ChunkedArray)` type checks and raises at
runtime.
- **Tuples:** rejected for `names=` in `pa.table` / `pa.record_batch`, and
in `rename_columns`, `drop_columns` and `Table.drop`, although they work at
runtime. `RecordBatch.select` rejects mixed names and indices, unlike
`Table.select`.
- **Housekeeping:** a per-file `# mypy: warn-unused-ignores` in
`test_tensor.py` and `test_device.py` (already set globally),
`pytest.importorskip("numpy")` in two stub tests, the `_compute.pyi` header
comment naming `pyarrow.lib`, and `= ...` defaults in `Bool8Array.to_numpy` and
`binary()`.
<details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]