AlenkaF commented on PR #55264: URL: https://github.com/apache/arrow/pull/55264#issuecomment-6528432757
@tadeja @rok I have checked typed.pyi and decided to push a commit (https://github.com/apache/arrow/pull/55264/commits/4b97c69b88801db1bd9f73ce76b84d8a9d1674bf) after all =) Instead of suggestions I will post the reasoning behind the changes here: * NumPy is also an optional dependency (next to pandas) so I added `# type: ignore[..]` there too. Locally mypy and pyright both pass with only `# type: ignore[..]` so I removed pyright ignores * ty doesn't honor mypy's `# type: ignore[...]` and `# ty: ignore[unresolved-import]` fails if numpy/pandas are installed, so I disabled ty unresolved-import in pyproject.toml instead (mypy and pyright still catch broken imports). * I removed `noqa: F401` on the `lib` import and ruff didn't complain * I have changed some classmethods in stubs as they are defined as static methods in source (types.pxi). Example: `_import_from_c` in `DataType`, `Field` and `Schema`. * I propose to use `typing_extensions.CapsuleType` instead of `Any` in `__arrow_c_schema__`. Claude added a test for this case, I have reviewed and changed it a bit. `typing_extensions` dependency had to be added. Note: we should leave the Protocols in `_stubs_typing.pyi` at `Any`. Polars' public `pl.DataFrame` and `pl.Series`, for example, annotate `__arrow_c_stream__ -> object` so narrowing the Protocol would reject them * https://github.com/pola-rs/polars/blob/fe540d641165534f2a5e571234c3ee78d51587ee/py-polars/src/polars/dataframe/frame.py#L1587 * https://github.com/pola-rs/polars/blob/fe540d641165534f2a5e571234c3ee78d51587ee/py-polars/src/polars/series/series.py#L1632 Claude suggested to add a test for Protocols also (under the Table stubs tests, to make sure they do not become to narrow) but I think these should be added later. * `KeyValueMetadata` stubs do not include `keys()`, `values()` and `items()` stubs because they inherit them from `Mapping[bytes, bytes]`: https://github.com/python/typeshed/blob/17cbf254205549ccdcf886fadf8c265e634ad596/stdlib/typing.pyi#L823-L825 but our code overrides all three as generators, not Mapping views (probably so that duplicate keys are kept): https://github.com/apache/arrow/blob/9b9fad6810850c3b884e14617356500143ef4aeb/python/pyarrow/types.pxi#L2409-L2419 I have added overrides to the stubs so they are in sync with runtime and added ignores because of ``` Return type "Iterator[bytes]" of "keys" incompatible with return type "KeysView[bytes]" in supertype "typing.Mapping" [override] ``` I have also added a test `test_key_value_metadata`. I think code overrides need a separate issue as we might want to decide on changing the overrides in `types.pxi` `KeyValueMetadata` class to return Mapping views instead of generators (and to somehow keep support of duplicate keys). * `keys_sorted` in `map_` take `bool`, `None` is by truthiness converted to `False` in Cython but I think it should not be suggested as a possible input so I changed the default from `None` to `False`. Same for `dictionary().` * Included the use of `np.typing.DTypeLike` for parameter type in `from_numpy_dtype` https://numpy.org/devdocs/reference/typing.html#numpy.typing.DTypeLike * I have changed the `Field` methods that returned `Self` to return `Field` as the runtime always builds a plain `Field` via `pyarrow_wrap_field`. Future TODO: * `string_to_tzinfo`, `tzinfo_to_string` and `ensure_metadata` are not in the stubs currently (not public). We should add them to the stubtest allowlist at some point or think of adding them to the stubs. I have run * all 3 type checkers on this file with and without numpy/pandas installed * the tests * ruff linter check -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
