timsaucer opened a new pull request, #1786: URL: https://github.com/apache/datafusion-python/pull/1786
# Which issue does this PR close? No issue filed yet; happy to open one if preferred. # Rationale for this change The package ships `py.typed`, so downstream type checkers consume our annotations, but nothing in CI checks that those annotations are correct. Ruff's `ANN` rules only require annotations to exist. Running [ty](https://docs.astral.sh/ty/) against `python/datafusion` reported 82 diagnostics on `main`; after configuring imports, about 50 were real annotation drift, and one was a runtime bug (`@udtf()` without a name passed `None` into Rust and failed with `TypeError: 'None' is not an instance of 'str'`). ty is from Astral, like the ruff and uv tooling already used here, and is fast enough to run as a pre-commit hook. # What changes are included in this PR? Tooling: - Add `ty==0.0.84` to the `dev` dependency group. It is pinned exactly because ty is pre-1.0 and new releases can add diagnostics. - Add `[tool.ty]` config limiting the check to `python/datafusion`. `datafusion._internal` (compiled, no stubs) and the optional `pandas`/`polars` imports are allowed to be unresolved, so the check does not need a built wheel. - Run `ty check` in the `lint-python` job and as a local pre-commit hook. Fixes for the reported diagnostics: - `TableFunction`: the internal decorator helper now requires `name`, matching the public `udtf` overloads, so `@udtf()` fails with a normal missing-argument error instead of inside Rust. - `AggregateUDF.__init__`: the implementation now has the `None` defaults its FFI overload already advertised. `ScalarUDF` and `WindowUDF` get the same FFI overload, following the `AggregateUDF` pattern. - `expr_list_to_raw_expr_list` / `sort_list_to_raw_sort_list`: add `None`→`None` / sequence→`list` overloads so callers that unpack the result (`select`, `sort`, `window`, ...) type check, while optional `order_by` callers still pass `None` through. - Stop rebinding typed `*args` and parameters to values of other types (`concat`, `coalesce`, `make_array`, `repartition_by_hash`, `unnest_columns`, ...). - Import `warnings.deprecated` behind `sys.version_info >= (3, 13)` instead of `try`/`except`, and drop the unreachable `importlib_metadata` fallback (`requires-python >= 3.10`). - Smaller mismatches: `LogicalPlan.__eq__` takes `object`, `spark._coerce_i32` no longer claims to accept `None`, CSV `file_compression_type` is normalized before reaching `CsvReadOptions`, and one documented `cast` in `join` for the legacy tuple form of `on`. - Three inline ignores for intentional behavior: `Expr.__eq__`/`__ne__` return `Expr`, and the deprecated `CreateExternalTable.location` alias assignment. # Are there any user-facing changes? No API changes. `@udtf()` with no name still fails, but now with a standard Python `TypeError` for the missing argument. `AggregateUDF(name, exportable)` now works as its type signature already promised. Contributors need `ty` in their environment (`uv sync --dev`) for the new pre-commit hook. 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
