RogelioKG opened a new pull request, #59364:
URL: https://github.com/apache/spark/pull/59364
### What changes were proposed in this pull request?
Add type annotations to four internal functions in `pyspark.sql` and remove
the corresponding `# type: ignore[no-untyped-def]` comments:
| Function | Annotation | Based on |
|---|---|---|
| `check_string_field` in `pyspark/sql/functions/builtin.py` (`window`) |
`(field: Any, fieldName: str) -> None` | It validates arbitrary user input at
runtime, so `field` is intentionally `Any` |
| `infer_eval_type_for_udf` in `pyspark/sql/pandas/typehints.py` | `f:
Callable[..., Any]` | Both callers (`_create_py_udf` in classic and Connect)
declare `f: Callable[..., Any]` |
| `process` in
`pyspark/sql/connect/streaming/worker/foreach_batch_worker.py` | `(df_id: str,
batch_id: int, cloned_session_id: str) -> None` | Arguments come from
`utf8_deserializer.loads(...)` and `int(read_long(...))`; the JVM side writes
them with `PythonWorkerUtils.writeUTF` |
| `process` in `pyspark/sql/connect/streaming/worker/listener_worker.py` |
`(listener_event_str: str, listener_event_type: int) -> None` | Arguments come
from `utf8_deserializer.loads(...)` and `int(read_int(...))`; the JVM side
writes the event JSON with `PythonWorkerUtils.writeUTF` |
After this change there is no `type: ignore[no-untyped-def]` left in
`pyspark.sql`.
### Why are the changes needed?
Polish the type annotations for PySpark. With the suppressions in place,
mypy skipped the bodies of these functions entirely; now they are type-checked.
### Does this PR introduce _any_ user-facing change?
No. Only type annotations of internal functions are changed.
### How was this patch tested?
- `dev/lint-python --mypy` equivalent (`mypy --namespace-packages
--config-file python/mypy.ini python/pyspark`): the output is identical before
and after this change, and there are no errors in the four modified files.
- `ruff check` and `ruff format --check` pass on the modified files.
- The following tests pass locally:
- `python/run-tests --testnames 'pyspark.sql.functions.builtin'`
(doctests, including `window`)
- `python/run-tests --testnames 'pyspark.sql.tests.test_functions
FunctionsTests.test_window_time'`
- `python/run-tests --testnames 'pyspark.sql.tests.test_unified_udf'`
(exercises `infer_eval_type_for_udf` via `@udf`)
- `python/run-tests --testnames
'pyspark.sql.tests.connect.streaming.test_parity_foreach_batch'`
- `python/run-tests --testnames
'pyspark.sql.tests.connect.streaming.test_parity_listener'`
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 5.5)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
