This is an automated email from the ASF dual-hosted git repository.
gaogaotiantian pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new 0ff392a0ed23 [SPARK-60163][PYTHON] Remove type: ignore[no-untyped-def]
in pyspark.sql
0ff392a0ed23 is described below
commit 0ff392a0ed23cd570942e219b25d78e1ceb92a6b
Author: Jerry Su <[email protected]>
AuthorDate: Mon Nov 2 16:18:30 2026 -0800
[SPARK-60163][PYTHON] Remove type: ignore[no-untyped-def] in pyspark.sql
### What changes were proposed in this pull request?
Add type annotations to four internal functions in `pyspark.sql` and remove
the corresponding `# type: ignore[no-untyped-def]` comments:
| Function | Annotation | Based on |
|---|---|---|
| `check_string_field` in `pyspark/sql/functions/builtin.py` (`window`) |
`(field: Any, fieldName: str) -> None` | It validates arbitrary user input at
runtime, so `field` is intentionally `Any` |
| `infer_eval_type_for_udf` in `pyspark/sql/pandas/typehints.py` | `f:
Callable[..., Any]` | Both callers (`_create_py_udf` in classic and Connect)
declare `f: Callable[..., Any]` |
| `process` in
`pyspark/sql/connect/streaming/worker/foreach_batch_worker.py` | `(df_id: str,
batch_id: int, cloned_session_id: str) -> None` | Arguments come from
`utf8_deserializer.loads(...)` and `int(read_long(...))`; the JVM side writes
them with `PythonWorkerUtils.writeUTF` |
| `process` in `pyspark/sql/connect/streaming/worker/listener_worker.py` |
`(listener_event_str: str, listener_event_type: int) -> None` | Arguments come
from `utf8_deserializer.loads(...)` and `int(read_int(...))`; the JVM side
writes the event JSON with `PythonWorkerUtils.writeUTF` |
After this change there is no `type: ignore[no-untyped-def]` left in
`pyspark.sql`.
### Why are the changes needed?
Polish the type annotations for PySpark. With the suppressions in place,
mypy skipped the bodies of these functions entirely; now they are type-checked.
### Does this PR introduce _any_ user-facing change?
No. Only type annotations of internal functions are changed.
### How was this patch tested?
- `dev/lint-python --mypy` equivalent (`mypy --namespace-packages
--config-file python/mypy.ini python/pyspark`): the output is identical before
and after this change, and there are no errors in the four modified files.
- `ruff check` and `ruff format --check` pass on the modified files.
- The following tests pass locally:
- `python/run-tests --testnames 'pyspark.sql.functions.builtin'`
(doctests, including `window`)
- `python/run-tests --testnames 'pyspark.sql.tests.test_functions
FunctionsTests.test_window_time'`
- `python/run-tests --testnames 'pyspark.sql.tests.test_unified_udf'`
(exercises `infer_eval_type_for_udf` via `udf`)
- `python/run-tests --testnames
'pyspark.sql.tests.connect.streaming.test_parity_foreach_batch'`
- `python/run-tests --testnames
'pyspark.sql.tests.connect.streaming.test_parity_listener'`
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 5.5)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Closes #59364 from RogelioKG/type-hints.
Authored-by: Jerry Su <[email protected]>
Signed-off-by: Tian Gao <[email protected]>
(cherry picked from commit 676e64ba0eeb400173d70889ddad92bf0ee5d420)
Signed-off-by: Tian Gao <[email protected]>
---
python/pyspark/sql/connect/streaming/worker/foreach_batch_worker.py | 2 +-
python/pyspark/sql/connect/streaming/worker/listener_worker.py | 2 +-
python/pyspark/sql/functions/builtin.py | 2 +-
python/pyspark/sql/pandas/typehints.py | 4 ++--
4 files changed, 5 insertions(+), 5 deletions(-)
diff --git
a/python/pyspark/sql/connect/streaming/worker/foreach_batch_worker.py
b/python/pyspark/sql/connect/streaming/worker/foreach_batch_worker.py
index e7f7c83bfbdc..24fa78d4fdcd 100644
--- a/python/pyspark/sql/connect/streaming/worker/foreach_batch_worker.py
+++ b/python/pyspark/sql/connect/streaming/worker/foreach_batch_worker.py
@@ -47,7 +47,7 @@ def main(infile: IO, outfile: IO) -> None:
log_name = "Streaming ForeachBatch worker"
- def process(df_id, batch_id, cloned_session_id): # type:
ignore[no-untyped-def]
+ def process(df_id: str, batch_id: int, cloned_session_id: str) -> None:
global spark
# Lazily create or switch the SparkSession when the cloned session
changes.
# Use create() instead of getOrCreate() because the latter returns the
cached
diff --git a/python/pyspark/sql/connect/streaming/worker/listener_worker.py
b/python/pyspark/sql/connect/streaming/worker/listener_worker.py
index 166f41d4f02d..43e941a4cda2 100644
--- a/python/pyspark/sql/connect/streaming/worker/listener_worker.py
+++ b/python/pyspark/sql/connect/streaming/worker/listener_worker.py
@@ -77,7 +77,7 @@ def main(infile: IO, outfile: IO) -> None:
listener._set_spark_session(spark_connect_session)
assert listener.spark == spark_connect_session
- def process(listener_event_str, listener_event_type): # type:
ignore[no-untyped-def]
+ def process(listener_event_str: str, listener_event_type: int) -> None:
global spark
listener_event = json.loads(listener_event_str)
if listener_event_type == 0:
diff --git a/python/pyspark/sql/functions/builtin.py
b/python/pyspark/sql/functions/builtin.py
index 363359c7ebdc..2de5d232ef85 100644
--- a/python/pyspark/sql/functions/builtin.py
+++ b/python/pyspark/sql/functions/builtin.py
@@ -14483,7 +14483,7 @@ def window(
"""
from pyspark.sql.classic.column import _to_java_column
- def check_string_field(field, fieldName): # type: ignore[no-untyped-def]
+ def check_string_field(field: Any, fieldName: str) -> None:
if not field or type(field) is not str:
raise PySparkTypeError(
errorClass="NOT_EXPECTED_TYPE",
diff --git a/python/pyspark/sql/pandas/typehints.py
b/python/pyspark/sql/pandas/typehints.py
index 18578b94f4dd..cf475d16ac72 100644
--- a/python/pyspark/sql/pandas/typehints.py
+++ b/python/pyspark/sql/pandas/typehints.py
@@ -376,8 +376,8 @@ def infer_eval_type(
# infer the eval type for @udf
-def infer_eval_type_for_udf( # type: ignore[no-untyped-def]
- f,
+def infer_eval_type_for_udf(
+ f: Callable[..., Any],
) -> Optional[
Union[
"PandasScalarUDFType",
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]