This is an automated email from the ASF dual-hosted git repository.

gaogaotiantian pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/branch-4.x by this push:
     new 0ff392a0ed23 [SPARK-60163][PYTHON] Remove type: ignore[no-untyped-def] 
in pyspark.sql
0ff392a0ed23 is described below

commit 0ff392a0ed23cd570942e219b25d78e1ceb92a6b
Author: Jerry Su <[email protected]>
AuthorDate: Mon Nov 2 16:18:30 2026 -0800

    [SPARK-60163][PYTHON] Remove type: ignore[no-untyped-def] in pyspark.sql
    
    ### What changes were proposed in this pull request?
    
    Add type annotations to four internal functions in `pyspark.sql` and remove 
the corresponding `# type: ignore[no-untyped-def]` comments:
    
    | Function | Annotation | Based on |
    |---|---|---|
    | `check_string_field` in `pyspark/sql/functions/builtin.py` (`window`) | 
`(field: Any, fieldName: str) -> None` | It validates arbitrary user input at 
runtime, so `field` is intentionally `Any` |
    | `infer_eval_type_for_udf` in `pyspark/sql/pandas/typehints.py` | `f: 
Callable[..., Any]` | Both callers (`_create_py_udf` in classic and Connect) 
declare `f: Callable[..., Any]` |
    | `process` in 
`pyspark/sql/connect/streaming/worker/foreach_batch_worker.py` | `(df_id: str, 
batch_id: int, cloned_session_id: str) -> None` | Arguments come from 
`utf8_deserializer.loads(...)` and `int(read_long(...))`; the JVM side writes 
them with `PythonWorkerUtils.writeUTF` |
    | `process` in `pyspark/sql/connect/streaming/worker/listener_worker.py` | 
`(listener_event_str: str, listener_event_type: int) -> None` | Arguments come 
from `utf8_deserializer.loads(...)` and `int(read_int(...))`; the JVM side 
writes the event JSON with `PythonWorkerUtils.writeUTF` |
    
    After this change there is no `type: ignore[no-untyped-def]` left in 
`pyspark.sql`.
    
    ### Why are the changes needed?
    
    Polish the type annotations for PySpark. With the suppressions in place, 
mypy skipped the bodies of these functions entirely; now they are type-checked.
    
    ### Does this PR introduce _any_ user-facing change?
    
    No. Only type annotations of internal functions are changed.
    
    ### How was this patch tested?
    
    - `dev/lint-python --mypy` equivalent (`mypy --namespace-packages 
--config-file python/mypy.ini python/pyspark`): the output is identical before 
and after this change, and there are no errors in the four modified files.
    - `ruff check` and `ruff format --check` pass on the modified files.
    - The following tests pass locally:
      - `python/run-tests --testnames 'pyspark.sql.functions.builtin'` 
(doctests, including `window`)
      - `python/run-tests --testnames 'pyspark.sql.tests.test_functions 
FunctionsTests.test_window_time'`
      - `python/run-tests --testnames 'pyspark.sql.tests.test_unified_udf'` 
(exercises `infer_eval_type_for_udf` via `udf`)
      - `python/run-tests --testnames 
'pyspark.sql.tests.connect.streaming.test_parity_foreach_batch'`
      - `python/run-tests --testnames 
'pyspark.sql.tests.connect.streaming.test_parity_listener'`
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Code (Claude Opus 5.5)
    
    🤖 Generated with [Claude Code](https://claude.com/claude-code)
    
    Closes #59364 from RogelioKG/type-hints.
    
    Authored-by: Jerry Su <[email protected]>
    Signed-off-by: Tian Gao <[email protected]>
    (cherry picked from commit 676e64ba0eeb400173d70889ddad92bf0ee5d420)
    Signed-off-by: Tian Gao <[email protected]>
---
 python/pyspark/sql/connect/streaming/worker/foreach_batch_worker.py | 2 +-
 python/pyspark/sql/connect/streaming/worker/listener_worker.py      | 2 +-
 python/pyspark/sql/functions/builtin.py                             | 2 +-
 python/pyspark/sql/pandas/typehints.py                              | 4 ++--
 4 files changed, 5 insertions(+), 5 deletions(-)

diff --git 
a/python/pyspark/sql/connect/streaming/worker/foreach_batch_worker.py 
b/python/pyspark/sql/connect/streaming/worker/foreach_batch_worker.py
index e7f7c83bfbdc..24fa78d4fdcd 100644
--- a/python/pyspark/sql/connect/streaming/worker/foreach_batch_worker.py
+++ b/python/pyspark/sql/connect/streaming/worker/foreach_batch_worker.py
@@ -47,7 +47,7 @@ def main(infile: IO, outfile: IO) -> None:
 
     log_name = "Streaming ForeachBatch worker"
 
-    def process(df_id, batch_id, cloned_session_id):  # type: 
ignore[no-untyped-def]
+    def process(df_id: str, batch_id: int, cloned_session_id: str) -> None:
         global spark
         # Lazily create or switch the SparkSession when the cloned session 
changes.
         # Use create() instead of getOrCreate() because the latter returns the 
cached
diff --git a/python/pyspark/sql/connect/streaming/worker/listener_worker.py 
b/python/pyspark/sql/connect/streaming/worker/listener_worker.py
index 166f41d4f02d..43e941a4cda2 100644
--- a/python/pyspark/sql/connect/streaming/worker/listener_worker.py
+++ b/python/pyspark/sql/connect/streaming/worker/listener_worker.py
@@ -77,7 +77,7 @@ def main(infile: IO, outfile: IO) -> None:
     listener._set_spark_session(spark_connect_session)
     assert listener.spark == spark_connect_session
 
-    def process(listener_event_str, listener_event_type):  # type: 
ignore[no-untyped-def]
+    def process(listener_event_str: str, listener_event_type: int) -> None:
         global spark
         listener_event = json.loads(listener_event_str)
         if listener_event_type == 0:
diff --git a/python/pyspark/sql/functions/builtin.py 
b/python/pyspark/sql/functions/builtin.py
index 363359c7ebdc..2de5d232ef85 100644
--- a/python/pyspark/sql/functions/builtin.py
+++ b/python/pyspark/sql/functions/builtin.py
@@ -14483,7 +14483,7 @@ def window(
     """
     from pyspark.sql.classic.column import _to_java_column
 
-    def check_string_field(field, fieldName):  # type: ignore[no-untyped-def]
+    def check_string_field(field: Any, fieldName: str) -> None:
         if not field or type(field) is not str:
             raise PySparkTypeError(
                 errorClass="NOT_EXPECTED_TYPE",
diff --git a/python/pyspark/sql/pandas/typehints.py 
b/python/pyspark/sql/pandas/typehints.py
index 18578b94f4dd..cf475d16ac72 100644
--- a/python/pyspark/sql/pandas/typehints.py
+++ b/python/pyspark/sql/pandas/typehints.py
@@ -376,8 +376,8 @@ def infer_eval_type(
 
 
 # infer the eval type for @udf
-def infer_eval_type_for_udf(  # type: ignore[no-untyped-def]
-    f,
+def infer_eval_type_for_udf(
+    f: Callable[..., Any],
 ) -> Optional[
     Union[
         "PandasScalarUDFType",


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to