This is an automated email from the ASF dual-hosted git repository.

gaogaotiantian pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/master by this push:
     new 076d39bb3ed1 [SPARK-60174][PYTHON][TEST] Remove type: 
ignore[union-attr] in pyspark.sql.worker.utils
076d39bb3ed1 is described below

commit 076d39bb3ed12618b422862664971711329c07a4
Author: Anna <[email protected]>
AuthorDate: Thu Nov 5 12:23:47 2026 -0800

    [SPARK-60174][PYTHON][TEST] Remove type: ignore[union-attr] in 
pyspark.sql.worker.utils
    
    ### What changes were proposed in this pull request?
    In `worker_run` (`python/pyspark/sql/worker/utils.py`), read 
`sys.modules["__main__"].__spec__` into a local variable and check `spec is not 
None` before accessing `spec.name`, instead of suppressing the mypy error with 
`# type: ignore[union-attr]`.
    
    Before:
    ```python
    worker_module = (
        sys.modules["__main__"].__spec__.name  # type: ignore[union-attr]
    )
    ```
    
    After:
    ```python
    spec = sys.modules["__main__"].__spec__
    worker_module = spec.name if spec is not None else "__main__"
    ```
    
    The surrounding `try/except Exception` fallback to `"__main__"` is kept.
    
    ### Why are the changes needed?
    `__spec__` is declared as `ModuleSpec | None` on `types.ModuleType` in 
typeshed (`stdlib/types.pyi`), so mypy reports `union-attr` on `.name`.
    
    At runtime CPython initializes `__spec__` to `None` for a module 
(`Objects/moduleobject.c`) and it stays `None` unless the module is loaded 
through the import system, for example when run with `python -m`.
    
    ### Does this PR introduce _any_ user-facing change?
    No.
    
    ### How was this patch tested?
    - `dev/lint-python --ruff` and `--mypy` pass. With only the ignore removed, 
mypy reports `Item "None" of "ModuleSpec | None" has no attribute "name"  
[union-attr]`.
    - Compared the old and new expressions with `__spec__` set (via `python 
-m`), `__spec__` is `None`, `__main__` missing from `sys.modules`, and no 
`__spec__` attribute: identical results in all four.
    - Existing `pyspark.sql.tests.test_python_datasource` passes (it does not 
exercise the changed lines, since `main.__module__` is the test module).
    
    ### Was this patch authored or co-authored using generative AI tooling?
    Generated-by: Claude Code (Sonnet 5.5)
    
    Closes #59376 from soaring0616/SPARK-60174-worker-utils-union-attr.
    
    Authored-by: Anna <[email protected]>
    Signed-off-by: Tian Gao <[email protected]>
---
 python/pyspark/sql/worker/utils.py | 5 ++---
 1 file changed, 2 insertions(+), 3 deletions(-)

diff --git a/python/pyspark/sql/worker/utils.py 
b/python/pyspark/sql/worker/utils.py
index 2069cdc9e6d0..3952bdbfb893 100644
--- a/python/pyspark/sql/worker/utils.py
+++ b/python/pyspark/sql/worker/utils.py
@@ -121,9 +121,8 @@ def worker_run(main: Callable, infile: IO, outfile: IO) -> 
None:
 
             if main.__module__ == "__main__":
                 try:
-                    worker_module = (
-                        sys.modules["__main__"].__spec__.name  # type: 
ignore[union-attr]
-                    )
+                    spec = sys.modules["__main__"].__spec__
+                    worker_module = spec.name if spec is not None else 
"__main__"
                 except Exception:
                     worker_module = "__main__"
             else:


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to