This is an automated email from the ASF dual-hosted git repository.
gaogaotiantian pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 076d39bb3ed1 [SPARK-60174][PYTHON][TEST] Remove type:
ignore[union-attr] in pyspark.sql.worker.utils
076d39bb3ed1 is described below
commit 076d39bb3ed12618b422862664971711329c07a4
Author: Anna <[email protected]>
AuthorDate: Thu Nov 5 12:23:47 2026 -0800
[SPARK-60174][PYTHON][TEST] Remove type: ignore[union-attr] in
pyspark.sql.worker.utils
### What changes were proposed in this pull request?
In `worker_run` (`python/pyspark/sql/worker/utils.py`), read
`sys.modules["__main__"].__spec__` into a local variable and check `spec is not
None` before accessing `spec.name`, instead of suppressing the mypy error with
`# type: ignore[union-attr]`.
Before:
```python
worker_module = (
sys.modules["__main__"].__spec__.name # type: ignore[union-attr]
)
```
After:
```python
spec = sys.modules["__main__"].__spec__
worker_module = spec.name if spec is not None else "__main__"
```
The surrounding `try/except Exception` fallback to `"__main__"` is kept.
### Why are the changes needed?
`__spec__` is declared as `ModuleSpec | None` on `types.ModuleType` in
typeshed (`stdlib/types.pyi`), so mypy reports `union-attr` on `.name`.
At runtime CPython initializes `__spec__` to `None` for a module
(`Objects/moduleobject.c`) and it stays `None` unless the module is loaded
through the import system, for example when run with `python -m`.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
- `dev/lint-python --ruff` and `--mypy` pass. With only the ignore removed,
mypy reports `Item "None" of "ModuleSpec | None" has no attribute "name"
[union-attr]`.
- Compared the old and new expressions with `__spec__` set (via `python
-m`), `__spec__` is `None`, `__main__` missing from `sys.modules`, and no
`__spec__` attribute: identical results in all four.
- Existing `pyspark.sql.tests.test_python_datasource` passes (it does not
exercise the changed lines, since `main.__module__` is the test module).
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Sonnet 5.5)
Closes #59376 from soaring0616/SPARK-60174-worker-utils-union-attr.
Authored-by: Anna <[email protected]>
Signed-off-by: Tian Gao <[email protected]>
---
python/pyspark/sql/worker/utils.py | 5 ++---
1 file changed, 2 insertions(+), 3 deletions(-)
diff --git a/python/pyspark/sql/worker/utils.py
b/python/pyspark/sql/worker/utils.py
index 2069cdc9e6d0..3952bdbfb893 100644
--- a/python/pyspark/sql/worker/utils.py
+++ b/python/pyspark/sql/worker/utils.py
@@ -121,9 +121,8 @@ def worker_run(main: Callable, infile: IO, outfile: IO) ->
None:
if main.__module__ == "__main__":
try:
- worker_module = (
- sys.modules["__main__"].__spec__.name # type:
ignore[union-attr]
- )
+ spec = sys.modules["__main__"].__spec__
+ worker_module = spec.name if spec is not None else
"__main__"
except Exception:
worker_module = "__main__"
else:
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]