Spenserrrr opened a new pull request, #57407:
URL: https://github.com/apache/spark/pull/57407
### What changes were proposed in this pull request?
This PR removes `# type: ignore[name-defined]` suppressions (and their paired
`# noqa: F821`) in `pyspark.sql` by resolving the underlying issue rather
than
masking it. It has two parts:
1. **Resolve forward-reference annotations via `TYPE_CHECKING` imports.**
Several
annotations referenced a type that was only imported at call time (inside
a
function body, to avoid a circular import at module load), so it was not
resolvable to the type checker and was suppressed with
`# type: ignore[name-defined] # noqa: F821`. Adding the type to the
module's
`if TYPE_CHECKING:` block makes the forward reference resolve cleanly, so
both
suppressions can be removed:
- `python/pyspark/sql/classic/dataframe.py` -- `DataFrame.plot` return
type
`PySparkPlotAccessor`
- `python/pyspark/sql/connect/dataframe.py` -- `DataFrame.plot` return
type
`PySparkPlotAccessor`
- `python/pyspark/sql/streaming/listener.py` -- `StreamingQueryListener`
`_set_spark_session` / `spark` getter / `spark` setter, annotated with
`SparkSession`
2. **Fix an undefined `sc` in two doctest runners.** In the `_test()`
functions of
`python/pyspark/sql/streaming/query.py` and
`python/pyspark/sql/streaming/listener.py`, the `except Py4JError:`
fallback
calls `SparkSession(sc)`, but `sc` was never defined in the function --
the
`# type: ignore[name-defined] # noqa: F821` was masking a genuinely
undefined
name. The equivalent `_test()` helpers in `readwriter.py` and
`observation.py`
define `sc = SparkContext("local[4]", "PythonTest")` before the `try`;
this PR
applies the same, and drops the suppression.
Two `[name-defined]` suppressions are intentionally NOT touched here; they
are a
separate concern from forward references and covered by part 2 above only
where
`sc` is the culprit.
### Why are the changes needed?
- Part 1: the suppressions hid nothing dangerous, but they added noise and
made
the annotations unresolvable to tooling; resolving them via `TYPE_CHECKING`
documents the real type and lets mypy check it.
- Part 2: this is a latent bug. If `SparkSession._getActiveSessionOrCreate()`
raised `Py4JError`, the fallback would fail with
`NameError: name 'sc' is not defined` instead of creating a session. The
fix
makes the fallback actually work, matching the established pattern in the
other
`_test()` helpers.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
- `mypy --config-file python/mypy.ini` on all touched files reports
`Success: no issues found` with no remaining `[name-defined]` suppressions
in
the changed locations. Stub versions were matched to the CI lint image
(`dev/spark-test-image/lint/Dockerfile`) so results are CI-faithful.
- Runtime imports of the touched modules succeed (no circular-import
regression
from the added `TYPE_CHECKING` imports, which are not evaluated at
runtime).
- The affected doctests run via `python/run-tests`
(`pyspark.sql.streaming.query`, `pyspark.sql.streaming.listener`).
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Opus 4.8)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
