This is an automated email from the ASF dual-hosted git repository.
gaogaotiantian pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new de4453d42354 [SPARK-58244][PYTHON] Remove type: ignore[name-defined]
for DataFrame.plot and streaming _test() in pyspark.sql
de4453d42354 is described below
commit de4453d42354052e765149d38e0bbb8a5ef67146
Author: Spenser Sun <[email protected]>
AuthorDate: Wed Jul 22 14:56:22 2026 -0700
[SPARK-58244][PYTHON] Remove type: ignore[name-defined] for DataFrame.plot
and streaming _test() in pyspark.sql
### What changes were proposed in this pull request?
This PR removes part of the`# type: ignore[name-defined]` suppressions and
their paired`# noqa: F821` in `pyspark.sql`. It has two parts:
1. `DataFrame.plot` forward-reference annotation. The plot property in
`classic/dataframe.py` and `connect/dataframe.py` returns
`"PySparkPlotAccessor"`, but that type was only imported inside the method
body. Adding it to the module's `if TYPE_CHECKING:` block makes the forward
reference resolve cleanly.
2. **Fix an undefined `sc` in two doctest runners.** In the `_test()`
functions of `python/pyspark/sql/streaming/query.py` and
`python/pyspark/sql/streaming/listener.py`, the `except Py4JError:` fallback
calls `SparkSession(sc)`, but `sc` was never defined in the function. The
equivalent `_test()` helpers in `readwriter.py` and `observation.py` define `sc
= SparkContext("local[4]", "PythonTest")` before the `try`; this PR applies the
same, and drops the suppression.
Note: the [name-defined] suppressions on
`StreamingQueryListener._set_spark_session` and its spark getter/setter are
intentionally left in place. That class is shared by both classic and Connect,
but the session-injection mechanism is Connect-only; annotating it properly
needs a generics-based design.
### Why are the changes needed?
Part 1 documents the real type and lets the type checker verify it, instead
of suppressing an unresolvable annotation. Part 2 fixes a small bug: if
`SparkSession._getActiveSessionOrCreate()` raised `Py4JError`, the fallback
would fail with `NameError: name 'sc' is not defined` instead of creating a
session.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
- `mypy --config-file python/mypy.ini` over python/pyspark tree.
`Success: no issues found` with no remaining `[name-defined]`
suppressions.
- Runtime imports of the touched modules succeed.
- The affected doctests run via `python/run-tests`
(`pyspark.sql.streaming.query`, `pyspark.sql.streaming.listener`).
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Opus 4.8)
Closes #57407 from Spenserrrr/typeignore.
Authored-by: Spenser Sun <[email protected]>
Signed-off-by: Tian Gao <[email protected]>
(cherry picked from commit 7d8a19023a1453e8fff8b1a32b256ae94a1832a2)
Signed-off-by: Tian Gao <[email protected]>
---
python/pyspark/sql/classic/dataframe.py | 3 ++-
python/pyspark/sql/connect/dataframe.py | 3 ++-
python/pyspark/sql/streaming/listener.py | 4 +++-
python/pyspark/sql/streaming/query.py | 4 +++-
4 files changed, 10 insertions(+), 4 deletions(-)
diff --git a/python/pyspark/sql/classic/dataframe.py
b/python/pyspark/sql/classic/dataframe.py
index 9599fad3bcda..eeed6fef44aa 100644
--- a/python/pyspark/sql/classic/dataframe.py
+++ b/python/pyspark/sql/classic/dataframe.py
@@ -99,6 +99,7 @@ if TYPE_CHECKING:
from pyspark.sql.group import GroupedData
from pyspark.sql.observation import Observation
from pyspark.sql.metrics import ExecutionInfo
+ from pyspark.sql.plot import PySparkPlotAccessor
class DataFrame(ParentDataFrame, PandasMapOpsMixin, PandasConversionMixin):
@@ -2009,7 +2010,7 @@ class DataFrame(ParentDataFrame, PandasMapOpsMixin,
PandasConversionMixin):
)
@property
- def plot(self) -> "PySparkPlotAccessor": # type: ignore[name-defined] #
noqa: F821
+ def plot(self) -> "PySparkPlotAccessor":
from pyspark.sql.plot import PySparkPlotAccessor
return PySparkPlotAccessor(self)
diff --git a/python/pyspark/sql/connect/dataframe.py
b/python/pyspark/sql/connect/dataframe.py
index 99ee07709937..093489757115 100644
--- a/python/pyspark/sql/connect/dataframe.py
+++ b/python/pyspark/sql/connect/dataframe.py
@@ -107,6 +107,7 @@ if TYPE_CHECKING:
from pyspark.sql.connect.session import SparkSession
from pyspark.pandas.frame import DataFrame as PandasOnSparkDataFrame
from pyspark.sql.metrics import ExecutionInfo
+ from pyspark.sql.plot import PySparkPlotAccessor
class DataFrame(ParentDataFrame):
@@ -2435,7 +2436,7 @@ class DataFrame(ParentDataFrame):
return self._execution_info
@property
- def plot(self) -> "PySparkPlotAccessor": # type: ignore[name-defined] #
noqa: F821
+ def plot(self) -> "PySparkPlotAccessor":
from pyspark.sql.plot import PySparkPlotAccessor
return PySparkPlotAccessor(self)
diff --git a/python/pyspark/sql/streaming/listener.py
b/python/pyspark/sql/streaming/listener.py
index effd4550d842..8799c4a11373 100644
--- a/python/pyspark/sql/streaming/listener.py
+++ b/python/pyspark/sql/streaming/listener.py
@@ -1115,6 +1115,7 @@ def _test() -> None:
import sys
import doctest
import os
+ from pyspark.core.context import SparkContext
from pyspark.sql import SparkSession
import pyspark.sql.streaming.listener
from py4j.protocol import Py4JError
@@ -1122,10 +1123,11 @@ def _test() -> None:
os.chdir(os.environ["SPARK_HOME"])
globs = pyspark.sql.streaming.listener.__dict__.copy()
+ sc = SparkContext("local[4]", "PythonTest")
try:
spark = SparkSession._getActiveSessionOrCreate()
except Py4JError:
- spark = SparkSession(sc) # type: ignore[name-defined] # noqa: F821
+ spark = SparkSession(sc)
globs["spark"] = spark
diff --git a/python/pyspark/sql/streaming/query.py
b/python/pyspark/sql/streaming/query.py
index b903cca9089f..03d04cbbc4a5 100644
--- a/python/pyspark/sql/streaming/query.py
+++ b/python/pyspark/sql/streaming/query.py
@@ -781,6 +781,7 @@ def _test() -> None:
import doctest
import os
import sys
+ from pyspark.core.context import SparkContext
from pyspark.sql import SparkSession
import pyspark.sql.streaming.query
from py4j.protocol import Py4JError
@@ -788,10 +789,11 @@ def _test() -> None:
os.chdir(os.environ["SPARK_HOME"])
globs = pyspark.sql.streaming.query.__dict__.copy()
+ sc = SparkContext("local[4]", "PythonTest")
try:
spark = SparkSession._getActiveSessionOrCreate()
except Py4JError:
- spark = SparkSession(sc) # type: ignore[name-defined] # noqa: F821
+ spark = SparkSession(sc)
globs["spark"] = spark
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
