This is an automated email from the ASF dual-hosted git repository.
gaogaotiantian pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 8faf7fd1f8b2 [SPARK-60194][PYTHON] Improve type annotation for
StreamingQueryListener sessions
8faf7fd1f8b2 is described below
commit 8faf7fd1f8b265d9be2b60cf98f12d1e14fc3c39
Author: Tim Hsiung <[email protected]>
AuthorDate: Thu Nov 5 16:34:07 2026 -0800
[SPARK-60194][PYTHON] Improve type annotation for StreamingQueryListener
sessions
### What changes were proposed in this pull request?
- Import the classic and Connect `SparkSession` types under `TYPE_CHECKING`
and annotate the listener session with their union, removing three
`name-defined` ignores and corresponding `F821` suppressions.
- Declare `_sparkSession` with an optional type and a `None` default,
replacing the getter's `getattr` with direct access.
### Why are the changes needed?
`SparkSession` was an unresolved annotation. Listeners accept both classic
and Connect sessions, which are separate types. The explicit default permits
direct access before binding without requiring subclasses to call
`super().__init__()`.
### Does this PR introduce _any_ user-facing change?
Static type checkers can now resolve the listener session type. Runtime
session-binding behavior is unchanged.
### How was this patch tested?
- Mypy passed on `python/pyspark/sql/streaming/listener.py` and
`python/pyspark/sql/connect/streaming/query.py`.
- Local runtime probes using session stand-ins verified initial `None`,
binding, repeated-binding no-overwrite behavior, direct setter assignment,
instance isolation, and subclass initialization without `super()`.
- Ruff lint/format checks and `git diff --check` passed.
No new tests were added. The full streaming integration suite was not run;
runtime validation was limited to session-binding logic.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: pi 1.1.0
Closes #59398 from bearomorphism/typing-streaming-listener-session.
Authored-by: Tim Hsiung <[email protected]>
Signed-off-by: Tian Gao <[email protected]>
---
python/pyspark/sql/streaming/listener.py | 15 ++++++++++-----
1 file changed, 10 insertions(+), 5 deletions(-)
diff --git a/python/pyspark/sql/streaming/listener.py
b/python/pyspark/sql/streaming/listener.py
index 76fec6cf159b..24ea2f2b43bb 100644
--- a/python/pyspark/sql/streaming/listener.py
+++ b/python/pyspark/sql/streaming/listener.py
@@ -17,7 +17,7 @@
import json
import uuid
from abc import ABC, abstractmethod
-from typing import TYPE_CHECKING, Any, Dict, List, Optional, Set
+from typing import TYPE_CHECKING, Any, Dict, List, Optional, Set, Union
from pyspark import cloudpickle
from pyspark.sql.types import Row
@@ -27,6 +27,9 @@ __all__ = ["StreamingQueryListener"]
if TYPE_CHECKING:
from py4j.java_gateway import JavaObject
+ from pyspark.sql.connect.session import SparkSession as ConnectSparkSession
+ from pyspark.sql.session import SparkSession
+
class StreamingQueryListener(ABC):
"""
@@ -63,19 +66,21 @@ class StreamingQueryListener(ABC):
>>> spark.streams.addListener(MyListener())
"""
+ _sparkSession: Optional[Union["SparkSession", "ConnectSparkSession"]] =
None
+
def _set_spark_session(
self,
- session: "SparkSession", # type: ignore[name-defined] # noqa: F821
+ session: Union["SparkSession", "ConnectSparkSession"],
) -> None:
if self.spark is None:
self.spark = session
@property
- def spark(self) -> Optional["SparkSession"]: # type: ignore[name-defined]
# noqa: F821
- return getattr(self, "_sparkSession", None)
+ def spark(self) -> Optional[Union["SparkSession", "ConnectSparkSession"]]:
+ return self._sparkSession
@spark.setter
- def spark(self, session: "SparkSession") -> None: # type:
ignore[name-defined] # noqa: F821
+ def spark(self, session: Union["SparkSession", "ConnectSparkSession"]) ->
None:
# For backward compatibility
self._sparkSession = session
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]