This is an automated email from the ASF dual-hosted git repository.

HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git


The following commit(s) were added to refs/heads/master by this push:
     new 4b05e8da [SPARK-61151][CONNECT][PYTHON] Vendor 
pyspark.sql.streaming.listener, of the listener and its event and progress 
classes, in the Python client as PySpark has it
4b05e8da is described below

commit 4b05e8daa1db50af4f04adf6797315d1a2646371
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Sun Nov 8 14:14:23 2026 +0900

    [SPARK-61151][CONNECT][PYTHON] Vendor pyspark.sql.streaming.listener, of 
the listener and its event and progress classes, in the Python client as 
PySpark has it
    
    ### What changes were proposed in this pull request?
    
    [SPARK-61151](https://issues.apache.org/jira/browse/SPARK-61151): The 
client's `pyspark.sql.streaming.listener` was its own rewrite of PySpark's: 699 
lines, of which most differ from PySpark 4.3.0's 1145. The module is pure 
Python and needs nothing of the client, so it is now PySpark 4.3.0's, vendored 
verbatim, and `check_vendored_upstream.sh` checks it.
    
    So, as PySpark's:
    - `StreamingQueryProgress.numInputRows`, `inputRowsPerSecond` and 
`processedRowsPerSecond` sum those of the sources when the JSON of the server 
leaves them out, as it does
    - the items of a progress, such as `progress["id"]`, are those of its JSON: 
a string, rather than a `uuid.UUID`
    - `StreamingQueryListener` has `spark` and `_set_spark_session`
    - `__all__` is `["StreamingQueryListener"]`
    
    The client builds the events and the progresses with their `fromJson`, 
which PySpark's module has too.
    
    ### Why are the changes needed?
    
    - `q.lastProgress.numInputRows` and `q.recentProgress[i].numInputRows` were 
`None`.
    - `lastProgress["id"] == q.id` was `False`, a `uuid.UUID` against a string.
    - A listener had no `spark`.
    
    Apache Spark's tests failed on this client:
    - `test_batch_dataframe_in_batch_session_sql_is_supported` and 
`test_batch_dataframe_join_in_batch_session_is_supported` of 
`test_parity_foreach_batch.py` (`'>' not supported between instances of 
'NoneType' and 'int'`)
    - `test_stream_status_and_progress`, `test_streaming_progress` and 
`test_streaming_query_functions_basic` of `test_parity_streaming.py` 
(`UUID(...) != '...'`)
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes. The progresses, the events and the listeners of streaming queries are 
those of PySpark 4.3.0.
    
    ### How was this patch tested?
    
    - A new offline test parses the progress of the server, of two sources: its 
totals of rows and rates, those of the sources and the sink, the event of a 
progress, `__all__`, and the `spark` of a listener. It passes with official 
PySpark 4.3.0, and fails without this change.
    - A new live test checks that the progress of a `rate` query sums the rows 
and the rates of its sources. It fails without this change.
    - The progress of a `rate` query reads the same as with official PySpark 
4.3.0 (`numInputRows`, `inputRowsPerSecond`, `sources`, the keys of its JSON).
    - Apache Spark's `test_parity_streaming.py` runs on this client against a 
local server: 29 tests pass, and 1, `test_query_manager_no_recreation`, is 
fixed by SPARK-61162. Before this change, 4 failed. The two tests of 
`test_parity_foreach_batch.py` above pass.
    - The lib tests of both crates (596 passed), the build of all tests, also 
with `wasm-udf`, the offline Python suites (2213 passed, 8 skipped), and the 
live drop-in and Arrow suites (197 passed) and live cases (10 passed) against a 
local Spark 4.3.0 server. Clippy reports no diagnostics in the changed lines.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Opus 5.5
    
    This pull request and its description were written by Isaac.
    
    Closes #350 from HyukjinKwon/SPARK-61151-streaming-listener.
    
    Authored-by: Hyukjin Kwon <[email protected]>
    Signed-off-by: Hyukjin Kwon <[email protected]>
---
 python/pyspark/sql/streaming/listener.py | 986 ++++++++++++++++++++++---------
 python/tests/test_dropin_e2e.py          |  18 +
 python/tests/test_dropin_offline.py      |  44 ++
 scripts/check_vendored_upstream.sh       |   3 +
 4 files changed, 781 insertions(+), 270 deletions(-)

diff --git a/python/pyspark/sql/streaming/listener.py 
b/python/pyspark/sql/streaming/listener.py
index 7733dd3e..8799c4a1 100644
--- a/python/pyspark/sql/streaming/listener.py
+++ b/python/pyspark/sql/streaming/listener.py
@@ -14,87 +14,167 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 #
-
-"""
-Streaming Query Listener - defines listener interface and event classes
-for streaming query lifecycle management.
-
-This module vendors classes from pyspark.sql.streaming.listener for use
-in the Connect streaming client.
-"""
-
-import json
 import uuid
-from abc import ABC, abstractmethod
+import json
 from typing import Any, Dict, List, Optional, Set, TYPE_CHECKING
+from abc import ABC, abstractmethod
+
 from pyspark.sql import Row
+from pyspark import cloudpickle
 
-if TYPE_CHECKING:
-    pass
+__all__ = ["StreamingQueryListener"]
 
-__all__ = [
-    "StreamingQueryListener",
-    "QueryStartedEvent",
-    "QueryProgressEvent",
-    "QueryIdleEvent",
-    "QueryTerminatedEvent",
-    "StreamingQueryProgress",
-    "StateOperatorProgress",
-    "SourceProgress",
-    "SinkProgress",
-]
+if TYPE_CHECKING:
+    from py4j.java_gateway import JavaObject
 
 
 class StreamingQueryListener(ABC):
     """
-    Interface for listening to events related to StreamingQuery.
+    Interface for listening to events related to 
:class:`~pyspark.sql.streaming.StreamingQuery`.
+
+    .. versionadded:: 3.4.0
 
     Notes
     -----
     The methods are not thread-safe as they may be called from different 
threads.
-    The events received are identical with Scala API.
+    The events received are identical with Scala API. Refer to its 
documentation.
+
+    This API is evolving.
 
     Examples
     --------
     >>> class MyListener(StreamingQueryListener):
     ...    def onQueryStarted(self, event: QueryStartedEvent) -> None:
+    ...        # Do something with event.
     ...        pass
     ...
     ...    def onQueryProgress(self, event: QueryProgressEvent) -> None:
+    ...        # Do something with event.
     ...        pass
     ...
     ...    def onQueryIdle(self, event: QueryIdleEvent) -> None:
+    ...        # Do something with event.
     ...        pass
     ...
     ...    def onQueryTerminated(self, event: QueryTerminatedEvent) -> None:
+    ...        # Do something with event.
     ...        pass
+    ...
+    >>> spark.streams.addListener(MyListener())
     """
 
+    def _set_spark_session(
+        self,
+        session: "SparkSession",  # type: ignore[name-defined] # noqa: F821
+    ) -> None:
+        if self.spark is None:
+            self.spark = session
+
+    @property
+    def spark(self) -> Optional["SparkSession"]:  # type: ignore[name-defined] 
# noqa: F821
+        return getattr(self, "_sparkSession", None)
+
+    @spark.setter
+    def spark(self, session: "SparkSession") -> None:  # type: 
ignore[name-defined] # noqa: F821
+        # For backward compatibility
+        self._sparkSession = session
+
     def _init_listener_id(self) -> None:
         self._id = str(uuid.uuid4())
 
     @abstractmethod
     def onQueryStarted(self, event: "QueryStartedEvent") -> None:
-        """Called when a query is started."""
+        """
+        Called when a query is started.
+
+        Notes
+        -----
+        This is called synchronously with 
:py:meth:`~pyspark.sql.streaming.DataStreamWriter.start`,
+        that is, `onQueryStart` will be called on all listeners before 
`DataStreamWriter.start()`
+        returns the corresponding 
:class:`~pyspark.sql.streaming.StreamingQuery`.
+        Please don't block this method as it will block your query.
+        """
         pass
 
     @abstractmethod
     def onQueryProgress(self, event: "QueryProgressEvent") -> None:
-        """Called when there is some status update."""
+        """
+        Called when there is some status update (ingestion rate updated, etc.)
+
+        Notes
+        -----
+        This method is asynchronous. The status in 
:class:`~pyspark.sql.streaming.StreamingQuery`
+        will always be latest no matter when this method is called. Therefore, 
the status of
+        :class:`~pyspark.sql.streaming.StreamingQuery`.
+        may be changed before/when you process the event. E.g., you may find
+        :class:`~pyspark.sql.streaming.StreamingQuery` is terminated when you 
are
+        processing `QueryProgressEvent`.
+        """
         pass
 
+    # NOTE: Do not mark this as abstract method, since we released this 
abstract class without
+    # this method in prior version and marking this as abstract method would 
break existing
+    # implementations.
     def onQueryIdle(self, event: "QueryIdleEvent") -> None:
-        """Called when the query is idle and waiting for new data to 
process."""
+        """
+        Called when the query is idle and waiting for new data to process.
+        """
         pass
 
     @abstractmethod
     def onQueryTerminated(self, event: "QueryTerminatedEvent") -> None:
-        """Called when a query is stopped, with or without error."""
+        """
+        Called when a query is stopped, with or without error.
+        """
         pass
 
+    @property
+    def _jlistener(self) -> "JavaObject":
+        from pyspark import SparkContext
+
+        if hasattr(self, "_jlistenerobj"):
+            return self._jlistenerobj
+
+        self._jlistenerobj: "JavaObject" = 
SparkContext._jvm.PythonStreamingQueryListenerWrapper(  # type: 
ignore[union-attr]
+            JStreamingQueryListener(self)
+        )
+        return self._jlistenerobj
+
+
+class JStreamingQueryListener:
+    """
+    Python class that implements Java interface by Py4J.
+    """
+
+    def __init__(self, pylistener: StreamingQueryListener) -> None:
+        self.pylistener = pylistener
+
+    def onQueryStarted(self, jevent: "JavaObject") -> None:
+        self.pylistener.onQueryStarted(QueryStartedEvent.fromJObject(jevent))
+
+    def onQueryProgress(self, jevent: "JavaObject") -> None:
+        self.pylistener.onQueryProgress(QueryProgressEvent.fromJObject(jevent))
+
+    def onQueryIdle(self, jevent: "JavaObject") -> None:
+        self.pylistener.onQueryIdle(QueryIdleEvent.fromJObject(jevent))
+
+    def onQueryTerminated(self, jevent: "JavaObject") -> None:
+        
self.pylistener.onQueryTerminated(QueryTerminatedEvent.fromJObject(jevent))
+
+    class Java:
+        implements = 
["org.apache.spark.sql.streaming.PythonStreamingQueryListener"]
+
 
 class QueryStartedEvent:
-    """Event representing the start of a query."""
+    """
+    Event representing the start of a query.
+
+    .. versionadded:: 3.4.0
+
+    Notes
+    -----
+    This API is evolving.
+    """
 
     def __init__(
         self,
@@ -110,61 +190,111 @@ class QueryStartedEvent:
         self._timestamp: str = timestamp
         self._jobTags: Set[str] = jobTags
 
+    @classmethod
+    def fromJObject(cls, jevent: "JavaObject") -> "QueryStartedEvent":
+        job_tags = set()
+        java_iterator = jevent.jobTags().iterator()
+        while java_iterator.hasNext():
+            job_tags.add(java_iterator.next().toString())
+
+        return cls(
+            id=uuid.UUID(jevent.id().toString()),
+            runId=uuid.UUID(jevent.runId().toString()),
+            name=jevent.name(),
+            timestamp=jevent.timestamp(),
+            jobTags=job_tags,
+        )
+
     @classmethod
     def fromJson(cls, j: Dict[str, Any]) -> "QueryStartedEvent":
-        job_tags = j.get("jobTags", [])
+        # Json4s will convert jobTags to a list, so we need to convert it back 
to a set.
+        job_tags = j["jobTags"] if "jobTags" in j else []
         return cls(
             id=uuid.UUID(j["id"]),
             runId=uuid.UUID(j["runId"]),
-            name=j.get("name"),
-            timestamp=j.get("timestamp", ""),
+            name=j["name"],
+            timestamp=j["timestamp"],
             jobTags=set(job_tags),
         )
 
     @property
     def id(self) -> uuid.UUID:
-        """A unique query id that persists across restarts."""
+        """
+        A unique query id that persists across restarts. See
+        py:meth:`~pyspark.sql.streaming.StreamingQuery.id`.
+        """
         return self._id
 
     @property
     def runId(self) -> uuid.UUID:
-        """A query id that is unique for every start/restart."""
+        """
+        A query id that is unique for every start/restart. See
+        py:meth:`~pyspark.sql.streaming.StreamingQuery.runId`.
+        """
         return self._runId
 
     @property
     def name(self) -> Optional[str]:
-        """User-specified name of the query, `None` if not specified."""
+        """
+        User-specified name of the query, `None` if not specified.
+        """
         return self._name
 
     @property
     def timestamp(self) -> str:
-        """The timestamp to start a query."""
+        """
+        The timestamp to start a query.
+        """
         return self._timestamp
 
     @property
     def jobTags(self) -> Set[str]:
-        """The job tags of the query."""
+        """
+        The job tags of the query.
+        """
         return self._jobTags
 
 
 class QueryProgressEvent:
-    """Event representing any progress updates in a query."""
+    """
+    Event representing any progress updates in a query.
+
+    .. versionadded:: 3.4.0
+
+    Notes
+    -----
+    This API is evolving.
+    """
 
     def __init__(self, progress: "StreamingQueryProgress") -> None:
         self._progress: StreamingQueryProgress = progress
 
+    @classmethod
+    def fromJObject(cls, jevent: "JavaObject") -> "QueryProgressEvent":
+        return 
cls(progress=StreamingQueryProgress.fromJObject(jevent.progress()))
+
     @classmethod
     def fromJson(cls, j: Dict[str, Any]) -> "QueryProgressEvent":
-        return cls(progress=StreamingQueryProgress.fromJson(j.get("progress", 
j)))
+        return cls(progress=StreamingQueryProgress.fromJson(j["progress"]))
 
     @property
     def progress(self) -> "StreamingQueryProgress":
-        """The query progress updates."""
+        """
+        The query progress updates.
+        """
         return self._progress
 
 
 class QueryIdleEvent:
-    """Event representing that query is idle and waiting for new data to 
process."""
+    """
+    Event representing that query is idle and waiting for new data to process.
+
+    .. versionadded:: 3.5.0
+
+    Notes
+    -----
+    This API is evolving.
+    """
 
     def __init__(self, id: uuid.UUID, runId: uuid.UUID, timestamp: str) -> 
None:
         self._id: uuid.UUID = id
@@ -172,31 +302,51 @@ class QueryIdleEvent:
         self._timestamp: str = timestamp
 
     @classmethod
-    def fromJson(cls, j: Dict[str, Any]) -> "QueryIdleEvent":
+    def fromJObject(cls, jevent: "JavaObject") -> "QueryIdleEvent":
         return cls(
-            id=uuid.UUID(j["id"]),
-            runId=uuid.UUID(j["runId"]),
-            timestamp=j.get("timestamp", ""),
+            id=uuid.UUID(jevent.id().toString()),
+            runId=uuid.UUID(jevent.runId().toString()),
+            timestamp=jevent.timestamp(),
         )
 
+    @classmethod
+    def fromJson(cls, j: Dict[str, Any]) -> "QueryIdleEvent":
+        return cls(id=uuid.UUID(j["id"]), runId=uuid.UUID(j["runId"]), 
timestamp=j["timestamp"])
+
     @property
     def id(self) -> uuid.UUID:
-        """A unique query id that persists across restarts."""
+        """
+        A unique query id that persists across restarts. See
+        py:meth:`~pyspark.sql.streaming.StreamingQuery.id`.
+        """
         return self._id
 
     @property
     def runId(self) -> uuid.UUID:
-        """A query id that is unique for every start/restart."""
+        """
+        A query id that is unique for every start/restart. See
+        py:meth:`~pyspark.sql.streaming.StreamingQuery.runId`.
+        """
         return self._runId
 
     @property
     def timestamp(self) -> str:
-        """The timestamp when the latest no-batch trigger happened."""
+        """
+        The timestamp when the latest no-batch trigger happened.
+        """
         return self._timestamp
 
 
 class QueryTerminatedEvent:
-    """Event representing that termination of a query."""
+    """
+    Event representing that termination of a query.
+
+    .. versionadded:: 3.4.0
+
+    Notes
+    -----
+    This API is evolving.
+    """
 
     def __init__(
         self,
@@ -210,38 +360,359 @@ class QueryTerminatedEvent:
         self._exception: Optional[str] = exception
         self._errorClassOnException: Optional[str] = errorClassOnException
 
+    @classmethod
+    def fromJObject(cls, jevent: "JavaObject") -> "QueryTerminatedEvent":
+        jexception = jevent.exception()
+        jerrorclass = jevent.errorClassOnException()
+        return cls(
+            id=uuid.UUID(jevent.id().toString()),
+            runId=uuid.UUID(jevent.runId().toString()),
+            exception=jexception.get() if jexception.isDefined() else None,
+            errorClassOnException=jerrorclass.get() if jerrorclass.isDefined() 
else None,
+        )
+
     @classmethod
     def fromJson(cls, j: Dict[str, Any]) -> "QueryTerminatedEvent":
         return cls(
             id=uuid.UUID(j["id"]),
             runId=uuid.UUID(j["runId"]),
-            exception=j.get("exception"),
-            errorClassOnException=j.get("errorClassOnException"),
+            exception=j["exception"],
+            errorClassOnException=j["errorClassOnException"],
         )
 
     @property
     def id(self) -> uuid.UUID:
-        """A unique query id that persists across restarts."""
+        """
+        A unique query id that persists across restarts. See
+        py:meth:`~pyspark.sql.streaming.StreamingQuery.id`.
+        """
         return self._id
 
     @property
     def runId(self) -> uuid.UUID:
-        """A query id that is unique for every start/restart."""
+        """
+        A query id that is unique for every start/restart. See
+        py:meth:`~pyspark.sql.streaming.StreamingQuery.runId`.
+        """
         return self._runId
 
     @property
     def exception(self) -> Optional[str]:
-        """The exception message if the query terminated with an exception."""
+        """
+        The exception message of the query if the query was terminated
+        with an exception. Otherwise, it will be `None`.
+        """
         return self._exception
 
     @property
     def errorClassOnException(self) -> Optional[str]:
-        """The error class from the exception if applicable."""
+        """
+        The error class from the exception if the query was terminated
+        with an exception which is a part of error class framework.
+        If the query was terminated without an exception, or the
+        exception is not a part of error class framework, it will be
+        `None`.
+
+        .. versionadded:: 3.5.0
+        """
         return self._errorClassOnException
 
 
+class StreamingQueryProgress(dict):
+    """
+    .. versionadded:: 3.4.0
+
+    .. versionchanged:: 4.0.0
+        Becomes a subclass of dict
+
+    Notes
+    -----
+    This API is evolving.
+    """
+
+    def __init__(
+        self,
+        id: uuid.UUID,
+        runId: uuid.UUID,
+        name: Optional[str],
+        timestamp: str,
+        batchId: int,
+        batchDuration: int,
+        durationMs: Dict[str, int],
+        eventTime: Dict[str, str],
+        stateOperators: List["StateOperatorProgress"],
+        sources: List["SourceProgress"],
+        sink: "SinkProgress",
+        numInputRows: Optional[int],
+        inputRowsPerSecond: Optional[float],
+        processedRowsPerSecond: Optional[float],
+        observedMetrics: Dict[str, Row],
+        jprogress: Optional["JavaObject"] = None,
+        jdict: Optional[Dict[str, Any]] = None,
+    ):
+        super().__init__(
+            id=id,
+            runId=runId,
+            name=name,
+            timestamp=timestamp,
+            batchId=batchId,
+            batchDuration=batchDuration,
+            durationMs=durationMs,
+            eventTime=eventTime,
+            stateOperators=stateOperators,
+            sources=sources,
+            sink=sink,
+            numInputRows=numInputRows,
+            inputRowsPerSecond=inputRowsPerSecond,
+            processedRowsPerSecond=processedRowsPerSecond,
+            observedMetrics=observedMetrics,
+        )
+        self._jprogress: Optional["JavaObject"] = jprogress
+        self._jdict: Optional[Dict[str, Any]] = jdict
+
+    @classmethod
+    def fromJObject(cls, jprogress: "JavaObject") -> "StreamingQueryProgress":
+        from pyspark import SparkContext
+
+        return cls(
+            jprogress=jprogress,
+            id=uuid.UUID(jprogress.id().toString()),
+            runId=uuid.UUID(jprogress.runId().toString()),
+            name=jprogress.name(),
+            timestamp=jprogress.timestamp(),
+            batchId=jprogress.batchId(),
+            batchDuration=jprogress.batchDuration(),
+            durationMs=dict(jprogress.durationMs()),
+            eventTime=dict(jprogress.eventTime()),
+            stateOperators=[
+                StateOperatorProgress.fromJObject(js) for js in 
jprogress.stateOperators()
+            ],
+            sources=[SourceProgress.fromJObject(js) for js in 
jprogress.sources()],
+            sink=SinkProgress.fromJObject(jprogress.sink()),
+            numInputRows=jprogress.numInputRows(),
+            inputRowsPerSecond=jprogress.inputRowsPerSecond(),
+            processedRowsPerSecond=jprogress.processedRowsPerSecond(),
+            observedMetrics={
+                k: cloudpickle.loads(
+                    SparkContext._jvm.PythonSQLUtils.toPyRow(jr)  # type: 
ignore[union-attr]
+                )
+                for k, jr in dict(jprogress.observedMetrics()).items()
+            },
+        )
+
+    @classmethod
+    def fromJson(cls, j: Dict[str, Any]) -> "StreamingQueryProgress":
+        return cls(
+            jdict=j,
+            id=uuid.UUID(j["id"]),
+            runId=uuid.UUID(j["runId"]),
+            name=j["name"],
+            timestamp=j["timestamp"],
+            batchId=j["batchId"],
+            batchDuration=j["batchDuration"],
+            durationMs=dict(j["durationMs"]) if "durationMs" in j else {},
+            eventTime=dict(j["eventTime"]) if "eventTime" in j else {},
+            stateOperators=[StateOperatorProgress.fromJson(s) for s in 
j["stateOperators"]],
+            sources=[SourceProgress.fromJson(s) for s in j["sources"]],
+            sink=SinkProgress.fromJson(j["sink"]),
+            numInputRows=j["numInputRows"] if "numInputRows" in j else None,
+            inputRowsPerSecond=j["inputRowsPerSecond"] if "inputRowsPerSecond" 
in j else None,
+            processedRowsPerSecond=(
+                j["processedRowsPerSecond"] if "processedRowsPerSecond" in j 
else None
+            ),
+            observedMetrics=(
+                {
+                    k: Row(*row_dict.keys())(*row_dict.values())  # Assume no 
nested rows
+                    for k, row_dict in j["observedMetrics"].items()
+                }
+                if "observedMetrics" in j
+                else {}
+            ),
+        )
+
+    @property
+    def id(self) -> uuid.UUID:
+        """
+        A unique query id that persists across restarts. See
+        py:meth:`~pyspark.sql.streaming.StreamingQuery.id`.
+        """
+        # Before Spark 4.0, StreamingQuery.lastProgress returns a dict, which 
casts id and runId
+        # to string. But here they are UUID.
+        # To prevent breaking change, do not cast them to string when accessed 
with attribute.
+        return super().__getitem__("id")
+
+    @property
+    def runId(self) -> uuid.UUID:
+        """
+        A query id that is unique for every start/restart. See
+        py:meth:`~pyspark.sql.streaming.StreamingQuery.runId`.
+        """
+        # Before Spark 4.0, StreamingQuery.lastProgress returns a dict, which 
casts id and runId
+        # to string. But here they are UUID.
+        # To prevent breaking change, do not cast them to string when accessed 
with attribute.
+        return super().__getitem__("runId")
+
+    @property
+    def name(self) -> Optional[str]:
+        """
+        User-specified name of the query, `None` if not specified.
+        """
+        return self["name"]
+
+    @property
+    def timestamp(self) -> str:
+        """
+        The timestamp to start a query.
+        """
+        return self["timestamp"]
+
+    @property
+    def batchId(self) -> int:
+        """
+        A unique id for the current batch of data being processed.  Note that 
in the
+        case of retries after a failure a given batchId my be executed more 
than once.
+        Similarly, when there is no data to be processed, the batchId will not 
be
+        incremented.
+        """
+        return self["batchId"]
+
+    @property
+    def batchDuration(self) -> int:
+        """
+        The process duration of each batch.
+        """
+        return self["batchDuration"]
+
+    @property
+    def durationMs(self) -> Dict[str, int]:
+        """
+        The amount of time taken to perform various operations in milliseconds.
+        """
+        return self["durationMs"]
+
+    @property
+    def eventTime(self) -> Dict[str, str]:
+        """
+        Statistics of event time seen in this batch. It may contain the 
following keys:
+
+        .. code-block:: python
+
+            {
+                "max": "2016-12-05T20:54:20.827Z",  # maximum event time seen 
in this trigger
+                "min": "2016-12-05T20:54:20.827Z",  # minimum event time seen 
in this trigger
+                "avg": "2016-12-05T20:54:20.827Z",  # average event time seen 
in this trigger
+                "watermark": "2016-12-05T20:54:20.827Z"  # watermark used in 
this trigger
+            }
+
+        All timestamps are in ISO8601 format, i.e. UTC timestamps.
+        """
+        return self["eventTime"]
+
+    @property
+    def stateOperators(self) -> List["StateOperatorProgress"]:
+        """
+        Information about operators in the query that store state.
+        """
+        return self["stateOperators"]
+
+    @property
+    def sources(self) -> List["SourceProgress"]:
+        """
+        detailed statistics on data being read from each of the streaming 
sources.
+        """
+        return self["sources"]
+
+    @property
+    def sink(self) -> "SinkProgress":
+        """
+        A unique query id that persists across restarts. See
+        py:meth:`~pyspark.sql.streaming.StreamingQuery.id`.
+        """
+        return self["sink"]
+
+    @property
+    def observedMetrics(self) -> Dict[str, Row]:
+        return self["observedMetrics"]
+
+    @property
+    def numInputRows(self) -> int:
+        """
+        The aggregate (across all sources) number of records processed in a 
trigger.
+        """
+        if self["numInputRows"] is not None:
+            return self["numInputRows"]
+        else:
+            return sum(s.numInputRows for s in self.sources)
+
+    @property
+    def inputRowsPerSecond(self) -> float:
+        """
+        The aggregate (across all sources) rate of data arriving.
+        """
+        if self["inputRowsPerSecond"] is not None:
+            return self["inputRowsPerSecond"]
+        else:
+            return sum(s.inputRowsPerSecond for s in self.sources)
+
+    @property
+    def processedRowsPerSecond(self) -> float:
+        """
+        The aggregate (across all sources) rate at which Spark is processing 
data.
+        """
+        if self["processedRowsPerSecond"] is not None:
+            return self["processedRowsPerSecond"]
+        else:
+            return sum(s.processedRowsPerSecond for s in self.sources)
+
+    @property
+    def json(self) -> str:
+        """
+        The compact JSON representation of this progress.
+        """
+        assert self._jdict is not None or self._jprogress is not None
+        if self._jprogress:
+            return self._jprogress.json()
+        else:
+            return json.dumps(self._jdict)
+
+    @property
+    def prettyJson(self) -> str:
+        """
+        The pretty (i.e. indented) JSON representation of this progress.
+        """
+        assert self._jdict is not None or self._jprogress is not None
+        if self._jprogress:
+            return self._jprogress.prettyJson()
+        else:
+            return json.dumps(self._jdict, indent=4)
+
+    def __getitem__(self, key: str) -> Any:
+        # Before Spark 4.0, StreamingQuery.lastProgress returns a dict, which 
casts id and runId
+        # to string. But here they are UUID.
+        # To prevent breaking change, also cast them to string when accessed 
with __getitem__.
+        if key == "id" or key == "runId":
+            return str(super().__getitem__(key))
+        else:
+            return super().__getitem__(key)
+
+    def __str__(self) -> str:
+        return self.prettyJson
+
+    def __repr__(self) -> str:
+        return self.prettyJson
+
+
 class StateOperatorProgress(dict):
-    """Progress information for state operators."""
+    """
+    .. versionadded:: 3.4.0
+
+    .. versionchanged:: 4.0.0
+        Becomes a subclass of dict
+
+    Notes
+    -----
+    This API is evolving.
+    """
 
     def __init__(
         self,
@@ -257,6 +728,7 @@ class StateOperatorProgress(dict):
         numShufflePartitions: int,
         numStateStoreInstances: int,
         customMetrics: Dict[str, int],
+        jprogress: Optional["JavaObject"] = None,
         jdict: Optional[Dict[str, Any]] = None,
     ):
         super().__init__(
@@ -273,24 +745,43 @@ class StateOperatorProgress(dict):
             numStateStoreInstances=numStateStoreInstances,
             customMetrics=customMetrics,
         )
+        self._jprogress: Optional["JavaObject"] = jprogress
         self._jdict: Optional[Dict[str, Any]] = jdict
 
+    @classmethod
+    def fromJObject(cls, jprogress: "JavaObject") -> "StateOperatorProgress":
+        return cls(
+            jprogress=jprogress,
+            operatorName=jprogress.operatorName(),
+            numRowsTotal=jprogress.numRowsTotal(),
+            numRowsUpdated=jprogress.numRowsUpdated(),
+            allUpdatesTimeMs=jprogress.allUpdatesTimeMs(),
+            numRowsRemoved=jprogress.numRowsRemoved(),
+            allRemovalsTimeMs=jprogress.allRemovalsTimeMs(),
+            commitTimeMs=jprogress.commitTimeMs(),
+            memoryUsedBytes=jprogress.memoryUsedBytes(),
+            numRowsDroppedByWatermark=jprogress.numRowsDroppedByWatermark(),
+            numShufflePartitions=jprogress.numShufflePartitions(),
+            numStateStoreInstances=jprogress.numStateStoreInstances(),
+            customMetrics=dict(jprogress.customMetrics()),
+        )
+
     @classmethod
     def fromJson(cls, j: Dict[str, Any]) -> "StateOperatorProgress":
         return cls(
             jdict=j,
-            operatorName=j.get("operatorName", ""),
-            numRowsTotal=j.get("numRowsTotal", 0),
-            numRowsUpdated=j.get("numRowsUpdated", 0),
-            numRowsRemoved=j.get("numRowsRemoved", 0),
-            allUpdatesTimeMs=j.get("allUpdatesTimeMs", 0),
-            allRemovalsTimeMs=j.get("allRemovalsTimeMs", 0),
-            commitTimeMs=j.get("commitTimeMs", 0),
-            memoryUsedBytes=j.get("memoryUsedBytes", 0),
-            numRowsDroppedByWatermark=j.get("numRowsDroppedByWatermark", 0),
-            numShufflePartitions=j.get("numShufflePartitions", 0),
-            numStateStoreInstances=j.get("numStateStoreInstances", 0),
-            customMetrics=dict(j.get("customMetrics", {})),
+            operatorName=j["operatorName"],
+            numRowsTotal=j["numRowsTotal"],
+            numRowsUpdated=j["numRowsUpdated"],
+            numRowsRemoved=j["numRowsRemoved"],
+            allUpdatesTimeMs=j["allUpdatesTimeMs"],
+            allRemovalsTimeMs=j["allRemovalsTimeMs"],
+            commitTimeMs=j["commitTimeMs"],
+            memoryUsedBytes=j["memoryUsedBytes"],
+            numRowsDroppedByWatermark=j["numRowsDroppedByWatermark"],
+            numShufflePartitions=j["numShufflePartitions"],
+            numStateStoreInstances=j["numStateStoreInstances"],
+            customMetrics=dict(j["customMetrics"]) if "customMetrics" in j 
else {},
         )
 
     @property
@@ -343,19 +834,25 @@ class StateOperatorProgress(dict):
 
     @property
     def json(self) -> str:
-        """The compact JSON representation of this progress."""
-        if self._jdict:
-            return json.dumps(self._jdict)
+        """
+        The compact JSON representation of this progress.
+        """
+        assert self._jdict is not None or self._jprogress is not None
+        if self._jprogress:
+            return self._jprogress.json()
         else:
-            return json.dumps(dict(self))
+            return json.dumps(self._jdict)
 
     @property
     def prettyJson(self) -> str:
-        """The pretty (indented) JSON representation of this progress."""
-        if self._jdict:
-            return json.dumps(self._jdict, indent=4)
+        """
+        The pretty (i.e. indented) JSON representation of this progress.
+        """
+        assert self._jdict is not None or self._jprogress is not None
+        if self._jprogress:
+            return self._jprogress.prettyJson()
         else:
-            return json.dumps(dict(self), indent=4)
+            return json.dumps(self._jdict, indent=4)
 
     def __str__(self) -> str:
         return self.prettyJson
@@ -365,7 +862,16 @@ class StateOperatorProgress(dict):
 
 
 class SourceProgress(dict):
-    """Progress information for data sources."""
+    """
+    .. versionadded:: 3.4.0
+
+    .. versionchanged:: 4.0.0
+        Becomes a subclass of dict
+
+    Notes
+    -----
+    This API is evolving.
+    """
 
     def __init__(
         self,
@@ -377,6 +883,7 @@ class SourceProgress(dict):
         inputRowsPerSecond: float,
         processedRowsPerSecond: float,
         metrics: Dict[str, str],
+        jprogress: Optional["JavaObject"] = None,
         jdict: Optional[Dict[str, Any]] = None,
     ) -> None:
         super().__init__(
@@ -389,11 +896,28 @@ class SourceProgress(dict):
             processedRowsPerSecond=processedRowsPerSecond,
             metrics=metrics,
         )
+        self._jprogress: Optional["JavaObject"] = jprogress
         self._jdict: Optional[Dict[str, Any]] = jdict
 
+    @classmethod
+    def fromJObject(cls, jprogress: "JavaObject") -> "SourceProgress":
+        return cls(
+            jprogress=jprogress,
+            description=jprogress.description(),
+            startOffset=str(jprogress.startOffset()),
+            endOffset=str(jprogress.endOffset()),
+            latestOffset=str(jprogress.latestOffset()),
+            numInputRows=jprogress.numInputRows(),
+            inputRowsPerSecond=jprogress.inputRowsPerSecond(),
+            processedRowsPerSecond=jprogress.processedRowsPerSecond(),
+            metrics=dict(jprogress.metrics()),
+        )
+
     @classmethod
     def fromJson(cls, j: Dict[str, Any]) -> "SourceProgress":
         def _to_json_string(value: Any) -> str:
+            """Convert offset value to JSON string. If already a string, 
return as-is.
+            If a dict/list, JSON-encode it."""
             if isinstance(value, str):
                 return value
             else:
@@ -401,49 +925,63 @@ class SourceProgress(dict):
 
         return cls(
             jdict=j,
-            description=j.get("description", ""),
-            startOffset=_to_json_string(j.get("startOffset", "")),
-            endOffset=_to_json_string(j.get("endOffset", "")),
-            latestOffset=_to_json_string(j.get("latestOffset", "")),
-            numInputRows=j.get("numInputRows", 0),
-            inputRowsPerSecond=j.get("inputRowsPerSecond", 0.0),
-            processedRowsPerSecond=j.get("processedRowsPerSecond", 0.0),
-            metrics=dict(j.get("metrics", {})),
+            description=j["description"],
+            startOffset=_to_json_string(j["startOffset"]),
+            endOffset=_to_json_string(j["endOffset"]),
+            latestOffset=_to_json_string(j["latestOffset"]),
+            numInputRows=j["numInputRows"],
+            inputRowsPerSecond=j["inputRowsPerSecond"],
+            processedRowsPerSecond=j["processedRowsPerSecond"],
+            metrics=dict(j["metrics"]) if "metrics" in j else {},
         )
 
     @property
     def description(self) -> str:
-        """Description of the source."""
+        """
+        Description of the source.
+        """
         return self["description"]
 
     @property
     def startOffset(self) -> str:
-        """The starting offset for data being read."""
+        """
+        The starting offset for data being read.
+        """
         return self["startOffset"]
 
     @property
     def endOffset(self) -> str:
-        """The ending offset for data being read."""
+        """
+        The ending offset for data being read.
+        """
         return self["endOffset"]
 
     @property
     def latestOffset(self) -> str:
-        """The latest offset from this source."""
+        """
+        The latest offset from this source.
+        """
         return self["latestOffset"]
 
     @property
     def numInputRows(self) -> int:
-        """The number of records read from this source."""
+        """
+        The number of records read from this source.
+        """
         return self["numInputRows"]
 
     @property
     def inputRowsPerSecond(self) -> float:
-        """The rate at which data is arriving from this source."""
+        """
+        The rate at which data is arriving from this source.
+        """
         return self["inputRowsPerSecond"]
 
     @property
     def processedRowsPerSecond(self) -> float:
-        """The rate at which data from this source is being processed by 
Spark."""
+        """
+        The rate at which data from this source is being processed by Spark.
+        """
         return self["processedRowsPerSecond"]
 
     @property
@@ -452,19 +990,25 @@ class SourceProgress(dict):
 
     @property
     def json(self) -> str:
-        """The compact JSON representation of this progress."""
-        if self._jdict:
-            return json.dumps(self._jdict)
+        """
+        The compact JSON representation of this progress.
+        """
+        assert self._jdict is not None or self._jprogress is not None
+        if self._jprogress:
+            return self._jprogress.json()
         else:
-            return json.dumps(dict(self))
+            return json.dumps(self._jdict)
 
     @property
     def prettyJson(self) -> str:
-        """The pretty (indented) JSON representation of this progress."""
-        if self._jdict:
-            return json.dumps(self._jdict, indent=4)
+        """
+        The pretty (i.e. indented) JSON representation of this progress.
+        """
+        assert self._jdict is not None or self._jprogress is not None
+        if self._jprogress:
+            return self._jprogress.prettyJson()
         else:
-            return json.dumps(dict(self), indent=4)
+            return json.dumps(self._jdict, indent=4)
 
     def __str__(self) -> str:
         return self.prettyJson
@@ -474,13 +1018,23 @@ class SourceProgress(dict):
 
 
 class SinkProgress(dict):
-    """Progress information for sinks."""
+    """
+    .. versionadded:: 3.4.0
+
+    .. versionchanged:: 4.0.0
+        Becomes a subclass of dict
+
+    Notes
+    -----
+    This API is evolving.
+    """
 
     def __init__(
         self,
         description: str,
         numOutputRows: int,
         metrics: Dict[str, str],
+        jprogress: Optional["JavaObject"] = None,
         jdict: Optional[Dict[str, Any]] = None,
     ) -> None:
         super().__init__(
@@ -488,25 +1042,40 @@ class SinkProgress(dict):
             numOutputRows=numOutputRows,
             metrics=metrics,
         )
+        self._jprogress: Optional["JavaObject"] = jprogress
         self._jdict: Optional[Dict[str, Any]] = jdict
 
+    @classmethod
+    def fromJObject(cls, jprogress: "JavaObject") -> "SinkProgress":
+        return cls(
+            jprogress=jprogress,
+            description=jprogress.description(),
+            numOutputRows=jprogress.numOutputRows(),
+            metrics=dict(jprogress.metrics()),
+        )
+
     @classmethod
     def fromJson(cls, j: Dict[str, Any]) -> "SinkProgress":
         return cls(
             jdict=j,
-            description=j.get("description", ""),
-            numOutputRows=j.get("numOutputRows", 0),
-            metrics=dict(j.get("metrics", {})),
+            description=j["description"],
+            numOutputRows=j["numOutputRows"],
+            metrics=dict(j["metrics"]) if "metrics" in j else {},
         )
 
     @property
     def description(self) -> str:
-        """Description of the sink."""
+        """
+        Description of the source.
+        """
         return self["description"]
 
     @property
     def numOutputRows(self) -> int:
-        """Number of rows written to the sink."""
+        """
+        Number of rows written to the sink or -1 for Continuous Mode 
(temporarily)
+        or Sink V1 (until decommissioned).
+        """
         return self["numOutputRows"]
 
     @property
@@ -515,19 +1084,25 @@ class SinkProgress(dict):
 
     @property
     def json(self) -> str:
-        """The compact JSON representation of this progress."""
-        if self._jdict:
-            return json.dumps(self._jdict)
+        """
+        The compact JSON representation of this progress.
+        """
+        assert self._jdict is not None or self._jprogress is not None
+        if self._jprogress:
+            return self._jprogress.json()
         else:
-            return json.dumps(dict(self))
+            return json.dumps(self._jdict)
 
     @property
     def prettyJson(self) -> str:
-        """The pretty (indented) JSON representation of this progress."""
-        if self._jdict:
-            return json.dumps(self._jdict, indent=4)
+        """
+        The pretty (i.e. indented) JSON representation of this progress.
+        """
+        assert self._jdict is not None or self._jprogress is not None
+        if self._jprogress:
+            return self._jprogress.prettyJson()
         else:
-            return json.dumps(dict(self), indent=4)
+            return json.dumps(self._jdict, indent=4)
 
     def __str__(self) -> str:
         return self.prettyJson
@@ -536,164 +1111,35 @@ class SinkProgress(dict):
         return self.prettyJson
 
 
-class StreamingQueryProgress(dict):
-    """Progress information for a streaming query."""
-
-    def __init__(
-        self,
-        id: uuid.UUID,
-        runId: uuid.UUID,
-        name: Optional[str],
-        timestamp: str,
-        batchId: int,
-        batchDuration: int,
-        durationMs: Dict[str, int],
-        eventTime: Dict[str, str],
-        stateOperators: List[StateOperatorProgress],
-        sources: List[SourceProgress],
-        sink: SinkProgress,
-        numInputRows: Optional[int],
-        inputRowsPerSecond: Optional[float],
-        processedRowsPerSecond: Optional[float],
-        observedMetrics: Dict[str, Row],
-        jdict: Optional[Dict[str, Any]] = None,
-    ):
-        super().__init__(
-            id=id,
-            runId=runId,
-            name=name,
-            timestamp=timestamp,
-            batchId=batchId,
-            batchDuration=batchDuration,
-            durationMs=durationMs,
-            eventTime=eventTime,
-            stateOperators=stateOperators,
-            sources=sources,
-            sink=sink,
-            numInputRows=numInputRows,
-            inputRowsPerSecond=inputRowsPerSecond,
-            processedRowsPerSecond=processedRowsPerSecond,
-            observedMetrics=observedMetrics,
-        )
-        self._jdict: Optional[Dict[str, Any]] = jdict
-
-    @classmethod
-    def fromJson(cls, j: Dict[str, Any]) -> "StreamingQueryProgress":
-        return cls(
-            jdict=j,
-            id=uuid.UUID(j.get("id", "")),
-            runId=uuid.UUID(j.get("runId", "")),
-            name=j.get("name"),
-            timestamp=j.get("timestamp", ""),
-            batchId=j.get("batchId", 0),
-            batchDuration=j.get("batchDuration", 0),
-            durationMs=dict(j.get("durationMs", {})),
-            eventTime=dict(j.get("eventTime", {})),
-            stateOperators=[StateOperatorProgress.fromJson(s) for s in 
j.get("stateOperators", [])],
-            sources=[SourceProgress.fromJson(s) for s in j.get("sources", [])],
-            sink=SinkProgress.fromJson(j.get("sink", {})),
-            numInputRows=j.get("numInputRows"),
-            inputRowsPerSecond=j.get("inputRowsPerSecond"),
-            processedRowsPerSecond=j.get("processedRowsPerSecond"),
-            observedMetrics={
-                k: Row(*row_dict.keys())(*row_dict.values())
-                for k, row_dict in j.get("observedMetrics", {}).items()
-            },
-        )
-
-    @property
-    def id(self) -> uuid.UUID:
-        """A unique query id that persists across restarts."""
-        return super().__getitem__("id")
-
-    @property
-    def runId(self) -> uuid.UUID:
-        """A query id that is unique for every start/restart."""
-        return super().__getitem__("runId")
-
-    @property
-    def name(self) -> Optional[str]:
-        """User-specified name of the query, `None` if not specified."""
-        return self["name"]
-
-    @property
-    def timestamp(self) -> str:
-        """The timestamp to start a query."""
-        return self["timestamp"]
-
-    @property
-    def batchId(self) -> int:
-        """The current batch id."""
-        return self["batchId"]
-
-    @property
-    def batchDuration(self) -> int:
-        """The duration of the batch in milliseconds."""
-        return self["batchDuration"]
+def _test() -> None:
+    import sys
+    import doctest
+    import os
+    from pyspark.core.context import SparkContext
+    from pyspark.sql import SparkSession
+    import pyspark.sql.streaming.listener
+    from py4j.protocol import Py4JError
 
-    @property
-    def durationMs(self) -> Dict[str, int]:
-        """The duration information."""
-        return self["durationMs"]
+    os.chdir(os.environ["SPARK_HOME"])
 
-    @property
-    def eventTime(self) -> Dict[str, str]:
-        """The event time information."""
-        return self["eventTime"]
+    globs = pyspark.sql.streaming.listener.__dict__.copy()
+    sc = SparkContext("local[4]", "PythonTest")
+    try:
+        spark = SparkSession._getActiveSessionOrCreate()
+    except Py4JError:
+        spark = SparkSession(sc)
 
-    @property
-    def stateOperators(self) -> List[StateOperatorProgress]:
-        """The state operator progress information."""
-        return self["stateOperators"]
+    globs["spark"] = spark
 
-    @property
-    def sources(self) -> List[SourceProgress]:
-        """The source progress information."""
-        return self["sources"]
+    failure_count, test_count = doctest.testmod(
+        pyspark.sql.streaming.listener,
+        globs=globs,
+    )
+    globs["spark"].stop()
 
-    @property
-    def sink(self) -> SinkProgress:
-        """The sink progress information."""
-        return self["sink"]
+    if failure_count:
+        sys.exit(-1)
 
-    @property
-    def numInputRows(self) -> Optional[int]:
-        """The number of input rows."""
-        return self["numInputRows"]
-
-    @property
-    def inputRowsPerSecond(self) -> Optional[float]:
-        """The input rows per second."""
-        return self["inputRowsPerSecond"]
 
-    @property
-    def processedRowsPerSecond(self) -> Optional[float]:
-        """The processed rows per second."""
-        return self["processedRowsPerSecond"]
-
-    @property
-    def observedMetrics(self) -> Dict[str, Row]:
-        """The observed metrics."""
-        return self["observedMetrics"]
-
-    @property
-    def json(self) -> str:
-        """The compact JSON representation of this progress."""
-        if self._jdict:
-            return json.dumps(self._jdict)
-        else:
-            return json.dumps(dict(self))
-
-    @property
-    def prettyJson(self) -> str:
-        """The pretty (indented) JSON representation of this progress."""
-        if self._jdict:
-            return json.dumps(self._jdict, indent=4)
-        else:
-            return json.dumps(dict(self), indent=4)
-
-    def __str__(self) -> str:
-        return self.prettyJson
-
-    def __repr__(self) -> str:
-        return self.prettyJson
+if __name__ == "__main__":
+    _test()
diff --git a/python/tests/test_dropin_e2e.py b/python/tests/test_dropin_e2e.py
index 01e51120..c644ed2d 100644
--- a/python/tests/test_dropin_e2e.py
+++ b/python/tests/test_dropin_e2e.py
@@ -2983,6 +2983,24 @@ def 
test_foreach_runs_a_function_and_a_writer_of_each_row_as_pyspark(session, tm
     assert e.value.getCondition() == "ATTRIBUTE_NOT_CALLABLE"
 
 
+def 
test_progress_of_queries_sums_the_rows_of_their_sources_as_pyspark(session):
+    # As PySpark's: the server leaves the total of the rows of a progress out 
of its JSON, of
+    # which the progress sums those of its sources.
+    query = (
+        session.readStream.format("rate").option("rowsPerSecond", 10).load()
+        .writeStream.format("noop").start()
+    )
+    try:
+        deadline = time.time() + 60
+        while not any(p.numInputRows for p in query.recentProgress) and 
time.time() < deadline:
+            time.sleep(0.5)
+        progress = next(p for p in query.recentProgress if p.numInputRows)
+        assert progress.numInputRows == sum(s.numInputRows for s in 
progress.sources) > 0
+        assert progress.inputRowsPerSecond == sum(s.inputRowsPerSecond for s 
in progress.sources)
+    finally:
+        query.stop()
+
+
 def test_dataframes_of_a_stream_are_streaming_as_pyspark(session):
     # As PySpark's, which asks the server: a DataFrame with a streaming source 
is streaming.
     stream = session.readStream.format("rate").load()
diff --git a/python/tests/test_dropin_offline.py 
b/python/tests/test_dropin_offline.py
index 5da128c7..aa99cbcf 100644
--- a/python/tests/test_dropin_offline.py
+++ b/python/tests/test_dropin_offline.py
@@ -6453,6 +6453,50 @@ def test_types_all_exports_spatial_and_variant():
         assert name in T.__all__, name
 
 
+def test_streaming_progress_and_listeners_are_those_of_pyspark():
+    # As PySpark's: the server leaves the totals of the rows out of the JSON 
of a progress, of
+    # which the progress sums those of its sources; and a listener has a 
session.
+    from pyspark.sql.streaming import listener as listener_module
+    from pyspark.sql.streaming.listener import (
+        QueryProgressEvent,
+        StreamingQueryListener,
+        StreamingQueryProgress,
+    )
+
+    source = {
+        "description": "RateStreamV2[rowsPerSecond=10, rampUpTimeSeconds=0, 
numPartitions=default",
+        "startOffset": "0", "endOffset": "1", "latestOffset": "1", 
"numInputRows": 10,
+        "inputRowsPerSecond": 666.5, "processedRowsPerSecond": 30.5, 
"metrics": {},
+    }
+    progress = {
+        "id": "74482783-456e-4ea8-ba81-a85565c714a3",
+        "runId": "1615bcbd-7213-4d70-b9a3-094da5d810d7", "name": None,
+        "timestamp": "2026-11-08T02:52:01.261Z", "batchId": 1, 
"batchDuration": 324,
+        "durationMs": {"triggerExecution": 324}, "eventTime": {}, 
"stateOperators": [],
+        "sources": [source, dict(source, numInputRows=5, 
inputRowsPerSecond=1.0)],
+        "sink": {"description": "NoopTable", "numOutputRows": 15, "metrics": 
{}},
+        "observedMetrics": {},
+    }
+    parsed = StreamingQueryProgress.fromJson(progress)
+    assert parsed.numInputRows == 15 and parsed.inputRowsPerSecond == 667.5
+    assert parsed.processedRowsPerSecond == 61.0 and parsed["numInputRows"] is 
None
+    assert [s.numInputRows for s in parsed.sources] == [10, 5] and 
parsed.sink.numOutputRows == 15
+    event = QueryProgressEvent.fromJson({"progress": progress})
+    assert event.progress.numInputRows == 15 and event.progress.batchId == 1
+    assert listener_module.__all__ == ["StreamingQueryListener"]
+
+    class Listener(StreamingQueryListener):
+        def onQueryStarted(self, event): ...
+        def onQueryProgress(self, event): ...
+        def onQueryIdle(self, event): ...
+        def onQueryTerminated(self, event): ...
+
+    listener = Listener()
+    assert listener.spark is None
+    listener._set_spark_session("session")
+    assert listener.spark == "session"
+
+
 def test_streaming_exports_the_stateful_processor_and_tws_tester():
     import pandas as pd
 
diff --git a/scripts/check_vendored_upstream.sh 
b/scripts/check_vendored_upstream.sh
index 5f7d65e7..bd9c0479 100644
--- a/scripts/check_vendored_upstream.sh
+++ b/scripts/check_vendored_upstream.sh
@@ -246,6 +246,9 @@ VENDORED_FILES=(
 
   # SQL: streaming. TwsTester runs a StatefulProcessor locally - pure upstream.
   "sql/streaming/tws_tester.py"
+  # The listener and its events and progress classes, which the client builds 
of the JSON of
+  # the server - pure upstream.
+  "sql/streaming/listener.py"
 
   # Streaming (excluding adapted context, dstream, util)
   "streaming/__init__.py"


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to