This is an automated email from the ASF dual-hosted git repository.
HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git
The following commit(s) were added to refs/heads/master by this push:
new 4b05e8da [SPARK-61151][CONNECT][PYTHON] Vendor
pyspark.sql.streaming.listener, of the listener and its event and progress
classes, in the Python client as PySpark has it
4b05e8da is described below
commit 4b05e8daa1db50af4f04adf6797315d1a2646371
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Sun Nov 8 14:14:23 2026 +0900
[SPARK-61151][CONNECT][PYTHON] Vendor pyspark.sql.streaming.listener, of
the listener and its event and progress classes, in the Python client as
PySpark has it
### What changes were proposed in this pull request?
[SPARK-61151](https://issues.apache.org/jira/browse/SPARK-61151): The
client's `pyspark.sql.streaming.listener` was its own rewrite of PySpark's: 699
lines, of which most differ from PySpark 4.3.0's 1145. The module is pure
Python and needs nothing of the client, so it is now PySpark 4.3.0's, vendored
verbatim, and `check_vendored_upstream.sh` checks it.
So, as PySpark's:
- `StreamingQueryProgress.numInputRows`, `inputRowsPerSecond` and
`processedRowsPerSecond` sum those of the sources when the JSON of the server
leaves them out, as it does
- the items of a progress, such as `progress["id"]`, are those of its JSON:
a string, rather than a `uuid.UUID`
- `StreamingQueryListener` has `spark` and `_set_spark_session`
- `__all__` is `["StreamingQueryListener"]`
The client builds the events and the progresses with their `fromJson`,
which PySpark's module has too.
### Why are the changes needed?
- `q.lastProgress.numInputRows` and `q.recentProgress[i].numInputRows` were
`None`.
- `lastProgress["id"] == q.id` was `False`, a `uuid.UUID` against a string.
- A listener had no `spark`.
Apache Spark's tests failed on this client:
- `test_batch_dataframe_in_batch_session_sql_is_supported` and
`test_batch_dataframe_join_in_batch_session_is_supported` of
`test_parity_foreach_batch.py` (`'>' not supported between instances of
'NoneType' and 'int'`)
- `test_stream_status_and_progress`, `test_streaming_progress` and
`test_streaming_query_functions_basic` of `test_parity_streaming.py`
(`UUID(...) != '...'`)
### Does this PR introduce _any_ user-facing change?
Yes. The progresses, the events and the listeners of streaming queries are
those of PySpark 4.3.0.
### How was this patch tested?
- A new offline test parses the progress of the server, of two sources: its
totals of rows and rates, those of the sources and the sink, the event of a
progress, `__all__`, and the `spark` of a listener. It passes with official
PySpark 4.3.0, and fails without this change.
- A new live test checks that the progress of a `rate` query sums the rows
and the rates of its sources. It fails without this change.
- The progress of a `rate` query reads the same as with official PySpark
4.3.0 (`numInputRows`, `inputRowsPerSecond`, `sources`, the keys of its JSON).
- Apache Spark's `test_parity_streaming.py` runs on this client against a
local server: 29 tests pass, and 1, `test_query_manager_no_recreation`, is
fixed by SPARK-61162. Before this change, 4 failed. The two tests of
`test_parity_foreach_batch.py` above pass.
- The lib tests of both crates (596 passed), the build of all tests, also
with `wasm-udf`, the offline Python suites (2213 passed, 8 skipped), and the
live drop-in and Arrow suites (197 passed) and live cases (10 passed) against a
local Spark 4.3.0 server. Clippy reports no diagnostics in the changed lines.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
This pull request and its description were written by Isaac.
Closes #350 from HyukjinKwon/SPARK-61151-streaming-listener.
Authored-by: Hyukjin Kwon <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
---
python/pyspark/sql/streaming/listener.py | 986 ++++++++++++++++++++++---------
python/tests/test_dropin_e2e.py | 18 +
python/tests/test_dropin_offline.py | 44 ++
scripts/check_vendored_upstream.sh | 3 +
4 files changed, 781 insertions(+), 270 deletions(-)
diff --git a/python/pyspark/sql/streaming/listener.py
b/python/pyspark/sql/streaming/listener.py
index 7733dd3e..8799c4a1 100644
--- a/python/pyspark/sql/streaming/listener.py
+++ b/python/pyspark/sql/streaming/listener.py
@@ -14,87 +14,167 @@
# See the License for the specific language governing permissions and
# limitations under the License.
#
-
-"""
-Streaming Query Listener - defines listener interface and event classes
-for streaming query lifecycle management.
-
-This module vendors classes from pyspark.sql.streaming.listener for use
-in the Connect streaming client.
-"""
-
-import json
import uuid
-from abc import ABC, abstractmethod
+import json
from typing import Any, Dict, List, Optional, Set, TYPE_CHECKING
+from abc import ABC, abstractmethod
+
from pyspark.sql import Row
+from pyspark import cloudpickle
-if TYPE_CHECKING:
- pass
+__all__ = ["StreamingQueryListener"]
-__all__ = [
- "StreamingQueryListener",
- "QueryStartedEvent",
- "QueryProgressEvent",
- "QueryIdleEvent",
- "QueryTerminatedEvent",
- "StreamingQueryProgress",
- "StateOperatorProgress",
- "SourceProgress",
- "SinkProgress",
-]
+if TYPE_CHECKING:
+ from py4j.java_gateway import JavaObject
class StreamingQueryListener(ABC):
"""
- Interface for listening to events related to StreamingQuery.
+ Interface for listening to events related to
:class:`~pyspark.sql.streaming.StreamingQuery`.
+
+ .. versionadded:: 3.4.0
Notes
-----
The methods are not thread-safe as they may be called from different
threads.
- The events received are identical with Scala API.
+ The events received are identical with Scala API. Refer to its
documentation.
+
+ This API is evolving.
Examples
--------
>>> class MyListener(StreamingQueryListener):
... def onQueryStarted(self, event: QueryStartedEvent) -> None:
+ ... # Do something with event.
... pass
...
... def onQueryProgress(self, event: QueryProgressEvent) -> None:
+ ... # Do something with event.
... pass
...
... def onQueryIdle(self, event: QueryIdleEvent) -> None:
+ ... # Do something with event.
... pass
...
... def onQueryTerminated(self, event: QueryTerminatedEvent) -> None:
+ ... # Do something with event.
... pass
+ ...
+ >>> spark.streams.addListener(MyListener())
"""
+ def _set_spark_session(
+ self,
+ session: "SparkSession", # type: ignore[name-defined] # noqa: F821
+ ) -> None:
+ if self.spark is None:
+ self.spark = session
+
+ @property
+ def spark(self) -> Optional["SparkSession"]: # type: ignore[name-defined]
# noqa: F821
+ return getattr(self, "_sparkSession", None)
+
+ @spark.setter
+ def spark(self, session: "SparkSession") -> None: # type:
ignore[name-defined] # noqa: F821
+ # For backward compatibility
+ self._sparkSession = session
+
def _init_listener_id(self) -> None:
self._id = str(uuid.uuid4())
@abstractmethod
def onQueryStarted(self, event: "QueryStartedEvent") -> None:
- """Called when a query is started."""
+ """
+ Called when a query is started.
+
+ Notes
+ -----
+ This is called synchronously with
:py:meth:`~pyspark.sql.streaming.DataStreamWriter.start`,
+ that is, `onQueryStart` will be called on all listeners before
`DataStreamWriter.start()`
+ returns the corresponding
:class:`~pyspark.sql.streaming.StreamingQuery`.
+ Please don't block this method as it will block your query.
+ """
pass
@abstractmethod
def onQueryProgress(self, event: "QueryProgressEvent") -> None:
- """Called when there is some status update."""
+ """
+ Called when there is some status update (ingestion rate updated, etc.)
+
+ Notes
+ -----
+ This method is asynchronous. The status in
:class:`~pyspark.sql.streaming.StreamingQuery`
+ will always be latest no matter when this method is called. Therefore,
the status of
+ :class:`~pyspark.sql.streaming.StreamingQuery`.
+ may be changed before/when you process the event. E.g., you may find
+ :class:`~pyspark.sql.streaming.StreamingQuery` is terminated when you
are
+ processing `QueryProgressEvent`.
+ """
pass
+ # NOTE: Do not mark this as abstract method, since we released this
abstract class without
+ # this method in prior version and marking this as abstract method would
break existing
+ # implementations.
def onQueryIdle(self, event: "QueryIdleEvent") -> None:
- """Called when the query is idle and waiting for new data to
process."""
+ """
+ Called when the query is idle and waiting for new data to process.
+ """
pass
@abstractmethod
def onQueryTerminated(self, event: "QueryTerminatedEvent") -> None:
- """Called when a query is stopped, with or without error."""
+ """
+ Called when a query is stopped, with or without error.
+ """
pass
+ @property
+ def _jlistener(self) -> "JavaObject":
+ from pyspark import SparkContext
+
+ if hasattr(self, "_jlistenerobj"):
+ return self._jlistenerobj
+
+ self._jlistenerobj: "JavaObject" =
SparkContext._jvm.PythonStreamingQueryListenerWrapper( # type:
ignore[union-attr]
+ JStreamingQueryListener(self)
+ )
+ return self._jlistenerobj
+
+
+class JStreamingQueryListener:
+ """
+ Python class that implements Java interface by Py4J.
+ """
+
+ def __init__(self, pylistener: StreamingQueryListener) -> None:
+ self.pylistener = pylistener
+
+ def onQueryStarted(self, jevent: "JavaObject") -> None:
+ self.pylistener.onQueryStarted(QueryStartedEvent.fromJObject(jevent))
+
+ def onQueryProgress(self, jevent: "JavaObject") -> None:
+ self.pylistener.onQueryProgress(QueryProgressEvent.fromJObject(jevent))
+
+ def onQueryIdle(self, jevent: "JavaObject") -> None:
+ self.pylistener.onQueryIdle(QueryIdleEvent.fromJObject(jevent))
+
+ def onQueryTerminated(self, jevent: "JavaObject") -> None:
+
self.pylistener.onQueryTerminated(QueryTerminatedEvent.fromJObject(jevent))
+
+ class Java:
+ implements =
["org.apache.spark.sql.streaming.PythonStreamingQueryListener"]
+
class QueryStartedEvent:
- """Event representing the start of a query."""
+ """
+ Event representing the start of a query.
+
+ .. versionadded:: 3.4.0
+
+ Notes
+ -----
+ This API is evolving.
+ """
def __init__(
self,
@@ -110,61 +190,111 @@ class QueryStartedEvent:
self._timestamp: str = timestamp
self._jobTags: Set[str] = jobTags
+ @classmethod
+ def fromJObject(cls, jevent: "JavaObject") -> "QueryStartedEvent":
+ job_tags = set()
+ java_iterator = jevent.jobTags().iterator()
+ while java_iterator.hasNext():
+ job_tags.add(java_iterator.next().toString())
+
+ return cls(
+ id=uuid.UUID(jevent.id().toString()),
+ runId=uuid.UUID(jevent.runId().toString()),
+ name=jevent.name(),
+ timestamp=jevent.timestamp(),
+ jobTags=job_tags,
+ )
+
@classmethod
def fromJson(cls, j: Dict[str, Any]) -> "QueryStartedEvent":
- job_tags = j.get("jobTags", [])
+ # Json4s will convert jobTags to a list, so we need to convert it back
to a set.
+ job_tags = j["jobTags"] if "jobTags" in j else []
return cls(
id=uuid.UUID(j["id"]),
runId=uuid.UUID(j["runId"]),
- name=j.get("name"),
- timestamp=j.get("timestamp", ""),
+ name=j["name"],
+ timestamp=j["timestamp"],
jobTags=set(job_tags),
)
@property
def id(self) -> uuid.UUID:
- """A unique query id that persists across restarts."""
+ """
+ A unique query id that persists across restarts. See
+ py:meth:`~pyspark.sql.streaming.StreamingQuery.id`.
+ """
return self._id
@property
def runId(self) -> uuid.UUID:
- """A query id that is unique for every start/restart."""
+ """
+ A query id that is unique for every start/restart. See
+ py:meth:`~pyspark.sql.streaming.StreamingQuery.runId`.
+ """
return self._runId
@property
def name(self) -> Optional[str]:
- """User-specified name of the query, `None` if not specified."""
+ """
+ User-specified name of the query, `None` if not specified.
+ """
return self._name
@property
def timestamp(self) -> str:
- """The timestamp to start a query."""
+ """
+ The timestamp to start a query.
+ """
return self._timestamp
@property
def jobTags(self) -> Set[str]:
- """The job tags of the query."""
+ """
+ The job tags of the query.
+ """
return self._jobTags
class QueryProgressEvent:
- """Event representing any progress updates in a query."""
+ """
+ Event representing any progress updates in a query.
+
+ .. versionadded:: 3.4.0
+
+ Notes
+ -----
+ This API is evolving.
+ """
def __init__(self, progress: "StreamingQueryProgress") -> None:
self._progress: StreamingQueryProgress = progress
+ @classmethod
+ def fromJObject(cls, jevent: "JavaObject") -> "QueryProgressEvent":
+ return
cls(progress=StreamingQueryProgress.fromJObject(jevent.progress()))
+
@classmethod
def fromJson(cls, j: Dict[str, Any]) -> "QueryProgressEvent":
- return cls(progress=StreamingQueryProgress.fromJson(j.get("progress",
j)))
+ return cls(progress=StreamingQueryProgress.fromJson(j["progress"]))
@property
def progress(self) -> "StreamingQueryProgress":
- """The query progress updates."""
+ """
+ The query progress updates.
+ """
return self._progress
class QueryIdleEvent:
- """Event representing that query is idle and waiting for new data to
process."""
+ """
+ Event representing that query is idle and waiting for new data to process.
+
+ .. versionadded:: 3.5.0
+
+ Notes
+ -----
+ This API is evolving.
+ """
def __init__(self, id: uuid.UUID, runId: uuid.UUID, timestamp: str) ->
None:
self._id: uuid.UUID = id
@@ -172,31 +302,51 @@ class QueryIdleEvent:
self._timestamp: str = timestamp
@classmethod
- def fromJson(cls, j: Dict[str, Any]) -> "QueryIdleEvent":
+ def fromJObject(cls, jevent: "JavaObject") -> "QueryIdleEvent":
return cls(
- id=uuid.UUID(j["id"]),
- runId=uuid.UUID(j["runId"]),
- timestamp=j.get("timestamp", ""),
+ id=uuid.UUID(jevent.id().toString()),
+ runId=uuid.UUID(jevent.runId().toString()),
+ timestamp=jevent.timestamp(),
)
+ @classmethod
+ def fromJson(cls, j: Dict[str, Any]) -> "QueryIdleEvent":
+ return cls(id=uuid.UUID(j["id"]), runId=uuid.UUID(j["runId"]),
timestamp=j["timestamp"])
+
@property
def id(self) -> uuid.UUID:
- """A unique query id that persists across restarts."""
+ """
+ A unique query id that persists across restarts. See
+ py:meth:`~pyspark.sql.streaming.StreamingQuery.id`.
+ """
return self._id
@property
def runId(self) -> uuid.UUID:
- """A query id that is unique for every start/restart."""
+ """
+ A query id that is unique for every start/restart. See
+ py:meth:`~pyspark.sql.streaming.StreamingQuery.runId`.
+ """
return self._runId
@property
def timestamp(self) -> str:
- """The timestamp when the latest no-batch trigger happened."""
+ """
+ The timestamp when the latest no-batch trigger happened.
+ """
return self._timestamp
class QueryTerminatedEvent:
- """Event representing that termination of a query."""
+ """
+ Event representing that termination of a query.
+
+ .. versionadded:: 3.4.0
+
+ Notes
+ -----
+ This API is evolving.
+ """
def __init__(
self,
@@ -210,38 +360,359 @@ class QueryTerminatedEvent:
self._exception: Optional[str] = exception
self._errorClassOnException: Optional[str] = errorClassOnException
+ @classmethod
+ def fromJObject(cls, jevent: "JavaObject") -> "QueryTerminatedEvent":
+ jexception = jevent.exception()
+ jerrorclass = jevent.errorClassOnException()
+ return cls(
+ id=uuid.UUID(jevent.id().toString()),
+ runId=uuid.UUID(jevent.runId().toString()),
+ exception=jexception.get() if jexception.isDefined() else None,
+ errorClassOnException=jerrorclass.get() if jerrorclass.isDefined()
else None,
+ )
+
@classmethod
def fromJson(cls, j: Dict[str, Any]) -> "QueryTerminatedEvent":
return cls(
id=uuid.UUID(j["id"]),
runId=uuid.UUID(j["runId"]),
- exception=j.get("exception"),
- errorClassOnException=j.get("errorClassOnException"),
+ exception=j["exception"],
+ errorClassOnException=j["errorClassOnException"],
)
@property
def id(self) -> uuid.UUID:
- """A unique query id that persists across restarts."""
+ """
+ A unique query id that persists across restarts. See
+ py:meth:`~pyspark.sql.streaming.StreamingQuery.id`.
+ """
return self._id
@property
def runId(self) -> uuid.UUID:
- """A query id that is unique for every start/restart."""
+ """
+ A query id that is unique for every start/restart. See
+ py:meth:`~pyspark.sql.streaming.StreamingQuery.runId`.
+ """
return self._runId
@property
def exception(self) -> Optional[str]:
- """The exception message if the query terminated with an exception."""
+ """
+ The exception message of the query if the query was terminated
+ with an exception. Otherwise, it will be `None`.
+ """
return self._exception
@property
def errorClassOnException(self) -> Optional[str]:
- """The error class from the exception if applicable."""
+ """
+ The error class from the exception if the query was terminated
+ with an exception which is a part of error class framework.
+ If the query was terminated without an exception, or the
+ exception is not a part of error class framework, it will be
+ `None`.
+
+ .. versionadded:: 3.5.0
+ """
return self._errorClassOnException
+class StreamingQueryProgress(dict):
+ """
+ .. versionadded:: 3.4.0
+
+ .. versionchanged:: 4.0.0
+ Becomes a subclass of dict
+
+ Notes
+ -----
+ This API is evolving.
+ """
+
+ def __init__(
+ self,
+ id: uuid.UUID,
+ runId: uuid.UUID,
+ name: Optional[str],
+ timestamp: str,
+ batchId: int,
+ batchDuration: int,
+ durationMs: Dict[str, int],
+ eventTime: Dict[str, str],
+ stateOperators: List["StateOperatorProgress"],
+ sources: List["SourceProgress"],
+ sink: "SinkProgress",
+ numInputRows: Optional[int],
+ inputRowsPerSecond: Optional[float],
+ processedRowsPerSecond: Optional[float],
+ observedMetrics: Dict[str, Row],
+ jprogress: Optional["JavaObject"] = None,
+ jdict: Optional[Dict[str, Any]] = None,
+ ):
+ super().__init__(
+ id=id,
+ runId=runId,
+ name=name,
+ timestamp=timestamp,
+ batchId=batchId,
+ batchDuration=batchDuration,
+ durationMs=durationMs,
+ eventTime=eventTime,
+ stateOperators=stateOperators,
+ sources=sources,
+ sink=sink,
+ numInputRows=numInputRows,
+ inputRowsPerSecond=inputRowsPerSecond,
+ processedRowsPerSecond=processedRowsPerSecond,
+ observedMetrics=observedMetrics,
+ )
+ self._jprogress: Optional["JavaObject"] = jprogress
+ self._jdict: Optional[Dict[str, Any]] = jdict
+
+ @classmethod
+ def fromJObject(cls, jprogress: "JavaObject") -> "StreamingQueryProgress":
+ from pyspark import SparkContext
+
+ return cls(
+ jprogress=jprogress,
+ id=uuid.UUID(jprogress.id().toString()),
+ runId=uuid.UUID(jprogress.runId().toString()),
+ name=jprogress.name(),
+ timestamp=jprogress.timestamp(),
+ batchId=jprogress.batchId(),
+ batchDuration=jprogress.batchDuration(),
+ durationMs=dict(jprogress.durationMs()),
+ eventTime=dict(jprogress.eventTime()),
+ stateOperators=[
+ StateOperatorProgress.fromJObject(js) for js in
jprogress.stateOperators()
+ ],
+ sources=[SourceProgress.fromJObject(js) for js in
jprogress.sources()],
+ sink=SinkProgress.fromJObject(jprogress.sink()),
+ numInputRows=jprogress.numInputRows(),
+ inputRowsPerSecond=jprogress.inputRowsPerSecond(),
+ processedRowsPerSecond=jprogress.processedRowsPerSecond(),
+ observedMetrics={
+ k: cloudpickle.loads(
+ SparkContext._jvm.PythonSQLUtils.toPyRow(jr) # type:
ignore[union-attr]
+ )
+ for k, jr in dict(jprogress.observedMetrics()).items()
+ },
+ )
+
+ @classmethod
+ def fromJson(cls, j: Dict[str, Any]) -> "StreamingQueryProgress":
+ return cls(
+ jdict=j,
+ id=uuid.UUID(j["id"]),
+ runId=uuid.UUID(j["runId"]),
+ name=j["name"],
+ timestamp=j["timestamp"],
+ batchId=j["batchId"],
+ batchDuration=j["batchDuration"],
+ durationMs=dict(j["durationMs"]) if "durationMs" in j else {},
+ eventTime=dict(j["eventTime"]) if "eventTime" in j else {},
+ stateOperators=[StateOperatorProgress.fromJson(s) for s in
j["stateOperators"]],
+ sources=[SourceProgress.fromJson(s) for s in j["sources"]],
+ sink=SinkProgress.fromJson(j["sink"]),
+ numInputRows=j["numInputRows"] if "numInputRows" in j else None,
+ inputRowsPerSecond=j["inputRowsPerSecond"] if "inputRowsPerSecond"
in j else None,
+ processedRowsPerSecond=(
+ j["processedRowsPerSecond"] if "processedRowsPerSecond" in j
else None
+ ),
+ observedMetrics=(
+ {
+ k: Row(*row_dict.keys())(*row_dict.values()) # Assume no
nested rows
+ for k, row_dict in j["observedMetrics"].items()
+ }
+ if "observedMetrics" in j
+ else {}
+ ),
+ )
+
+ @property
+ def id(self) -> uuid.UUID:
+ """
+ A unique query id that persists across restarts. See
+ py:meth:`~pyspark.sql.streaming.StreamingQuery.id`.
+ """
+ # Before Spark 4.0, StreamingQuery.lastProgress returns a dict, which
casts id and runId
+ # to string. But here they are UUID.
+ # To prevent breaking change, do not cast them to string when accessed
with attribute.
+ return super().__getitem__("id")
+
+ @property
+ def runId(self) -> uuid.UUID:
+ """
+ A query id that is unique for every start/restart. See
+ py:meth:`~pyspark.sql.streaming.StreamingQuery.runId`.
+ """
+ # Before Spark 4.0, StreamingQuery.lastProgress returns a dict, which
casts id and runId
+ # to string. But here they are UUID.
+ # To prevent breaking change, do not cast them to string when accessed
with attribute.
+ return super().__getitem__("runId")
+
+ @property
+ def name(self) -> Optional[str]:
+ """
+ User-specified name of the query, `None` if not specified.
+ """
+ return self["name"]
+
+ @property
+ def timestamp(self) -> str:
+ """
+ The timestamp to start a query.
+ """
+ return self["timestamp"]
+
+ @property
+ def batchId(self) -> int:
+ """
+ A unique id for the current batch of data being processed. Note that
in the
+ case of retries after a failure a given batchId my be executed more
than once.
+ Similarly, when there is no data to be processed, the batchId will not
be
+ incremented.
+ """
+ return self["batchId"]
+
+ @property
+ def batchDuration(self) -> int:
+ """
+ The process duration of each batch.
+ """
+ return self["batchDuration"]
+
+ @property
+ def durationMs(self) -> Dict[str, int]:
+ """
+ The amount of time taken to perform various operations in milliseconds.
+ """
+ return self["durationMs"]
+
+ @property
+ def eventTime(self) -> Dict[str, str]:
+ """
+ Statistics of event time seen in this batch. It may contain the
following keys:
+
+ .. code-block:: python
+
+ {
+ "max": "2016-12-05T20:54:20.827Z", # maximum event time seen
in this trigger
+ "min": "2016-12-05T20:54:20.827Z", # minimum event time seen
in this trigger
+ "avg": "2016-12-05T20:54:20.827Z", # average event time seen
in this trigger
+ "watermark": "2016-12-05T20:54:20.827Z" # watermark used in
this trigger
+ }
+
+ All timestamps are in ISO8601 format, i.e. UTC timestamps.
+ """
+ return self["eventTime"]
+
+ @property
+ def stateOperators(self) -> List["StateOperatorProgress"]:
+ """
+ Information about operators in the query that store state.
+ """
+ return self["stateOperators"]
+
+ @property
+ def sources(self) -> List["SourceProgress"]:
+ """
+ detailed statistics on data being read from each of the streaming
sources.
+ """
+ return self["sources"]
+
+ @property
+ def sink(self) -> "SinkProgress":
+ """
+ A unique query id that persists across restarts. See
+ py:meth:`~pyspark.sql.streaming.StreamingQuery.id`.
+ """
+ return self["sink"]
+
+ @property
+ def observedMetrics(self) -> Dict[str, Row]:
+ return self["observedMetrics"]
+
+ @property
+ def numInputRows(self) -> int:
+ """
+ The aggregate (across all sources) number of records processed in a
trigger.
+ """
+ if self["numInputRows"] is not None:
+ return self["numInputRows"]
+ else:
+ return sum(s.numInputRows for s in self.sources)
+
+ @property
+ def inputRowsPerSecond(self) -> float:
+ """
+ The aggregate (across all sources) rate of data arriving.
+ """
+ if self["inputRowsPerSecond"] is not None:
+ return self["inputRowsPerSecond"]
+ else:
+ return sum(s.inputRowsPerSecond for s in self.sources)
+
+ @property
+ def processedRowsPerSecond(self) -> float:
+ """
+ The aggregate (across all sources) rate at which Spark is processing
data.
+ """
+ if self["processedRowsPerSecond"] is not None:
+ return self["processedRowsPerSecond"]
+ else:
+ return sum(s.processedRowsPerSecond for s in self.sources)
+
+ @property
+ def json(self) -> str:
+ """
+ The compact JSON representation of this progress.
+ """
+ assert self._jdict is not None or self._jprogress is not None
+ if self._jprogress:
+ return self._jprogress.json()
+ else:
+ return json.dumps(self._jdict)
+
+ @property
+ def prettyJson(self) -> str:
+ """
+ The pretty (i.e. indented) JSON representation of this progress.
+ """
+ assert self._jdict is not None or self._jprogress is not None
+ if self._jprogress:
+ return self._jprogress.prettyJson()
+ else:
+ return json.dumps(self._jdict, indent=4)
+
+ def __getitem__(self, key: str) -> Any:
+ # Before Spark 4.0, StreamingQuery.lastProgress returns a dict, which
casts id and runId
+ # to string. But here they are UUID.
+ # To prevent breaking change, also cast them to string when accessed
with __getitem__.
+ if key == "id" or key == "runId":
+ return str(super().__getitem__(key))
+ else:
+ return super().__getitem__(key)
+
+ def __str__(self) -> str:
+ return self.prettyJson
+
+ def __repr__(self) -> str:
+ return self.prettyJson
+
+
class StateOperatorProgress(dict):
- """Progress information for state operators."""
+ """
+ .. versionadded:: 3.4.0
+
+ .. versionchanged:: 4.0.0
+ Becomes a subclass of dict
+
+ Notes
+ -----
+ This API is evolving.
+ """
def __init__(
self,
@@ -257,6 +728,7 @@ class StateOperatorProgress(dict):
numShufflePartitions: int,
numStateStoreInstances: int,
customMetrics: Dict[str, int],
+ jprogress: Optional["JavaObject"] = None,
jdict: Optional[Dict[str, Any]] = None,
):
super().__init__(
@@ -273,24 +745,43 @@ class StateOperatorProgress(dict):
numStateStoreInstances=numStateStoreInstances,
customMetrics=customMetrics,
)
+ self._jprogress: Optional["JavaObject"] = jprogress
self._jdict: Optional[Dict[str, Any]] = jdict
+ @classmethod
+ def fromJObject(cls, jprogress: "JavaObject") -> "StateOperatorProgress":
+ return cls(
+ jprogress=jprogress,
+ operatorName=jprogress.operatorName(),
+ numRowsTotal=jprogress.numRowsTotal(),
+ numRowsUpdated=jprogress.numRowsUpdated(),
+ allUpdatesTimeMs=jprogress.allUpdatesTimeMs(),
+ numRowsRemoved=jprogress.numRowsRemoved(),
+ allRemovalsTimeMs=jprogress.allRemovalsTimeMs(),
+ commitTimeMs=jprogress.commitTimeMs(),
+ memoryUsedBytes=jprogress.memoryUsedBytes(),
+ numRowsDroppedByWatermark=jprogress.numRowsDroppedByWatermark(),
+ numShufflePartitions=jprogress.numShufflePartitions(),
+ numStateStoreInstances=jprogress.numStateStoreInstances(),
+ customMetrics=dict(jprogress.customMetrics()),
+ )
+
@classmethod
def fromJson(cls, j: Dict[str, Any]) -> "StateOperatorProgress":
return cls(
jdict=j,
- operatorName=j.get("operatorName", ""),
- numRowsTotal=j.get("numRowsTotal", 0),
- numRowsUpdated=j.get("numRowsUpdated", 0),
- numRowsRemoved=j.get("numRowsRemoved", 0),
- allUpdatesTimeMs=j.get("allUpdatesTimeMs", 0),
- allRemovalsTimeMs=j.get("allRemovalsTimeMs", 0),
- commitTimeMs=j.get("commitTimeMs", 0),
- memoryUsedBytes=j.get("memoryUsedBytes", 0),
- numRowsDroppedByWatermark=j.get("numRowsDroppedByWatermark", 0),
- numShufflePartitions=j.get("numShufflePartitions", 0),
- numStateStoreInstances=j.get("numStateStoreInstances", 0),
- customMetrics=dict(j.get("customMetrics", {})),
+ operatorName=j["operatorName"],
+ numRowsTotal=j["numRowsTotal"],
+ numRowsUpdated=j["numRowsUpdated"],
+ numRowsRemoved=j["numRowsRemoved"],
+ allUpdatesTimeMs=j["allUpdatesTimeMs"],
+ allRemovalsTimeMs=j["allRemovalsTimeMs"],
+ commitTimeMs=j["commitTimeMs"],
+ memoryUsedBytes=j["memoryUsedBytes"],
+ numRowsDroppedByWatermark=j["numRowsDroppedByWatermark"],
+ numShufflePartitions=j["numShufflePartitions"],
+ numStateStoreInstances=j["numStateStoreInstances"],
+ customMetrics=dict(j["customMetrics"]) if "customMetrics" in j
else {},
)
@property
@@ -343,19 +834,25 @@ class StateOperatorProgress(dict):
@property
def json(self) -> str:
- """The compact JSON representation of this progress."""
- if self._jdict:
- return json.dumps(self._jdict)
+ """
+ The compact JSON representation of this progress.
+ """
+ assert self._jdict is not None or self._jprogress is not None
+ if self._jprogress:
+ return self._jprogress.json()
else:
- return json.dumps(dict(self))
+ return json.dumps(self._jdict)
@property
def prettyJson(self) -> str:
- """The pretty (indented) JSON representation of this progress."""
- if self._jdict:
- return json.dumps(self._jdict, indent=4)
+ """
+ The pretty (i.e. indented) JSON representation of this progress.
+ """
+ assert self._jdict is not None or self._jprogress is not None
+ if self._jprogress:
+ return self._jprogress.prettyJson()
else:
- return json.dumps(dict(self), indent=4)
+ return json.dumps(self._jdict, indent=4)
def __str__(self) -> str:
return self.prettyJson
@@ -365,7 +862,16 @@ class StateOperatorProgress(dict):
class SourceProgress(dict):
- """Progress information for data sources."""
+ """
+ .. versionadded:: 3.4.0
+
+ .. versionchanged:: 4.0.0
+ Becomes a subclass of dict
+
+ Notes
+ -----
+ This API is evolving.
+ """
def __init__(
self,
@@ -377,6 +883,7 @@ class SourceProgress(dict):
inputRowsPerSecond: float,
processedRowsPerSecond: float,
metrics: Dict[str, str],
+ jprogress: Optional["JavaObject"] = None,
jdict: Optional[Dict[str, Any]] = None,
) -> None:
super().__init__(
@@ -389,11 +896,28 @@ class SourceProgress(dict):
processedRowsPerSecond=processedRowsPerSecond,
metrics=metrics,
)
+ self._jprogress: Optional["JavaObject"] = jprogress
self._jdict: Optional[Dict[str, Any]] = jdict
+ @classmethod
+ def fromJObject(cls, jprogress: "JavaObject") -> "SourceProgress":
+ return cls(
+ jprogress=jprogress,
+ description=jprogress.description(),
+ startOffset=str(jprogress.startOffset()),
+ endOffset=str(jprogress.endOffset()),
+ latestOffset=str(jprogress.latestOffset()),
+ numInputRows=jprogress.numInputRows(),
+ inputRowsPerSecond=jprogress.inputRowsPerSecond(),
+ processedRowsPerSecond=jprogress.processedRowsPerSecond(),
+ metrics=dict(jprogress.metrics()),
+ )
+
@classmethod
def fromJson(cls, j: Dict[str, Any]) -> "SourceProgress":
def _to_json_string(value: Any) -> str:
+ """Convert offset value to JSON string. If already a string,
return as-is.
+ If a dict/list, JSON-encode it."""
if isinstance(value, str):
return value
else:
@@ -401,49 +925,63 @@ class SourceProgress(dict):
return cls(
jdict=j,
- description=j.get("description", ""),
- startOffset=_to_json_string(j.get("startOffset", "")),
- endOffset=_to_json_string(j.get("endOffset", "")),
- latestOffset=_to_json_string(j.get("latestOffset", "")),
- numInputRows=j.get("numInputRows", 0),
- inputRowsPerSecond=j.get("inputRowsPerSecond", 0.0),
- processedRowsPerSecond=j.get("processedRowsPerSecond", 0.0),
- metrics=dict(j.get("metrics", {})),
+ description=j["description"],
+ startOffset=_to_json_string(j["startOffset"]),
+ endOffset=_to_json_string(j["endOffset"]),
+ latestOffset=_to_json_string(j["latestOffset"]),
+ numInputRows=j["numInputRows"],
+ inputRowsPerSecond=j["inputRowsPerSecond"],
+ processedRowsPerSecond=j["processedRowsPerSecond"],
+ metrics=dict(j["metrics"]) if "metrics" in j else {},
)
@property
def description(self) -> str:
- """Description of the source."""
+ """
+ Description of the source.
+ """
return self["description"]
@property
def startOffset(self) -> str:
- """The starting offset for data being read."""
+ """
+ The starting offset for data being read.
+ """
return self["startOffset"]
@property
def endOffset(self) -> str:
- """The ending offset for data being read."""
+ """
+ The ending offset for data being read.
+ """
return self["endOffset"]
@property
def latestOffset(self) -> str:
- """The latest offset from this source."""
+ """
+ The latest offset from this source.
+ """
return self["latestOffset"]
@property
def numInputRows(self) -> int:
- """The number of records read from this source."""
+ """
+ The number of records read from this source.
+ """
return self["numInputRows"]
@property
def inputRowsPerSecond(self) -> float:
- """The rate at which data is arriving from this source."""
+ """
+ The rate at which data is arriving from this source.
+ """
return self["inputRowsPerSecond"]
@property
def processedRowsPerSecond(self) -> float:
- """The rate at which data from this source is being processed by
Spark."""
+ """
+ The rate at which data from this source is being processed by Spark.
+ """
return self["processedRowsPerSecond"]
@property
@@ -452,19 +990,25 @@ class SourceProgress(dict):
@property
def json(self) -> str:
- """The compact JSON representation of this progress."""
- if self._jdict:
- return json.dumps(self._jdict)
+ """
+ The compact JSON representation of this progress.
+ """
+ assert self._jdict is not None or self._jprogress is not None
+ if self._jprogress:
+ return self._jprogress.json()
else:
- return json.dumps(dict(self))
+ return json.dumps(self._jdict)
@property
def prettyJson(self) -> str:
- """The pretty (indented) JSON representation of this progress."""
- if self._jdict:
- return json.dumps(self._jdict, indent=4)
+ """
+ The pretty (i.e. indented) JSON representation of this progress.
+ """
+ assert self._jdict is not None or self._jprogress is not None
+ if self._jprogress:
+ return self._jprogress.prettyJson()
else:
- return json.dumps(dict(self), indent=4)
+ return json.dumps(self._jdict, indent=4)
def __str__(self) -> str:
return self.prettyJson
@@ -474,13 +1018,23 @@ class SourceProgress(dict):
class SinkProgress(dict):
- """Progress information for sinks."""
+ """
+ .. versionadded:: 3.4.0
+
+ .. versionchanged:: 4.0.0
+ Becomes a subclass of dict
+
+ Notes
+ -----
+ This API is evolving.
+ """
def __init__(
self,
description: str,
numOutputRows: int,
metrics: Dict[str, str],
+ jprogress: Optional["JavaObject"] = None,
jdict: Optional[Dict[str, Any]] = None,
) -> None:
super().__init__(
@@ -488,25 +1042,40 @@ class SinkProgress(dict):
numOutputRows=numOutputRows,
metrics=metrics,
)
+ self._jprogress: Optional["JavaObject"] = jprogress
self._jdict: Optional[Dict[str, Any]] = jdict
+ @classmethod
+ def fromJObject(cls, jprogress: "JavaObject") -> "SinkProgress":
+ return cls(
+ jprogress=jprogress,
+ description=jprogress.description(),
+ numOutputRows=jprogress.numOutputRows(),
+ metrics=dict(jprogress.metrics()),
+ )
+
@classmethod
def fromJson(cls, j: Dict[str, Any]) -> "SinkProgress":
return cls(
jdict=j,
- description=j.get("description", ""),
- numOutputRows=j.get("numOutputRows", 0),
- metrics=dict(j.get("metrics", {})),
+ description=j["description"],
+ numOutputRows=j["numOutputRows"],
+ metrics=dict(j["metrics"]) if "metrics" in j else {},
)
@property
def description(self) -> str:
- """Description of the sink."""
+ """
+ Description of the source.
+ """
return self["description"]
@property
def numOutputRows(self) -> int:
- """Number of rows written to the sink."""
+ """
+ Number of rows written to the sink or -1 for Continuous Mode
(temporarily)
+ or Sink V1 (until decommissioned).
+ """
return self["numOutputRows"]
@property
@@ -515,19 +1084,25 @@ class SinkProgress(dict):
@property
def json(self) -> str:
- """The compact JSON representation of this progress."""
- if self._jdict:
- return json.dumps(self._jdict)
+ """
+ The compact JSON representation of this progress.
+ """
+ assert self._jdict is not None or self._jprogress is not None
+ if self._jprogress:
+ return self._jprogress.json()
else:
- return json.dumps(dict(self))
+ return json.dumps(self._jdict)
@property
def prettyJson(self) -> str:
- """The pretty (indented) JSON representation of this progress."""
- if self._jdict:
- return json.dumps(self._jdict, indent=4)
+ """
+ The pretty (i.e. indented) JSON representation of this progress.
+ """
+ assert self._jdict is not None or self._jprogress is not None
+ if self._jprogress:
+ return self._jprogress.prettyJson()
else:
- return json.dumps(dict(self), indent=4)
+ return json.dumps(self._jdict, indent=4)
def __str__(self) -> str:
return self.prettyJson
@@ -536,164 +1111,35 @@ class SinkProgress(dict):
return self.prettyJson
-class StreamingQueryProgress(dict):
- """Progress information for a streaming query."""
-
- def __init__(
- self,
- id: uuid.UUID,
- runId: uuid.UUID,
- name: Optional[str],
- timestamp: str,
- batchId: int,
- batchDuration: int,
- durationMs: Dict[str, int],
- eventTime: Dict[str, str],
- stateOperators: List[StateOperatorProgress],
- sources: List[SourceProgress],
- sink: SinkProgress,
- numInputRows: Optional[int],
- inputRowsPerSecond: Optional[float],
- processedRowsPerSecond: Optional[float],
- observedMetrics: Dict[str, Row],
- jdict: Optional[Dict[str, Any]] = None,
- ):
- super().__init__(
- id=id,
- runId=runId,
- name=name,
- timestamp=timestamp,
- batchId=batchId,
- batchDuration=batchDuration,
- durationMs=durationMs,
- eventTime=eventTime,
- stateOperators=stateOperators,
- sources=sources,
- sink=sink,
- numInputRows=numInputRows,
- inputRowsPerSecond=inputRowsPerSecond,
- processedRowsPerSecond=processedRowsPerSecond,
- observedMetrics=observedMetrics,
- )
- self._jdict: Optional[Dict[str, Any]] = jdict
-
- @classmethod
- def fromJson(cls, j: Dict[str, Any]) -> "StreamingQueryProgress":
- return cls(
- jdict=j,
- id=uuid.UUID(j.get("id", "")),
- runId=uuid.UUID(j.get("runId", "")),
- name=j.get("name"),
- timestamp=j.get("timestamp", ""),
- batchId=j.get("batchId", 0),
- batchDuration=j.get("batchDuration", 0),
- durationMs=dict(j.get("durationMs", {})),
- eventTime=dict(j.get("eventTime", {})),
- stateOperators=[StateOperatorProgress.fromJson(s) for s in
j.get("stateOperators", [])],
- sources=[SourceProgress.fromJson(s) for s in j.get("sources", [])],
- sink=SinkProgress.fromJson(j.get("sink", {})),
- numInputRows=j.get("numInputRows"),
- inputRowsPerSecond=j.get("inputRowsPerSecond"),
- processedRowsPerSecond=j.get("processedRowsPerSecond"),
- observedMetrics={
- k: Row(*row_dict.keys())(*row_dict.values())
- for k, row_dict in j.get("observedMetrics", {}).items()
- },
- )
-
- @property
- def id(self) -> uuid.UUID:
- """A unique query id that persists across restarts."""
- return super().__getitem__("id")
-
- @property
- def runId(self) -> uuid.UUID:
- """A query id that is unique for every start/restart."""
- return super().__getitem__("runId")
-
- @property
- def name(self) -> Optional[str]:
- """User-specified name of the query, `None` if not specified."""
- return self["name"]
-
- @property
- def timestamp(self) -> str:
- """The timestamp to start a query."""
- return self["timestamp"]
-
- @property
- def batchId(self) -> int:
- """The current batch id."""
- return self["batchId"]
-
- @property
- def batchDuration(self) -> int:
- """The duration of the batch in milliseconds."""
- return self["batchDuration"]
+def _test() -> None:
+ import sys
+ import doctest
+ import os
+ from pyspark.core.context import SparkContext
+ from pyspark.sql import SparkSession
+ import pyspark.sql.streaming.listener
+ from py4j.protocol import Py4JError
- @property
- def durationMs(self) -> Dict[str, int]:
- """The duration information."""
- return self["durationMs"]
+ os.chdir(os.environ["SPARK_HOME"])
- @property
- def eventTime(self) -> Dict[str, str]:
- """The event time information."""
- return self["eventTime"]
+ globs = pyspark.sql.streaming.listener.__dict__.copy()
+ sc = SparkContext("local[4]", "PythonTest")
+ try:
+ spark = SparkSession._getActiveSessionOrCreate()
+ except Py4JError:
+ spark = SparkSession(sc)
- @property
- def stateOperators(self) -> List[StateOperatorProgress]:
- """The state operator progress information."""
- return self["stateOperators"]
+ globs["spark"] = spark
- @property
- def sources(self) -> List[SourceProgress]:
- """The source progress information."""
- return self["sources"]
+ failure_count, test_count = doctest.testmod(
+ pyspark.sql.streaming.listener,
+ globs=globs,
+ )
+ globs["spark"].stop()
- @property
- def sink(self) -> SinkProgress:
- """The sink progress information."""
- return self["sink"]
+ if failure_count:
+ sys.exit(-1)
- @property
- def numInputRows(self) -> Optional[int]:
- """The number of input rows."""
- return self["numInputRows"]
-
- @property
- def inputRowsPerSecond(self) -> Optional[float]:
- """The input rows per second."""
- return self["inputRowsPerSecond"]
- @property
- def processedRowsPerSecond(self) -> Optional[float]:
- """The processed rows per second."""
- return self["processedRowsPerSecond"]
-
- @property
- def observedMetrics(self) -> Dict[str, Row]:
- """The observed metrics."""
- return self["observedMetrics"]
-
- @property
- def json(self) -> str:
- """The compact JSON representation of this progress."""
- if self._jdict:
- return json.dumps(self._jdict)
- else:
- return json.dumps(dict(self))
-
- @property
- def prettyJson(self) -> str:
- """The pretty (indented) JSON representation of this progress."""
- if self._jdict:
- return json.dumps(self._jdict, indent=4)
- else:
- return json.dumps(dict(self), indent=4)
-
- def __str__(self) -> str:
- return self.prettyJson
-
- def __repr__(self) -> str:
- return self.prettyJson
+if __name__ == "__main__":
+ _test()
diff --git a/python/tests/test_dropin_e2e.py b/python/tests/test_dropin_e2e.py
index 01e51120..c644ed2d 100644
--- a/python/tests/test_dropin_e2e.py
+++ b/python/tests/test_dropin_e2e.py
@@ -2983,6 +2983,24 @@ def
test_foreach_runs_a_function_and_a_writer_of_each_row_as_pyspark(session, tm
assert e.value.getCondition() == "ATTRIBUTE_NOT_CALLABLE"
+def
test_progress_of_queries_sums_the_rows_of_their_sources_as_pyspark(session):
+ # As PySpark's: the server leaves the total of the rows of a progress out
of its JSON, of
+ # which the progress sums those of its sources.
+ query = (
+ session.readStream.format("rate").option("rowsPerSecond", 10).load()
+ .writeStream.format("noop").start()
+ )
+ try:
+ deadline = time.time() + 60
+ while not any(p.numInputRows for p in query.recentProgress) and
time.time() < deadline:
+ time.sleep(0.5)
+ progress = next(p for p in query.recentProgress if p.numInputRows)
+ assert progress.numInputRows == sum(s.numInputRows for s in
progress.sources) > 0
+ assert progress.inputRowsPerSecond == sum(s.inputRowsPerSecond for s
in progress.sources)
+ finally:
+ query.stop()
+
+
def test_dataframes_of_a_stream_are_streaming_as_pyspark(session):
# As PySpark's, which asks the server: a DataFrame with a streaming source
is streaming.
stream = session.readStream.format("rate").load()
diff --git a/python/tests/test_dropin_offline.py
b/python/tests/test_dropin_offline.py
index 5da128c7..aa99cbcf 100644
--- a/python/tests/test_dropin_offline.py
+++ b/python/tests/test_dropin_offline.py
@@ -6453,6 +6453,50 @@ def test_types_all_exports_spatial_and_variant():
assert name in T.__all__, name
+def test_streaming_progress_and_listeners_are_those_of_pyspark():
+ # As PySpark's: the server leaves the totals of the rows out of the JSON
of a progress, of
+ # which the progress sums those of its sources; and a listener has a
session.
+ from pyspark.sql.streaming import listener as listener_module
+ from pyspark.sql.streaming.listener import (
+ QueryProgressEvent,
+ StreamingQueryListener,
+ StreamingQueryProgress,
+ )
+
+ source = {
+ "description": "RateStreamV2[rowsPerSecond=10, rampUpTimeSeconds=0,
numPartitions=default",
+ "startOffset": "0", "endOffset": "1", "latestOffset": "1",
"numInputRows": 10,
+ "inputRowsPerSecond": 666.5, "processedRowsPerSecond": 30.5,
"metrics": {},
+ }
+ progress = {
+ "id": "74482783-456e-4ea8-ba81-a85565c714a3",
+ "runId": "1615bcbd-7213-4d70-b9a3-094da5d810d7", "name": None,
+ "timestamp": "2026-11-08T02:52:01.261Z", "batchId": 1,
"batchDuration": 324,
+ "durationMs": {"triggerExecution": 324}, "eventTime": {},
"stateOperators": [],
+ "sources": [source, dict(source, numInputRows=5,
inputRowsPerSecond=1.0)],
+ "sink": {"description": "NoopTable", "numOutputRows": 15, "metrics":
{}},
+ "observedMetrics": {},
+ }
+ parsed = StreamingQueryProgress.fromJson(progress)
+ assert parsed.numInputRows == 15 and parsed.inputRowsPerSecond == 667.5
+ assert parsed.processedRowsPerSecond == 61.0 and parsed["numInputRows"] is
None
+ assert [s.numInputRows for s in parsed.sources] == [10, 5] and
parsed.sink.numOutputRows == 15
+ event = QueryProgressEvent.fromJson({"progress": progress})
+ assert event.progress.numInputRows == 15 and event.progress.batchId == 1
+ assert listener_module.__all__ == ["StreamingQueryListener"]
+
+ class Listener(StreamingQueryListener):
+ def onQueryStarted(self, event): ...
+ def onQueryProgress(self, event): ...
+ def onQueryIdle(self, event): ...
+ def onQueryTerminated(self, event): ...
+
+ listener = Listener()
+ assert listener.spark is None
+ listener._set_spark_session("session")
+ assert listener.spark == "session"
+
+
def test_streaming_exports_the_stateful_processor_and_tws_tester():
import pandas as pd
diff --git a/scripts/check_vendored_upstream.sh
b/scripts/check_vendored_upstream.sh
index 5f7d65e7..bd9c0479 100644
--- a/scripts/check_vendored_upstream.sh
+++ b/scripts/check_vendored_upstream.sh
@@ -246,6 +246,9 @@ VENDORED_FILES=(
# SQL: streaming. TwsTester runs a StatefulProcessor locally - pure upstream.
"sql/streaming/tws_tester.py"
+ # The listener and its events and progress classes, which the client builds
of the JSON of
+ # the server - pure upstream.
+ "sql/streaming/listener.py"
# Streaming (excluding adapted context, dstream, util)
"streaming/__init__.py"
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]