This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new af2d2ee1153 [SPARK-40081][PYTHON][DOCS] Add document parameters for
pyspark.sql.streaming.query
af2d2ee1153 is described below
commit af2d2ee11530a98e748518a81925f3f0a2392209
Author: Qian.Sun <[email protected]>
AuthorDate: Tue Aug 23 09:40:54 2022 +0900
[SPARK-40081][PYTHON][DOCS] Add document parameters for
pyspark.sql.streaming.query
### What changes were proposed in this pull request?
As mentioned
https://github.com/apache/spark/pull/37482#issuecomment-1212245579, this PR
proposes to add document parameters for `pyspark.sql.streaming.query`.
### Why are the changes needed?
To make the documentation more readable.
### Does this PR introduce _any_ user-facing change?
Yes, it changes the documentation.
### How was this patch tested?
Manually ran each doctest.
Closes #37587 from dcoliversun/SPARK-40081.
Lead-authored-by: Qian.Sun <[email protected]>
Co-authored-by: Hyukjin Kwon <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
---
python/pyspark/sql/streaming/query.py | 95 +++++++++++++++++++++++++++++++++--
1 file changed, 92 insertions(+), 3 deletions(-)
diff --git a/python/pyspark/sql/streaming/query.py
b/python/pyspark/sql/streaming/query.py
index aaf0b677335..e99a01bcfff 100644
--- a/python/pyspark/sql/streaming/query.py
+++ b/python/pyspark/sql/streaming/query.py
@@ -53,6 +53,11 @@ class StreamingQuery:
There can only be one query with the same id active in a Spark cluster.
Also see, `runId`.
+ Returns
+ -------
+ str
+ The unique id of query that persists across restarts from
checkpoint data.
+
Examples
--------
>>> sdf = spark.readStream.format("rate").load()
@@ -74,6 +79,11 @@ class StreamingQuery:
Returns the unique id of this query that does not persist across
restarts. That is, every
query that is started (or restarted from checkpoint) will have a
different runId.
+ Returns
+ -------
+ str
+ The unique id of query that does not persist across restarts.
+
Examples
--------
>>> sdf = spark.readStream.format("rate").load()
@@ -97,6 +107,11 @@ class StreamingQuery:
as `dataframe.writeStream.queryName("query").start()`.
This name, if set, must be unique across all active queries.
+ Returns
+ -------
+ str
+ The user-specified name of the query, or null if not specified.
+
Examples
--------
>>> sdf = spark.readStream.format("rate").load()
@@ -117,6 +132,11 @@ class StreamingQuery:
"""
Whether this streaming query is currently active or not.
+ Returns
+ -------
+ bool
+ The result whether specified streaming query is currently active
or not.
+
Examples
--------
>>> sdf = spark.readStream.format("rate").load()
@@ -142,6 +162,18 @@ class StreamingQuery:
throws :class:`StreamingQueryException`, if `this` query has
terminated with an exception
+ Parameters
+ ----------
+ timeout : int, optional
+ default ``None``. The waiting time for specified streaming query
to terminate.
+
+ Returns
+ -------
+ bool, optional
+ The result whether specified streaming query has terminated or not
within the `timeout`
+ seconds if `timeout` is set. The :class:`StreamingQueryException`
will be thrown if the
+ query has terminated with an exception.
+
Examples
--------
>>> sdf = spark.readStream.format("rate").load()
@@ -167,6 +199,11 @@ class StreamingQuery:
"""
Returns the current status of the query.
+ Returns
+ -------
+ dict
+ The current status of the specified query.
+
Examples
--------
>>> sdf = spark.readStream.format("rate").load()
@@ -189,6 +226,12 @@ class StreamingQuery:
The number of progress updates retained for each stream is configured
by Spark session
configuration `spark.sql.streaming.numRecentProgressUpdates`.
+ Returns
+ -------
+ list
+ List of dict which is the most recent
:class:`StreamingQueryProgress` updates
+ for this query.
+
Examples
--------
>>> sdf = spark.readStream.format("rate").load()
@@ -213,7 +256,9 @@ class StreamingQuery:
Returns
-------
- dict
+ dict, optional
+ The most recent :class:`StreamingQueryProgress` update of this
streaming query or
+ None if there were no progress updates.
Examples
--------
@@ -360,6 +405,11 @@ class StreamingQueryManager:
.. versionadded:: 2.0.0
+ Returns
+ -------
+ list
+ The active queries associated with this :class:`SparkSession`.
+
Examples
--------
>>> sdf = spark.readStream.format("rate").load()
@@ -381,11 +431,24 @@ class StreamingQueryManager:
def get(self, id: str) -> StreamingQuery:
"""
- Returns an active query from this SparkSession or throws exception if
an active query
- with this name doesn't exist.
+ Returns an active query from this :class:`SparkSession`.
.. versionadded:: 2.0.0
+ Parameters
+ ----------
+ id : str
+ The unique id of specified query.
+
+ Returns
+ -------
+ :class:`StreamingQuery`
+ An active query with `id` from this SparkSession.
+
+ Notes
+ -----
+ Exception will be thrown if an active query with this id does not
exist.
+
Examples
--------
>>> sdf = spark.readStream.format("rate").load()
@@ -429,6 +492,18 @@ class StreamingQueryManager:
throws :class:`StreamingQueryException`, if `this` query has
terminated with an exception
+ Parameters
+ ----------
+ timeout : int, optional
+ default ``None``. The waiting time for any streaming query to
terminate.
+
+ Returns
+ -------
+ bool, optional
+ The result whether any streaming query has terminated or not
within the `timeout`
+ seconds if `timeout` is set. The :class:`StreamingQueryException`
will be thrown if any
+ query has terminated with an exception.
+
Examples
--------
>>> sdf = spark.readStream.format("rate").load()
@@ -468,6 +543,12 @@ class StreamingQueryManager:
.. versionadded:: 3.4.0
+ Parameters
+ ----------
+ listener : :class:`StreamingQueryListener`
+ A :class:`StreamingQueryListener` to receive up-calls for life
cycle events of
+ :class:`~pyspark.sql.streaming.StreamingQuery`.
+
Examples
--------
>>> from pyspark.sql.streaming import StreamingQueryListener
@@ -506,6 +587,14 @@ class StreamingQueryManager:
.. versionadded:: 3.4.0
+ Parameters
+ ----------
+ listener : :class:`StreamingQueryListener`
+ A :class:`StreamingQueryListener` to receive up-calls for life
cycle events of
+ :class:`~pyspark.sql.streaming.StreamingQuery`.
+
+ Examples
+ --------
>>> from pyspark.sql.streaming import StreamingQueryListener
>>> class TestListener(StreamingQueryListener):
... def onQueryStarted(self, event):
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
