This is an automated email from the ASF dual-hosted git repository.

HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git


The following commit(s) were added to refs/heads/master by this push:
     new 0043c6d  [SPARK-61055][CONNECT][PYTHON] Return the status and the 
exception of a StreamingQuery, and check the timeouts of its waits, in the 
Python client as PySpark does
0043c6d is described below

commit 0043c6d227e0e6769555949384733cb6e54fdf90
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Sat Nov 7 12:15:40 2026 +0900

    [SPARK-61055][CONNECT][PYTHON] Return the status and the exception of a 
StreamingQuery, and check the timeouts of its waits, in the Python client as 
PySpark does
    
    ### What changes were proposed in this pull request?
    
    [SPARK-61055](https://issues.apache.org/jira/browse/SPARK-61055): 
`StreamingQuery.status` and `StreamingQuery.exception()` now return what 
PySpark Connect's do, and the waits of a query check their timeouts as 
PySpark's.
    
    - `query.status` is a dict of the `message`, `isDataAvailable` and 
`isTriggerActive` of the query.
    - `query.exception()` of a failed query returns the 
`StreamingQueryException` of `pyspark.errors`:
      - of the message of the server, without the type of its Java exception, 
and with its JVM stack trace
      - of the error class as its reason
    - `pyspark.sql.streaming.StreamingQueryException` and 
`pyspark.sql.streaming.query.StreamingQueryException` are that class of 
`pyspark.errors`, as in PySpark.
    - `awaitTermination(timeout)` and 
`spark.streams.awaitAnyTermination(timeout)` raise `VALUE_NOT_POSITIVE` of 
`timeout` for a timeout that is not a positive number.
    - `explain(extended)` takes an int as a bool, as the proto field of PySpark 
does.
    
    The `StreamingQueryStatus` and `StreamingQueryException` classes of the 
extension, which PySpark does not have, are removed. The core 
`StreamingQueryException` now keeps the stack trace of the server.
    
    ### Why are the changes needed?
    
    - `query.status` was an object of `status_message`, `is_data_available` and 
`is_trigger_active` attributes, neither subscriptable nor iterable: 
`query.status["message"]` raised a `TypeError`.
    - `query.exception()` returned an object that was not an exception, whose 
`str()` was its repr.
    - `pyspark.sql.streaming.StreamingQueryException` was that class, so 
`except StreamingQueryException:` raised a `TypeError`.
    - The timeouts raised `INVALID_TIMEOUT` or `TypeError`s, and `explain(1)` 
raised a `TypeError`.
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes. These return and raise what PySpark 4.3.0 does.
    
    ### How was this patch tested?
    
    - A new live test checks the dict of `status`, the 
`StreamingQueryException` of a failed query, its message and JVM stack trace, 
and that `awaitTermination` raises it. It passes with official PySpark 4.3.0, 
and fails without this change.
    - A new offline test checks the class of `StreamingQueryException`, the 
errors of the timeouts, and those of `explain`.
    - On a local server, `status` and `exception()` of a failed query give the 
dict, the class and the 125 lines of the message of official PySpark 4.3.0.
    - The lib tests of both crates (580 passed), the build of all tests, also 
with `wasm-udf`, the offline Python suites (1847 passed, 8 skipped), and the 
live drop-in and Arrow suites (116 passed, without the test of 
`StreamingQuery.explain`, which hangs on a slow machine and #260 fixes) and 
live cases (10 passed) against a local Spark 4.3.0 server. Clippy reports no 
diagnostics in the changed lines.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Opus 5.5
    
    This pull request and its description were written by Isaac.
    
    Closes #263 from HyukjinKwon/SPARK-61055-query-results.
    
    Authored-by: Hyukjin Kwon <[email protected]>
    Signed-off-by: Hyukjin Kwon <[email protected]>
---
 crates/pyspark-rs/src/lib.rs             |   4 +-
 crates/pyspark-rs/src/streaming.rs       | 148 +++++++++++++++----------------
 crates/spark-connect/src/streaming.rs    |   3 +
 python/pyspark/sql/streaming/__init__.py |   4 +-
 python/pyspark/sql/streaming/query.py    |  13 +--
 python/tests/test_dropin_e2e.py          |  36 ++++++++
 python/tests/test_dropin_offline.py      |  34 +++++++
 7 files changed, 152 insertions(+), 90 deletions(-)

diff --git a/crates/pyspark-rs/src/lib.rs b/crates/pyspark-rs/src/lib.rs
index d9cbbad..43d462c 100644
--- a/crates/pyspark-rs/src/lib.rs
+++ b/crates/pyspark-rs/src/lib.rs
@@ -81,7 +81,7 @@ use session::{PyHook, PySparkSession, PySparkSessionBuilder};
 use stat::PyStatFunctions;
 use streaming::{
     PyDataStreamReader, PyDataStreamWriter, PyListenerEventStream, 
PyStreamingQuery,
-    PyStreamingQueryException, PyStreamingQueryManager, 
PyStreamingQueryStatus, PyTrigger,
+    PyStreamingQueryManager, PyTrigger,
 };
 use types::{
     PyAnsiIntervalType, PyAnyTimeType, PyArrayType, PyAtomicType, 
PyBinaryType, PyBooleanType,
@@ -163,8 +163,6 @@ fn _pyspark(_py: Python<'_>, m: &Bound<'_, PyModule>) -> 
PyResult<()> {
     m.add_class::<PyDataStreamWriter>()?;
     m.add_class::<PyTrigger>()?;
     m.add_class::<PyStreamingQuery>()?;
-    m.add_class::<PyStreamingQueryStatus>()?;
-    m.add_class::<PyStreamingQueryException>()?;
     m.add_class::<PyStreamingQueryManager>()?;
     m.add_class::<PyListenerEventStream>()?;
 
diff --git a/crates/pyspark-rs/src/streaming.rs 
b/crates/pyspark-rs/src/streaming.rs
index c1f6130..e75107a 100644
--- a/crates/pyspark-rs/src/streaming.rs
+++ b/crates/pyspark-rs/src/streaming.rs
@@ -4,8 +4,7 @@ use pyo3::prelude::*;
 use pyo3::types::PyDict;
 use spark_connect::streaming::{
     DataStreamReader, DataStreamWriter, ListenerEventStream, StreamingQuery,
-    StreamingQueryException, StreamingQueryListener, 
StreamingQueryListenerEvent,
-    StreamingQueryManager, StreamingQueryStatus, Trigger,
+    StreamingQueryListener, StreamingQueryListenerEvent, 
StreamingQueryManager, Trigger,
 };
 use spark_connect::udf::PythonUDFPayload;
 use std::collections::HashMap;
@@ -811,73 +810,46 @@ impl PyDataStreamWriter {
     }
 }
 
-/// Python wrapper for StreamingQueryStatus.
-#[pyclass(
-    name = "StreamingQueryStatus",
-    module = "pyspark.sql.streaming.query",
-    from_py_object
-)]
-#[derive(Clone)]
-pub struct PyStreamingQueryStatus {
-    inner: StreamingQueryStatus,
-}
-
-impl PyStreamingQueryStatus {
-    pub fn new(status: StreamingQueryStatus) -> Self {
-        PyStreamingQueryStatus { inner: status }
-    }
-}
-
-#[pymethods]
-impl PyStreamingQueryStatus {
-    #[getter]
-    fn is_active(&self) -> bool {
-        self.inner.is_active
-    }
-
-    #[getter]
-    fn status_message(&self) -> String {
-        self.inner.status_message.clone()
-    }
-
-    #[getter]
-    fn is_data_available(&self) -> bool {
-        self.inner.is_data_available
-    }
-
-    #[getter]
-    fn is_trigger_active(&self) -> bool {
-        self.inner.is_trigger_active
+/// The timeout of `awaitTermination` and `awaitAnyTermination`, in seconds, 
as PySpark's take
+/// it: an int or a float, which must be positive, else `VALUE_NOT_POSITIVE` 
of the name of its
+/// type.
+fn positive_timeout(timeout: Option<&Bound<'_, PyAny>>) -> 
PyResult<Option<f64>> {
+    let Some(timeout) = timeout else {
+        return Ok(None);
+    };
+    let number = timeout.is_instance_of::<pyo3::types::PyInt>()
+        || timeout.is_instance_of::<pyo3::types::PyFloat>();
+    match number.then(|| timeout.extract::<f64>()).transpose()? {
+        Some(seconds) if seconds > 0.0 => Ok(Some(seconds)),
+        _ => {
+            let type_name = timeout.get_type().name()?.to_string();
+            Err(crate::column::raise_pyspark(
+                timeout.py(),
+                "PySparkValueError",
+                "VALUE_NOT_POSITIVE",
+                &[("arg_name", "timeout"), ("arg_value", &type_name)],
+            ))
+        }
     }
 }
 
-/// Python wrapper for StreamingQueryException.
-#[pyclass(
-    name = "StreamingQueryException",
-    module = "pyspark.sql.streaming.query",
-    from_py_object
-)]
-#[derive(Clone)]
-pub struct PyStreamingQueryException {
-    inner: StreamingQueryException,
-}
+/// A bool argument that PySpark sets on a proto bool field, which takes a 
bool or an int, and
+/// raises the `TypeError` of protobuf for another value.
+pub(crate) struct ProtoBool(pub(crate) bool);
 
-impl PyStreamingQueryException {
-    pub fn new(exc: StreamingQueryException) -> Self {
-        PyStreamingQueryException { inner: exc }
-    }
-}
+impl<'a, 'py> FromPyObject<'a, 'py> for ProtoBool {
+    type Error = PyErr;
 
-#[pymethods]
-impl PyStreamingQueryException {
-    #[getter]
-    fn message(&self) -> String {
-        self.inner.message.clone()
-    }
-
-    #[getter]
-    fn error_class(&self) -> String {
-        self.inner.error_class.clone()
+    fn extract(obj: Borrowed<'a, 'py, PyAny>) -> PyResult<Self> {
+        if obj.is_instance_of::<pyo3::types::PyInt>() {
+            return Ok(ProtoBool(obj.is_truthy()?));
+        }
+        let shown = obj.repr()?;
+        Err(pyo3::exceptions::PyTypeError::new_err(format!(
+            "Cannot set 
spark.connect.StreamingQueryCommand.ExplainCommand.extended to {shown}: \
+             {shown} has type {}, but expected one of: (<class 'bool'>, <class 
'int'>)",
+            obj.get_type().repr()?
+        )))
     }
 }
 
@@ -915,10 +887,16 @@ impl PyStreamingQuery {
         self.inner.is_active().to_pyerr()
     }
 
+    /// The status of the query, as PySpark's: a dict of its `message`, 
`isDataAvailable` and
+    /// `isTriggerActive`.
     #[getter]
-    fn status(&self) -> PyResult<PyStreamingQueryStatus> {
+    fn status<'py>(&self, py: Python<'py>) -> PyResult<Bound<'py, PyDict>> {
         let status = self.inner.status().to_pyerr()?;
-        Ok(PyStreamingQueryStatus::new(status))
+        let dict = PyDict::new(py);
+        dict.set_item("message", status.status_message)?;
+        dict.set_item("isDataAvailable", status.is_data_available)?;
+        dict.set_item("isTriggerActive", status.is_trigger_active)?;
+        Ok(dict)
     }
 
     fn stop(&self) -> PyResult<()> {
@@ -926,7 +904,8 @@ impl PyStreamingQuery {
     }
 
     #[pyo3(signature = (timeout=None))]
-    fn awaitTermination(&self, timeout: Option<f64>) -> PyResult<Option<bool>> 
{
+    fn awaitTermination(&self, timeout: Option<&Bound<'_, PyAny>>) -> 
PyResult<Option<bool>> {
+        let timeout = positive_timeout(timeout)?;
         self.inner.await_termination(timeout).to_pyerr()
     }
 
@@ -946,16 +925,36 @@ impl PyStreamingQuery {
 
     /// Print the plan of the query with Python's `print`, as PySpark's
     /// `StreamingQuery.explain(extended=False)`.
-    #[pyo3(signature = (extended=false))]
-    fn explain(&self, py: Python<'_>, extended: bool) -> PyResult<()> {
-        let plan = py.detach(|| self.inner.explain(extended)).to_pyerr()?;
+    #[pyo3(signature = (extended=ProtoBool(false)))]
+    fn explain(&self, py: Python<'_>, extended: ProtoBool) -> PyResult<()> {
+        let plan = py
+            .detach(|| self.inner.explain(extended.0).map_err(Box::new))
+            .to_pyerr()?;
         py.import("builtins")?.call_method1("print", (plan,))?;
         Ok(())
     }
 
-    fn exception(&self) -> PyResult<Option<PyStreamingQueryException>> {
-        let exc = self.inner.exception().to_pyerr()?;
-        Ok(exc.map(PyStreamingQueryException::new))
+    /// The exception of a failed query, as PySpark's: the 
`StreamingQueryException` of
+    /// `pyspark.errors.exceptions.connect`, of the message of the server 
without the type of
+    /// its Java exception, and with its JVM stack trace, and of the error 
class as its reason.
+    fn exception(&self, py: Python<'_>) -> PyResult<Option<Py<PyAny>>> {
+        let Some(exception) = self.inner.exception().to_pyerr()? else {
+            return Ok(None);
+        };
+        // The message is the `toString` of the Java StreamingQueryException: 
drop its type.
+        let mut message = match exception.message.split_once(": ") {
+            Some((_, message)) => message.to_string(),
+            None => exception.message,
+        };
+        if let Some(stack_trace) = exception.stack_trace {
+            message.push_str(&format!("\n\nJVM stacktrace:\n{stack_trace}"));
+        }
+        let class = py
+            .import("pyspark.errors.exceptions.connect")?
+            .getattr("StreamingQueryException")?;
+        let kwargs = PyDict::new(py);
+        kwargs.set_item("reason", exception.error_class)?;
+        Ok(Some(class.call((message,), Some(&kwargs))?.unbind()))
     }
 }
 
@@ -985,7 +984,8 @@ impl PyStreamingQueryManager {
     }
 
     #[pyo3(signature = (timeout=None))]
-    fn awaitAnyTermination(&self, timeout: Option<f64>) -> 
PyResult<Option<bool>> {
+    fn awaitAnyTermination(&self, timeout: Option<&Bound<'_, PyAny>>) -> 
PyResult<Option<bool>> {
+        let timeout = positive_timeout(timeout)?;
         self.inner.await_any_termination(timeout).to_pyerr()
     }
 
diff --git a/crates/spark-connect/src/streaming.rs 
b/crates/spark-connect/src/streaming.rs
index e1648bb..80f17ac 100644
--- a/crates/spark-connect/src/streaming.rs
+++ b/crates/spark-connect/src/streaming.rs
@@ -645,6 +645,7 @@ impl StreamingQuery {
                     return Ok(Some(StreamingQueryException {
                         message: msg,
                         error_class: exc.error_class.unwrap_or_default(),
+                        stack_trace: exc.stack_trace,
                     }));
                 }
             }
@@ -725,6 +726,8 @@ pub struct StreamingQueryStatus {
 pub struct StreamingQueryException {
     pub message: String,
     pub error_class: String,
+    /// The JVM stack trace of the exception, if the server sends it.
+    pub stack_trace: Option<String>,
 }
 
 /// An iterator over streaming query listener events from the server.
diff --git a/python/pyspark/sql/streaming/__init__.py 
b/python/pyspark/sql/streaming/__init__.py
index 381df5a..c08cf4b 100644
--- a/python/pyspark/sql/streaming/__init__.py
+++ b/python/pyspark/sql/streaming/__init__.py
@@ -23,9 +23,8 @@ from pyspark.sql.streaming.readwriter import (
 from pyspark.sql.streaming.query import (
     StreamingQuery,
     StreamingQueryManager,
-    StreamingQueryStatus,
-    StreamingQueryException,
 )
+from pyspark.errors import StreamingQueryException
 from pyspark.sql.streaming.datasource import (
     ReadLimit,
     ReadAllAvailable,
@@ -49,7 +48,6 @@ __all__ = [
     "Trigger",
     "StreamingQuery",
     "StreamingQueryManager",
-    "StreamingQueryStatus",
     "StreamingQueryException",
     "ReadLimit",
     "ReadAllAvailable",
diff --git a/python/pyspark/sql/streaming/query.py 
b/python/pyspark/sql/streaming/query.py
index 9afbf2a..919fcb8 100644
--- a/python/pyspark/sql/streaming/query.py
+++ b/python/pyspark/sql/streaming/query.py
@@ -40,9 +40,9 @@ from pyspark.sql.streaming.listener import (
 from pyspark._pyspark import (
     StreamingQuery as _StreamingQuery,
     StreamingQueryManager as _StreamingQueryManager,
-    StreamingQueryStatus as _StreamingQueryStatus,
-    StreamingQueryException as _StreamingQueryException,
 )
+# As PySpark's, the exception of a failed query is that of pyspark.errors.
+from pyspark.errors import StreamingQueryException  # noqa: F401
 
 if TYPE_CHECKING:
     from pyspark.sql.session import SparkSession
@@ -55,8 +55,6 @@ _QUERY_IDLE_EVENT = 3
 
 # Re-export the core classes from the Rust extension
 StreamingQuery = _StreamingQuery
-StreamingQueryStatus = _StreamingQueryStatus
-StreamingQueryException = _StreamingQueryException
 
 
 def _dispatch_listener_event(listener, event_type, event_json):
@@ -86,9 +84,4 @@ def _dispatch_listener_event(listener, event_type, 
event_json):
 StreamingQueryManager = _StreamingQueryManager
 
 
-__all__ = [
-    "StreamingQuery",
-    "StreamingQueryManager",
-    "StreamingQueryStatus",
-    "StreamingQueryException",
-]
+__all__ = ["StreamingQuery", "StreamingQueryManager"]
diff --git a/python/tests/test_dropin_e2e.py b/python/tests/test_dropin_e2e.py
index 91445ae..9bb0ff7 100644
--- a/python/tests/test_dropin_e2e.py
+++ b/python/tests/test_dropin_e2e.py
@@ -1888,6 +1888,42 @@ def 
test_dataframes_of_a_stream_are_streaming_as_pyspark(session):
     assert session.range(1).isStreaming is False
 
 
+def 
test_streaming_queries_return_the_status_and_the_exception_of_pyspark(session):
+    # As PySpark's: the status of a query is a dict, and the exception of a 
failed query is
+    # the StreamingQueryException of pyspark.errors, of the message of the 
server and its JVM
+    # stack trace, which awaitTermination raises.
+    import time
+
+    from pyspark.errors import StreamingQueryException
+    from pyspark.sql import functions as F
+
+    stream = session.readStream.format("rate").load()
+    query = stream.writeStream.format("noop").start()
+    try:
+        assert type(query.status) is dict
+        assert sorted(query.status) == ["isDataAvailable", "isTriggerActive", 
"message"]
+        query.explain(1)
+        assert query.exception() is None
+    finally:
+        query.stop()
+    failing = stream.select(F.assert_true(F.col("value") < 
0)).writeStream.format("noop")
+    failed = failing.start()
+    try:
+        for _ in range(120):
+            if not failed.isActive:
+                break
+            time.sleep(0.5)
+        exception = failed.exception()
+        assert isinstance(exception, StreamingQueryException)
+        message = str(exception)
+        assert message.startswith("(STREAM_FAILED) [USER_RAISED_EXCEPTION]")
+        assert "JVM stacktrace:" in message
+        with pytest.raises(StreamingQueryException):
+            failed.awaitTermination(1)
+    finally:
+        failed.stop()
+
+
 def 
test_stream_writer_starts_a_query_of_its_real_time_trigger_as_pyspark(session):
     # As PySpark's, the real-time trigger reaches the server, which runs 
real-time mode for
     # some sinks only, not the memory sink.
diff --git a/python/tests/test_dropin_offline.py 
b/python/tests/test_dropin_offline.py
index f2a04b8..6a93c53 100644
--- a/python/tests/test_dropin_offline.py
+++ b/python/tests/test_dropin_offline.py
@@ -9323,6 +9323,40 @@ def 
test_dataframe_methods_raise_the_argument_errors_of_pyspark():
             call()
 
 
+def test_streaming_queries_check_their_timeouts_as_pyspark(mock_connect):
+    # As PySpark's: StreamingQueryException of pyspark.sql.streaming is that of
+    # pyspark.errors; the waits of a query and of spark.streams raise 
VALUE_NOT_POSITIVE of a
+    # timeout that is not a positive number, of the name of its type; and 
explain takes an
+    # int as a bool, but no other value.
+    import pyspark.errors
+    import pyspark.sql.streaming
+    import pyspark.sql.streaming.query
+    from pyspark.errors import PySparkValueError
+
+    exception = pyspark.errors.StreamingQueryException
+    assert pyspark.sql.streaming.StreamingQueryException is exception
+    assert pyspark.sql.streaming.query.StreamingQueryException is exception
+    assert not hasattr(pyspark.sql.streaming, "StreamingQueryStatus")
+    spark, service = mock_connect
+    # ExecutePlanResponse.write_stream_operation_start_result=8: query_id=1 
(id=1, run_id=2).
+    service.execute_payload = message_field(
+        8, message_field(1, message_field(1, "q") + message_field(2, "r"))
+    )
+    query = 
spark.readStream.format("rate").load().writeStream.format("noop").start()
+    for timeout in (0, -1, 0.0, "1", [1]):
+        for wait in (query.awaitTermination, 
spark.streams.awaitAnyTermination):
+            with pytest.raises(PySparkValueError) as e:
+                wait(timeout)
+            assert e.value.getCondition() == "VALUE_NOT_POSITIVE"
+            assert e.value.getMessageParameters() == {
+                "arg_name": "timeout",
+                "arg_value": type(timeout).__name__,
+            }
+    for extended in (1.5, "x", None):
+        with pytest.raises(TypeError, match="but expected one of"):
+            query.explain(extended)
+
+
 def test_stream_writer_setters_mutate_and_return_self(offline_spark):
     w = offline_spark.readStream.format("rate").load().writeStream
     assert w.outputMode("append") is w


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to