This is an automated email from the ASF dual-hosted git repository.
HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git
The following commit(s) were added to refs/heads/master by this push:
new 0043c6d [SPARK-61055][CONNECT][PYTHON] Return the status and the
exception of a StreamingQuery, and check the timeouts of its waits, in the
Python client as PySpark does
0043c6d is described below
commit 0043c6d227e0e6769555949384733cb6e54fdf90
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Sat Nov 7 12:15:40 2026 +0900
[SPARK-61055][CONNECT][PYTHON] Return the status and the exception of a
StreamingQuery, and check the timeouts of its waits, in the Python client as
PySpark does
### What changes were proposed in this pull request?
[SPARK-61055](https://issues.apache.org/jira/browse/SPARK-61055):
`StreamingQuery.status` and `StreamingQuery.exception()` now return what
PySpark Connect's do, and the waits of a query check their timeouts as
PySpark's.
- `query.status` is a dict of the `message`, `isDataAvailable` and
`isTriggerActive` of the query.
- `query.exception()` of a failed query returns the
`StreamingQueryException` of `pyspark.errors`:
- of the message of the server, without the type of its Java exception,
and with its JVM stack trace
- of the error class as its reason
- `pyspark.sql.streaming.StreamingQueryException` and
`pyspark.sql.streaming.query.StreamingQueryException` are that class of
`pyspark.errors`, as in PySpark.
- `awaitTermination(timeout)` and
`spark.streams.awaitAnyTermination(timeout)` raise `VALUE_NOT_POSITIVE` of
`timeout` for a timeout that is not a positive number.
- `explain(extended)` takes an int as a bool, as the proto field of PySpark
does.
The `StreamingQueryStatus` and `StreamingQueryException` classes of the
extension, which PySpark does not have, are removed. The core
`StreamingQueryException` now keeps the stack trace of the server.
### Why are the changes needed?
- `query.status` was an object of `status_message`, `is_data_available` and
`is_trigger_active` attributes, neither subscriptable nor iterable:
`query.status["message"]` raised a `TypeError`.
- `query.exception()` returned an object that was not an exception, whose
`str()` was its repr.
- `pyspark.sql.streaming.StreamingQueryException` was that class, so
`except StreamingQueryException:` raised a `TypeError`.
- The timeouts raised `INVALID_TIMEOUT` or `TypeError`s, and `explain(1)`
raised a `TypeError`.
### Does this PR introduce _any_ user-facing change?
Yes. These return and raise what PySpark 4.3.0 does.
### How was this patch tested?
- A new live test checks the dict of `status`, the
`StreamingQueryException` of a failed query, its message and JVM stack trace,
and that `awaitTermination` raises it. It passes with official PySpark 4.3.0,
and fails without this change.
- A new offline test checks the class of `StreamingQueryException`, the
errors of the timeouts, and those of `explain`.
- On a local server, `status` and `exception()` of a failed query give the
dict, the class and the 125 lines of the message of official PySpark 4.3.0.
- The lib tests of both crates (580 passed), the build of all tests, also
with `wasm-udf`, the offline Python suites (1847 passed, 8 skipped), and the
live drop-in and Arrow suites (116 passed, without the test of
`StreamingQuery.explain`, which hangs on a slow machine and #260 fixes) and
live cases (10 passed) against a local Spark 4.3.0 server. Clippy reports no
diagnostics in the changed lines.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
This pull request and its description were written by Isaac.
Closes #263 from HyukjinKwon/SPARK-61055-query-results.
Authored-by: Hyukjin Kwon <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
---
crates/pyspark-rs/src/lib.rs | 4 +-
crates/pyspark-rs/src/streaming.rs | 148 +++++++++++++++----------------
crates/spark-connect/src/streaming.rs | 3 +
python/pyspark/sql/streaming/__init__.py | 4 +-
python/pyspark/sql/streaming/query.py | 13 +--
python/tests/test_dropin_e2e.py | 36 ++++++++
python/tests/test_dropin_offline.py | 34 +++++++
7 files changed, 152 insertions(+), 90 deletions(-)
diff --git a/crates/pyspark-rs/src/lib.rs b/crates/pyspark-rs/src/lib.rs
index d9cbbad..43d462c 100644
--- a/crates/pyspark-rs/src/lib.rs
+++ b/crates/pyspark-rs/src/lib.rs
@@ -81,7 +81,7 @@ use session::{PyHook, PySparkSession, PySparkSessionBuilder};
use stat::PyStatFunctions;
use streaming::{
PyDataStreamReader, PyDataStreamWriter, PyListenerEventStream,
PyStreamingQuery,
- PyStreamingQueryException, PyStreamingQueryManager,
PyStreamingQueryStatus, PyTrigger,
+ PyStreamingQueryManager, PyTrigger,
};
use types::{
PyAnsiIntervalType, PyAnyTimeType, PyArrayType, PyAtomicType,
PyBinaryType, PyBooleanType,
@@ -163,8 +163,6 @@ fn _pyspark(_py: Python<'_>, m: &Bound<'_, PyModule>) ->
PyResult<()> {
m.add_class::<PyDataStreamWriter>()?;
m.add_class::<PyTrigger>()?;
m.add_class::<PyStreamingQuery>()?;
- m.add_class::<PyStreamingQueryStatus>()?;
- m.add_class::<PyStreamingQueryException>()?;
m.add_class::<PyStreamingQueryManager>()?;
m.add_class::<PyListenerEventStream>()?;
diff --git a/crates/pyspark-rs/src/streaming.rs
b/crates/pyspark-rs/src/streaming.rs
index c1f6130..e75107a 100644
--- a/crates/pyspark-rs/src/streaming.rs
+++ b/crates/pyspark-rs/src/streaming.rs
@@ -4,8 +4,7 @@ use pyo3::prelude::*;
use pyo3::types::PyDict;
use spark_connect::streaming::{
DataStreamReader, DataStreamWriter, ListenerEventStream, StreamingQuery,
- StreamingQueryException, StreamingQueryListener,
StreamingQueryListenerEvent,
- StreamingQueryManager, StreamingQueryStatus, Trigger,
+ StreamingQueryListener, StreamingQueryListenerEvent,
StreamingQueryManager, Trigger,
};
use spark_connect::udf::PythonUDFPayload;
use std::collections::HashMap;
@@ -811,73 +810,46 @@ impl PyDataStreamWriter {
}
}
-/// Python wrapper for StreamingQueryStatus.
-#[pyclass(
- name = "StreamingQueryStatus",
- module = "pyspark.sql.streaming.query",
- from_py_object
-)]
-#[derive(Clone)]
-pub struct PyStreamingQueryStatus {
- inner: StreamingQueryStatus,
-}
-
-impl PyStreamingQueryStatus {
- pub fn new(status: StreamingQueryStatus) -> Self {
- PyStreamingQueryStatus { inner: status }
- }
-}
-
-#[pymethods]
-impl PyStreamingQueryStatus {
- #[getter]
- fn is_active(&self) -> bool {
- self.inner.is_active
- }
-
- #[getter]
- fn status_message(&self) -> String {
- self.inner.status_message.clone()
- }
-
- #[getter]
- fn is_data_available(&self) -> bool {
- self.inner.is_data_available
- }
-
- #[getter]
- fn is_trigger_active(&self) -> bool {
- self.inner.is_trigger_active
+/// The timeout of `awaitTermination` and `awaitAnyTermination`, in seconds,
as PySpark's take
+/// it: an int or a float, which must be positive, else `VALUE_NOT_POSITIVE`
of the name of its
+/// type.
+fn positive_timeout(timeout: Option<&Bound<'_, PyAny>>) ->
PyResult<Option<f64>> {
+ let Some(timeout) = timeout else {
+ return Ok(None);
+ };
+ let number = timeout.is_instance_of::<pyo3::types::PyInt>()
+ || timeout.is_instance_of::<pyo3::types::PyFloat>();
+ match number.then(|| timeout.extract::<f64>()).transpose()? {
+ Some(seconds) if seconds > 0.0 => Ok(Some(seconds)),
+ _ => {
+ let type_name = timeout.get_type().name()?.to_string();
+ Err(crate::column::raise_pyspark(
+ timeout.py(),
+ "PySparkValueError",
+ "VALUE_NOT_POSITIVE",
+ &[("arg_name", "timeout"), ("arg_value", &type_name)],
+ ))
+ }
}
}
-/// Python wrapper for StreamingQueryException.
-#[pyclass(
- name = "StreamingQueryException",
- module = "pyspark.sql.streaming.query",
- from_py_object
-)]
-#[derive(Clone)]
-pub struct PyStreamingQueryException {
- inner: StreamingQueryException,
-}
+/// A bool argument that PySpark sets on a proto bool field, which takes a
bool or an int, and
+/// raises the `TypeError` of protobuf for another value.
+pub(crate) struct ProtoBool(pub(crate) bool);
-impl PyStreamingQueryException {
- pub fn new(exc: StreamingQueryException) -> Self {
- PyStreamingQueryException { inner: exc }
- }
-}
+impl<'a, 'py> FromPyObject<'a, 'py> for ProtoBool {
+ type Error = PyErr;
-#[pymethods]
-impl PyStreamingQueryException {
- #[getter]
- fn message(&self) -> String {
- self.inner.message.clone()
- }
-
- #[getter]
- fn error_class(&self) -> String {
- self.inner.error_class.clone()
+ fn extract(obj: Borrowed<'a, 'py, PyAny>) -> PyResult<Self> {
+ if obj.is_instance_of::<pyo3::types::PyInt>() {
+ return Ok(ProtoBool(obj.is_truthy()?));
+ }
+ let shown = obj.repr()?;
+ Err(pyo3::exceptions::PyTypeError::new_err(format!(
+ "Cannot set
spark.connect.StreamingQueryCommand.ExplainCommand.extended to {shown}: \
+ {shown} has type {}, but expected one of: (<class 'bool'>, <class
'int'>)",
+ obj.get_type().repr()?
+ )))
}
}
@@ -915,10 +887,16 @@ impl PyStreamingQuery {
self.inner.is_active().to_pyerr()
}
+ /// The status of the query, as PySpark's: a dict of its `message`,
`isDataAvailable` and
+ /// `isTriggerActive`.
#[getter]
- fn status(&self) -> PyResult<PyStreamingQueryStatus> {
+ fn status<'py>(&self, py: Python<'py>) -> PyResult<Bound<'py, PyDict>> {
let status = self.inner.status().to_pyerr()?;
- Ok(PyStreamingQueryStatus::new(status))
+ let dict = PyDict::new(py);
+ dict.set_item("message", status.status_message)?;
+ dict.set_item("isDataAvailable", status.is_data_available)?;
+ dict.set_item("isTriggerActive", status.is_trigger_active)?;
+ Ok(dict)
}
fn stop(&self) -> PyResult<()> {
@@ -926,7 +904,8 @@ impl PyStreamingQuery {
}
#[pyo3(signature = (timeout=None))]
- fn awaitTermination(&self, timeout: Option<f64>) -> PyResult<Option<bool>>
{
+ fn awaitTermination(&self, timeout: Option<&Bound<'_, PyAny>>) ->
PyResult<Option<bool>> {
+ let timeout = positive_timeout(timeout)?;
self.inner.await_termination(timeout).to_pyerr()
}
@@ -946,16 +925,36 @@ impl PyStreamingQuery {
/// Print the plan of the query with Python's `print`, as PySpark's
/// `StreamingQuery.explain(extended=False)`.
- #[pyo3(signature = (extended=false))]
- fn explain(&self, py: Python<'_>, extended: bool) -> PyResult<()> {
- let plan = py.detach(|| self.inner.explain(extended)).to_pyerr()?;
+ #[pyo3(signature = (extended=ProtoBool(false)))]
+ fn explain(&self, py: Python<'_>, extended: ProtoBool) -> PyResult<()> {
+ let plan = py
+ .detach(|| self.inner.explain(extended.0).map_err(Box::new))
+ .to_pyerr()?;
py.import("builtins")?.call_method1("print", (plan,))?;
Ok(())
}
- fn exception(&self) -> PyResult<Option<PyStreamingQueryException>> {
- let exc = self.inner.exception().to_pyerr()?;
- Ok(exc.map(PyStreamingQueryException::new))
+ /// The exception of a failed query, as PySpark's: the
`StreamingQueryException` of
+ /// `pyspark.errors.exceptions.connect`, of the message of the server
without the type of
+ /// its Java exception, and with its JVM stack trace, and of the error
class as its reason.
+ fn exception(&self, py: Python<'_>) -> PyResult<Option<Py<PyAny>>> {
+ let Some(exception) = self.inner.exception().to_pyerr()? else {
+ return Ok(None);
+ };
+ // The message is the `toString` of the Java StreamingQueryException:
drop its type.
+ let mut message = match exception.message.split_once(": ") {
+ Some((_, message)) => message.to_string(),
+ None => exception.message,
+ };
+ if let Some(stack_trace) = exception.stack_trace {
+ message.push_str(&format!("\n\nJVM stacktrace:\n{stack_trace}"));
+ }
+ let class = py
+ .import("pyspark.errors.exceptions.connect")?
+ .getattr("StreamingQueryException")?;
+ let kwargs = PyDict::new(py);
+ kwargs.set_item("reason", exception.error_class)?;
+ Ok(Some(class.call((message,), Some(&kwargs))?.unbind()))
}
}
@@ -985,7 +984,8 @@ impl PyStreamingQueryManager {
}
#[pyo3(signature = (timeout=None))]
- fn awaitAnyTermination(&self, timeout: Option<f64>) ->
PyResult<Option<bool>> {
+ fn awaitAnyTermination(&self, timeout: Option<&Bound<'_, PyAny>>) ->
PyResult<Option<bool>> {
+ let timeout = positive_timeout(timeout)?;
self.inner.await_any_termination(timeout).to_pyerr()
}
diff --git a/crates/spark-connect/src/streaming.rs
b/crates/spark-connect/src/streaming.rs
index e1648bb..80f17ac 100644
--- a/crates/spark-connect/src/streaming.rs
+++ b/crates/spark-connect/src/streaming.rs
@@ -645,6 +645,7 @@ impl StreamingQuery {
return Ok(Some(StreamingQueryException {
message: msg,
error_class: exc.error_class.unwrap_or_default(),
+ stack_trace: exc.stack_trace,
}));
}
}
@@ -725,6 +726,8 @@ pub struct StreamingQueryStatus {
pub struct StreamingQueryException {
pub message: String,
pub error_class: String,
+ /// The JVM stack trace of the exception, if the server sends it.
+ pub stack_trace: Option<String>,
}
/// An iterator over streaming query listener events from the server.
diff --git a/python/pyspark/sql/streaming/__init__.py
b/python/pyspark/sql/streaming/__init__.py
index 381df5a..c08cf4b 100644
--- a/python/pyspark/sql/streaming/__init__.py
+++ b/python/pyspark/sql/streaming/__init__.py
@@ -23,9 +23,8 @@ from pyspark.sql.streaming.readwriter import (
from pyspark.sql.streaming.query import (
StreamingQuery,
StreamingQueryManager,
- StreamingQueryStatus,
- StreamingQueryException,
)
+from pyspark.errors import StreamingQueryException
from pyspark.sql.streaming.datasource import (
ReadLimit,
ReadAllAvailable,
@@ -49,7 +48,6 @@ __all__ = [
"Trigger",
"StreamingQuery",
"StreamingQueryManager",
- "StreamingQueryStatus",
"StreamingQueryException",
"ReadLimit",
"ReadAllAvailable",
diff --git a/python/pyspark/sql/streaming/query.py
b/python/pyspark/sql/streaming/query.py
index 9afbf2a..919fcb8 100644
--- a/python/pyspark/sql/streaming/query.py
+++ b/python/pyspark/sql/streaming/query.py
@@ -40,9 +40,9 @@ from pyspark.sql.streaming.listener import (
from pyspark._pyspark import (
StreamingQuery as _StreamingQuery,
StreamingQueryManager as _StreamingQueryManager,
- StreamingQueryStatus as _StreamingQueryStatus,
- StreamingQueryException as _StreamingQueryException,
)
+# As PySpark's, the exception of a failed query is that of pyspark.errors.
+from pyspark.errors import StreamingQueryException # noqa: F401
if TYPE_CHECKING:
from pyspark.sql.session import SparkSession
@@ -55,8 +55,6 @@ _QUERY_IDLE_EVENT = 3
# Re-export the core classes from the Rust extension
StreamingQuery = _StreamingQuery
-StreamingQueryStatus = _StreamingQueryStatus
-StreamingQueryException = _StreamingQueryException
def _dispatch_listener_event(listener, event_type, event_json):
@@ -86,9 +84,4 @@ def _dispatch_listener_event(listener, event_type,
event_json):
StreamingQueryManager = _StreamingQueryManager
-__all__ = [
- "StreamingQuery",
- "StreamingQueryManager",
- "StreamingQueryStatus",
- "StreamingQueryException",
-]
+__all__ = ["StreamingQuery", "StreamingQueryManager"]
diff --git a/python/tests/test_dropin_e2e.py b/python/tests/test_dropin_e2e.py
index 91445ae..9bb0ff7 100644
--- a/python/tests/test_dropin_e2e.py
+++ b/python/tests/test_dropin_e2e.py
@@ -1888,6 +1888,42 @@ def
test_dataframes_of_a_stream_are_streaming_as_pyspark(session):
assert session.range(1).isStreaming is False
+def
test_streaming_queries_return_the_status_and_the_exception_of_pyspark(session):
+ # As PySpark's: the status of a query is a dict, and the exception of a
failed query is
+ # the StreamingQueryException of pyspark.errors, of the message of the
server and its JVM
+ # stack trace, which awaitTermination raises.
+ import time
+
+ from pyspark.errors import StreamingQueryException
+ from pyspark.sql import functions as F
+
+ stream = session.readStream.format("rate").load()
+ query = stream.writeStream.format("noop").start()
+ try:
+ assert type(query.status) is dict
+ assert sorted(query.status) == ["isDataAvailable", "isTriggerActive",
"message"]
+ query.explain(1)
+ assert query.exception() is None
+ finally:
+ query.stop()
+ failing = stream.select(F.assert_true(F.col("value") <
0)).writeStream.format("noop")
+ failed = failing.start()
+ try:
+ for _ in range(120):
+ if not failed.isActive:
+ break
+ time.sleep(0.5)
+ exception = failed.exception()
+ assert isinstance(exception, StreamingQueryException)
+ message = str(exception)
+ assert message.startswith("(STREAM_FAILED) [USER_RAISED_EXCEPTION]")
+ assert "JVM stacktrace:" in message
+ with pytest.raises(StreamingQueryException):
+ failed.awaitTermination(1)
+ finally:
+ failed.stop()
+
+
def
test_stream_writer_starts_a_query_of_its_real_time_trigger_as_pyspark(session):
# As PySpark's, the real-time trigger reaches the server, which runs
real-time mode for
# some sinks only, not the memory sink.
diff --git a/python/tests/test_dropin_offline.py
b/python/tests/test_dropin_offline.py
index f2a04b8..6a93c53 100644
--- a/python/tests/test_dropin_offline.py
+++ b/python/tests/test_dropin_offline.py
@@ -9323,6 +9323,40 @@ def
test_dataframe_methods_raise_the_argument_errors_of_pyspark():
call()
+def test_streaming_queries_check_their_timeouts_as_pyspark(mock_connect):
+ # As PySpark's: StreamingQueryException of pyspark.sql.streaming is that of
+ # pyspark.errors; the waits of a query and of spark.streams raise
VALUE_NOT_POSITIVE of a
+ # timeout that is not a positive number, of the name of its type; and
explain takes an
+ # int as a bool, but no other value.
+ import pyspark.errors
+ import pyspark.sql.streaming
+ import pyspark.sql.streaming.query
+ from pyspark.errors import PySparkValueError
+
+ exception = pyspark.errors.StreamingQueryException
+ assert pyspark.sql.streaming.StreamingQueryException is exception
+ assert pyspark.sql.streaming.query.StreamingQueryException is exception
+ assert not hasattr(pyspark.sql.streaming, "StreamingQueryStatus")
+ spark, service = mock_connect
+ # ExecutePlanResponse.write_stream_operation_start_result=8: query_id=1
(id=1, run_id=2).
+ service.execute_payload = message_field(
+ 8, message_field(1, message_field(1, "q") + message_field(2, "r"))
+ )
+ query =
spark.readStream.format("rate").load().writeStream.format("noop").start()
+ for timeout in (0, -1, 0.0, "1", [1]):
+ for wait in (query.awaitTermination,
spark.streams.awaitAnyTermination):
+ with pytest.raises(PySparkValueError) as e:
+ wait(timeout)
+ assert e.value.getCondition() == "VALUE_NOT_POSITIVE"
+ assert e.value.getMessageParameters() == {
+ "arg_name": "timeout",
+ "arg_value": type(timeout).__name__,
+ }
+ for extended in (1.5, "x", None):
+ with pytest.raises(TypeError, match="but expected one of"):
+ query.explain(extended)
+
+
def test_stream_writer_setters_mutate_and_return_self(offline_spark):
w = offline_spark.readStream.format("rate").load().writeStream
assert w.outputMode("append") is w
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]