This is an automated email from the ASF dual-hosted git repository.
HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git
The following commit(s) were added to refs/heads/master by this push:
new 93d2d07a [SPARK-61215][CONNECT][RUST] Match streaming listener parsing
and failure handling
93d2d07a is described below
commit 93d2d07a3fde156f10d98197d0d49795640bb8c7
Author: DB Tsai <[email protected]>
AuthorDate: Mon Nov 9 15:50:35 2026 +0900
[SPARK-61215][CONNECT][RUST] Match streaming listener parsing and failure
handling
### What changes were proposed in this pull request?
Propagate synchronous streaming query-start decoding errors from start()
and toTable(), including when no listeners are registered. Report asynchronous
transport, decoding and unknown-event errors once before clearing the affected
subscription. Decode each event once and share it across the listener snapshot,
including duplicate registrations.
Native Rust listeners retain a silent default error hook and can opt into
error reporting. Callback and warning hooks run without the listener-bus state
lock. Concurrent removal of a non-last listener waits through its event
callback or terminal error notification.
This independent patch is based on master
925dc876f63f1f779fe176f62575a0aa96cc6886, which already contains the listener
lifecycle and generation-scoped removal behavior.
### Why are the changes needed?
Malformed listener events can disappear or leave registrations active, and
terminal transport failures can clear listeners without PySpark's handler
warning. The caller should receive a malformed synchronous start event's
original exception, and asynchronous failures should emit the corresponding
handler warning while allowing listeners to be registered again.
### Does this PR introduce _any_ user-facing change?
Yes. Start-event decoding failures now reach the caller, and asynchronous
failures warn once and clear registrations. Individual callback Exceptions
continue to warn per listener while dispatch proceeds.
The documented limits remain:
- Normal stream EOF clears registrations silently; reference PySpark
retains them.
- An uncaught asynchronous callback BaseException, or a callback warning
promoted to an error, ends the subscription and clears registrations. Reference
PySpark can retain stale registrations. The terminal handler warning is best
effort if warning filters raise.
- Session stop() and close() cancel and clear listeners locally, without
listener-removal RPCs or waiting for active callbacks. The current callback
snapshot can finish.
- A registered listener can retain its session through listener.spark.
Initial listener RPC establishment follows the transport deadline/retry policy.
The inherited single-generation dispatch-marker boundary is unchanged.
### How was this patch tested?
- Formatting, release core/API library tests, integration-test compilation
with and without wasm-udf, and the extension build passed. Offline Python
recorded **2,605 passed and 8 inherited live-server skips**, including all **53
required listener cases**. All five new native unit tests passed. These were
executed on the pre-comment revision and carried to the current source through
exactly three documented comment-only replacements; they are not a fresh
complete execution on the comme [...]
- A fresh strict run on the current artifact passed **25 distinct Python
cases once each**, with all **75 setup/call/teardown reports** passing and no
required skips. Three selected native error regression unit cases also passed
separately. The strict run attested the loaded extension, shim and test
sources; it is not a stress-repeat result.
- Against a live Spark server, **7 native Python cases passed / 0 skipped**
and all three saved native streaming e2e callers passed, each **1 passed / 0
ignored**. The live Python selection deselected 194 other cases; its import
provenance is bound by the staging harness.
- Introspection across 12 API groups found no missing public names; Row had
no missing names and remained a tuple subclass. This checks names, not all
signatures or behavior. The no-stubs audit passed. Rustdoc exited 0 with two
inherited warnings in types.rs (unresolved from_json link and unclosed int
tag). Complete source review round 12 reported no findings.
- Workspace/all-target Clippy exited **101**. All 472 emitted source
diagnostics and 721 primary/secondary spans map to unchanged saved-master
statements and preserved current statements. This is source attribution, not an
executed master lint pass or successful Clippy run. These scoped results do not
certify the README's broad parity claim.
- This artifact's official configuration and DataFrame-creation modules ran
through its raw-transport plugin against a live JVM: **23 passed / 1 skipped /
0 failures or errors**. The current source, saved extension and reviewed source
match.
- Common transport regression evidence is carried from a separate pandas
artifact because all **53 shared transport/build/dependency inputs** match. Its
full ordinary phase covered **87 modules: 2,586 passed / 641 skipped / 1
manifest deselection**, with **one expected missing-catalog drift-check failure
and no unexpected failures**. Its separate cluster/resource phase covered **4
modules: 15 passed / 14 skipped**, including 10 functional passes and 5 guard
passes. Only that donor art [...]
The completed transport evidence and scoped native results were assessed
together. Saved harness, source, artifact and endpoint evidence are retained;
complete official/dependency whole-tree start/end attestations, per-case skip
provenance and raw per-module pytest exit codes were not all recorded. The
reported totals count test executions and do not establish whole-client parity.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: OpenAI Codex
This pull request and its description were written by OpenAI Codex.
Closes #398 from dbtsai/codex/residual-streaming-event-errors-master-925dc.
Authored-by: DB Tsai <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
---
crates/pyspark-rs/src/streaming.rs | 172 +++++++++--
crates/spark-connect/src/streaming.rs | 513 +++++++++++++++++++++++++++++--
docs/streaming.md | 29 ++
python/pyspark/sql/streaming/query.py | 48 ++-
python/tests/_mock_connect_server.py | 6 +-
python/tests/streaming_listener_cases.py | 300 +++++++++++++++++-
6 files changed, 1003 insertions(+), 65 deletions(-)
diff --git a/crates/pyspark-rs/src/streaming.rs
b/crates/pyspark-rs/src/streaming.rs
index 32a5b309..50cda3d8 100644
--- a/crates/pyspark-rs/src/streaming.rs
+++ b/crates/pyspark-rs/src/streaming.rs
@@ -7,43 +7,171 @@ use spark_connect::streaming::{
StreamingQueryListener, StreamingQueryListenerEvent,
StreamingQueryManager, Trigger,
};
use spark_connect::udf::PythonUDFPayload;
+use spark_connect_core::error::{Result as SparkResult, SparkError};
use std::collections::HashMap;
use std::sync::Arc;
use crate::dataframe::{PyDataFrame, WriterState};
use crate::errors::ResultExt;
-/// Adapts a Python `StreamingQueryListener` to the native Rust listener
trait: on each
-/// event it acquires the GIL and calls the Python-side dispatch helper (which
builds the
-/// typed event object and invokes the right `onQuery*` callback).
+/// Adapts Python listeners to the native bus: deserialize one typed event for
the whole
+/// snapshot, then dispatch it through Python's callback-exception isolation
helper. Decode
+/// errors return to the bus, and terminal failures produce the reference
handler warning.
struct PyListenerAdapter {
listener: Py<PyAny>,
}
+/// An unisolated dispatch failure. Keep Python errors intact so synchronous
starts
+/// can re-raise their original type; native listener errors remain typed
SparkErrors.
+enum ListenerDispatchError {
+ Python(PyErr),
+ Native(Box<SparkError>),
+}
+
+impl From<PyErr> for ListenerDispatchError {
+ fn from(error: PyErr) -> Self {
+ Self::Python(error)
+ }
+}
+
+/// Decode before any callback, sharing one Python object across every
registration.
+/// The same path handles synchronous starts and asynchronous protocol events,
including
+/// mixed native/Python snapshots. Keep native errors typed until the
reporting boundary.
+fn post_listener_event<'a>(
+ py: Python<'_>,
+ listeners: impl IntoIterator<Item = &'a dyn StreamingQueryListener>,
+ event: &StreamingQueryListenerEvent,
+ started: bool,
+) -> std::result::Result<(), ListenerDispatchError> {
+ let module = py.import("pyspark.sql.streaming.query")?;
+ let kwargs = PyDict::new(py);
+ kwargs.set_item("started", started)?;
+ let decoded = module.call_method(
+ "_deserialize_listener_event",
+ (event.event_type, event.event_json.as_str()),
+ Some(&kwargs),
+ )?;
+ for listener in listeners {
+ match listener
+ .as_any()
+ .and_then(|any| any.downcast_ref::<PyListenerAdapter>())
+ {
+ Some(adapter) => {
+ module.call_method1(
+ "_dispatch_listener_event",
+ (adapter.listener.bind(py), &decoded),
+ )?;
+ }
+ // Native callbacks keep their usual ability to wait for Python
work.
+ None if started => py.detach(|| listener.on_event(event)),
+ None => py
+ .detach(|| listener.try_on_event(event).map_err(Box::new))
+ .map_err(ListenerDispatchError::Native)?,
+ }
+ }
+ Ok(())
+}
+
+/// Any Python exception not isolated by the helper terminates the
subscription: this
+/// includes decoding errors, callback BaseException, and warnings promoted to
errors.
+/// Async reporting keeps only str(), without allocating an unconsumed
callback-registry id.
+fn async_dispatch_error(py: Python<'_>, error: ListenerDispatchError) ->
Box<SparkError> {
+ match error {
+ ListenerDispatchError::Python(error) => {
+ Box::new(SparkError::connect_msg(error.value(py).to_string()))
+ }
+ ListenerDispatchError::Native(error) => error,
+ }
+}
+
impl StreamingQueryListener for PyListenerAdapter {
fn on_event(&self, event: &StreamingQueryListenerEvent) {
+ // This infallible entry point is also used by the core's default
synchronous
+ // start handler. Python writer calls use post_started_event to
propagate errors.
Python::attach(|py| {
- let dispatch = py
- .import("pyspark.sql.streaming.query")
- .and_then(|m| m.getattr("_dispatch_listener_event"));
- if let Ok(dispatch) = dispatch {
- // Swallow listener/callback errors so one bad listener cannot
kill the
- // dispatch thread (reference pyspark also isolates callback
exceptions).
- let _ = dispatch.call1((
- self.listener.bind(py),
- event.event_type,
- event.event_json.as_str(),
- ));
- }
+ let _ = post_listener_event(
+ py,
+ std::iter::once(self as &dyn StreamingQueryListener),
+ event,
+ true,
+ );
});
}
+ fn try_on_event(&self, event: &StreamingQueryListenerEvent) ->
SparkResult<()> {
+ Python::attach(|py| {
+ post_listener_event(
+ py,
+ std::iter::once(self as &dyn StreamingQueryListener),
+ event,
+ false,
+ )
+ .map_err(|error| async_dispatch_error(py, error))
+ })
+ .map_err(|error| *error)
+ }
+
+ fn try_post_to_all(
+ &self,
+ listeners: &[Arc<dyn StreamingQueryListener>],
+ event: &StreamingQueryListenerEvent,
+ ) -> Option<SparkResult<()>> {
+ Some(
+ Python::attach(|py| {
+ post_listener_event(py, listeners.iter().map(Arc::as_ref),
event, false)
+ .map_err(|error| async_dispatch_error(py, error))
+ })
+ .map_err(|error| *error),
+ )
+ }
+
+ fn on_stream_error(&self, error: &SparkError) -> bool {
+ Python::attach(|py| {
+ let message =
crate::errors::spark_error_to_py_exception(error.clone())
+ .value(py)
+ .str()
+ .and_then(|text| Ok(text.to_cow()?.into_owned()))
+ .unwrap_or_else(|_| error.message());
+ let _ = warn(
+ py,
+ format!(
+ "StreamingQueryListenerBus Handler thread received
exception, all client side \
+ listeners are removed and handler thread is terminated.
The error is: {}",
+ message
+ ),
+ );
+ });
+ true
+ }
+
/// The adapter, of which `StreamingQueryManager._sqlb` gives back the
Python listener.
fn as_any(&self) -> Option<&dyn std::any::Any> {
Some(self)
}
}
+/// Parse a synchronous started event once, including when there are no
listeners, and
+/// dispatch its typed object before start/toTable returns. Preserve the
original Python
+/// decoding exception across the detached core call.
+#[allow(clippy::result_large_err)] // The core started-event handler requires
SparkResult.
+fn post_started_event(
+ manager: &StreamingQueryManager,
+ event: &StreamingQueryListenerEvent,
+) -> SparkResult<()> {
+ Python::attach(|py| {
+ let listeners = manager.listeners();
+ post_listener_event(py, listeners.iter().map(Arc::as_ref), event,
true).map_err(|error| {
+ match error {
+ ListenerDispatchError::Python(error) => {
+ Box::new(crate::errors::callback_error(error))
+ }
+ ListenerDispatchError::Native(error) => error,
+ }
+ })
+ })
+ .map_err(|error| *error)
+}
+
/// Apply a named read option to a streaming reader when supplied (mirrors the
batch
/// `set_opt` in readwriter.rs): `None` leaves the option unset (not the
string "None");
/// booleans lowercase to "true"/"false" via `coerce_option_value`.
@@ -856,8 +984,11 @@ impl PyDataStreamWriter {
) -> PyResult<PyStreamingQuery> {
let writer = self.apply_write_opts(format, outputMode, partitionBy,
queryName, options)?;
// Memory/console/foreach sinks take no path; the core treats "" as
unset.
- let query =
- crate::dataframe::detached(py, ||
writer.start(path.unwrap_or("")).map_err(Box::new))?;
+ let query = crate::dataframe::detached(py, || {
+ writer
+ .start_with_event_handler(path.unwrap_or(""),
post_started_event)
+ .map_err(Box::new)
+ })?;
Ok(PyStreamingQuery::new(query))
}
@@ -874,8 +1005,11 @@ impl PyDataStreamWriter {
options: Option<&Bound<'_, PyDict>>,
) -> PyResult<PyStreamingQuery> {
let writer = self.apply_write_opts(format, outputMode, partitionBy,
queryName, options)?;
- let query =
- crate::dataframe::detached(py, ||
writer.to_table(tableName).map_err(Box::new))?;
+ let query = crate::dataframe::detached(py, || {
+ writer
+ .to_table_with_event_handler(tableName, post_started_event)
+ .map_err(Box::new)
+ })?;
Ok(PyStreamingQuery::new(query))
}
diff --git a/crates/spark-connect/src/streaming.rs
b/crates/spark-connect/src/streaming.rs
index a736b0be..67e7dfb6 100644
--- a/crates/spark-connect/src/streaming.rs
+++ b/crates/spark-connect/src/streaming.rs
@@ -384,19 +384,49 @@ impl DataStreamWriter {
}
/// Start the streaming query writing to a path, returning a
StreamingQuery handle.
- pub fn start(mut self, path: &str) -> Result<StreamingQuery> {
+ pub fn start(self, path: &str) -> Result<StreamingQuery> {
+ self.start_with_event_handler(path, |manager, event| {
+ manager.post_to_all(event);
+ Ok(())
+ })
+ }
+
+ /// Start a query with a language-specific handler for its synchronous
started event.
+ /// The handler runs even when no listeners are registered and must
deserialize the event
+ /// before dispatching it. A handler error is returned instead of the
query handle.
+ pub fn start_with_event_handler(
+ mut self,
+ path: &str,
+ on_started: impl FnOnce(&StreamingQueryManager,
&StreamingQueryListenerEvent) -> Result<()>,
+ ) -> Result<StreamingQuery> {
self.path = Some(path.to_string());
- self._start_internal()
+ self._start_internal(on_started)
}
/// Start the streaming query writing to a table, returning a
StreamingQuery handle.
- pub fn to_table(mut self, table_name: &str) -> Result<StreamingQuery> {
+ pub fn to_table(self, table_name: &str) -> Result<StreamingQuery> {
+ self.to_table_with_event_handler(table_name, |manager, event| {
+ manager.post_to_all(event);
+ Ok(())
+ })
+ }
+
+ /// The table-writing counterpart of
+ /// [`start_with_event_handler`](Self::start_with_event_handler).
+ pub fn to_table_with_event_handler(
+ mut self,
+ table_name: &str,
+ on_started: impl FnOnce(&StreamingQueryManager,
&StreamingQueryListenerEvent) -> Result<()>,
+ ) -> Result<StreamingQuery> {
self.table_name = Some(table_name.to_string());
- self._start_internal()
+ self._start_internal(on_started)
}
/// Internal method to build and execute the write stream command.
- fn _start_internal(self) -> Result<StreamingQuery> {
+ fn _start_internal(
+ self,
+ on_started: impl FnOnce(&StreamingQueryManager,
&StreamingQueryListenerEvent) -> Result<()>,
+ ) -> Result<StreamingQuery> {
let mut write_op = proto::WriteStreamOperationStart::default();
write_op.input = Some(crate::plan::relation_with_ids(
&self.plan,
@@ -490,12 +520,13 @@ impl DataStreamWriter {
// with the result of its start, as the session has client-side
listeners, and the client
// posts it to them before it gives back the query.
if let Some(event_json) = started_event {
- self.session
- .streams()
- .post_to_all(&StreamingQueryListenerEvent {
+ on_started(
+ &self.session.streams(),
+ &StreamingQueryListenerEvent {
event_type: QUERY_STARTED_EVENT,
event_json,
- });
+ },
+ )?;
}
Ok(StreamingQuery {
@@ -910,6 +941,39 @@ pub struct StreamingQueryListenerEvent {
pub trait StreamingQueryListener: Send + Sync {
fn on_event(&self, event: &StreamingQueryListenerEvent);
+ /// Optionally handle an asynchronous bus event for the whole listener
snapshot, for
+ /// adapters that deserialize a shared event object once before invoking
any callback.
+ /// Synchronous started events instead use
[`DataStreamWriter::start_with_event_handler`] or
+ /// [`DataStreamWriter::to_table_with_event_handler`] for shared
deserialization.
+ /// The first adapter returning `Some` owns dispatch for every entry,
including native
+ /// listeners and duplicates. Returning `None` must not invoke callbacks;
the bus then
+ /// tries the next adapter or falls back to each listener's
+ /// [`try_on_event`](Self::try_on_event).
+ fn try_post_to_all(
+ &self,
+ _listeners: &[Arc<dyn StreamingQueryListener>],
+ _event: &StreamingQueryListenerEvent,
+ ) -> Option<Result<()>> {
+ None
+ }
+
+ /// Deserialize and dispatch an asynchronous event. An error is any
failure the adapter
+ /// does not isolate, including protocol/deserialization errors and
adapter-defined
+ /// unisolated callback failures. It terminates this subscription and
clears its registrations.
+ fn try_on_event(&self, event: &StreamingQueryListenerEvent) -> Result<()> {
+ self.on_event(event);
+ Ok(())
+ }
+
+ /// Report a terminal subscription failure from transport, deserialization
or dispatch.
+ /// Return true when reported, so the bus emits only one notification even
if a listener
+ /// is registered more than once. The bus invokes this without its state
lock, before
+ /// clearing the failed generation. Like an event callback, a concurrent
removal
+ /// of a non-last listener from another thread waits for this hook to
return.
+ fn on_stream_error(&self, _error: &SparkError) -> bool {
+ false
+ }
+
/// This listener as `Any`, for a client that wraps its own listeners to
find them among
/// [`StreamingQueryManager::listeners`]. `None` by default.
fn as_any(&self) -> Option<&dyn std::any::Any> {
@@ -934,13 +998,13 @@ struct ListenerBusState {
phase: ListenerBusPhase,
worker_id: Option<std::thread::ThreadId>,
generation: u64,
- /// The generation whose worker is delivering an event to its snapshot of
the listeners.
- /// PySpark's bus holds its lock while it delivers an event, so a removal
from another
- /// thread waits for it.
+ /// The generation whose worker is delivering an event, or reporting a
terminal
+ /// failure, to its snapshot of the listeners. PySpark's bus holds its
lock during
+ /// delivery, so a removal from another thread waits for it.
dispatching: Option<u64>,
}
-/// The delivery of one event by the worker of `generation`, until it is
dropped.
+/// An event delivery or terminal failure notification by the worker of
`generation`.
struct Dispatch<'a> {
bus: &'a Weak<ListenerBus>,
generation: u64,
@@ -995,14 +1059,13 @@ impl ListenerBus {
/// The worker retains neither the session nor its bus. The stream owns only
transport
/// and progress-handler state; dropping the last session/manager cancels the
reader.
fn run_listener_event_loop(
- mut stream: ListenerEventStream,
+ stream: impl IntoIterator<Item = Result<(i32, String)>>,
bus: Weak<ListenerBus>,
generation: u64,
) {
+ let mut stream = stream.into_iter();
+ let mut failure = None;
for item in &mut stream {
- let Ok((event_type, event_json)) = item else {
- break;
- };
let Some(owner) = bus.upgrade() else {
break;
};
@@ -1023,12 +1086,39 @@ fn run_listener_event_loop(
bus: &bus,
generation,
};
+ let (event_type, event_json) = match item {
+ Ok(event) => event,
+ Err(error) => {
+ failure = Some((listeners, error, dispatch));
+ break;
+ }
+ };
let event = StreamingQueryListenerEvent {
event_type,
event_json,
};
+ let dispatched = listeners
+ .iter()
+ .find_map(|listener| listener.try_post_to_all(&listeners, &event))
+ .unwrap_or_else(|| {
+ listeners
+ .iter()
+ .try_for_each(|listener| listener.try_on_event(&event))
+ });
+ if let Err(error) = dispatched {
+ failure = Some((listeners, error, dispatch));
+ break;
+ }
+ drop(dispatch);
+ }
+ if let Some((listeners, error, dispatch)) = failure {
+ // A removal from another thread waits for this terminal notification,
just
+ // as it waits for an ordinary event. No state lock or strong bus
owner is
+ // held across either callback path; the worker can still remove
itself.
for listener in listeners {
- listener.on_event(&event);
+ if listener.on_stream_error(&error) {
+ break;
+ }
}
drop(dispatch);
}
@@ -1176,10 +1266,10 @@ impl StreamingQueryManager {
};
if state.listeners.len() > 1 {
state.listeners.remove(index);
- // As PySpark's bus, whose delivery of an event holds the lock
of its removal:
- // the removal returns once the event in delivery has been
delivered, also to
- // the removed listener, which gets no events after it. A
callback of the bus
- // does not wait, as it would wait for its own delivery.
+ // As in PySpark's bus, a removal waits for delivery to the
snapshot,
+ // including the removed listener, to finish. This also covers
a
+ // terminal error notification. A callback of the bus does not
wait,
+ // as it would wait for its own delivery.
if state.worker_id != Some(std::thread::current().id()) {
let generation = state.generation;
while state.dispatching == Some(generation) {
@@ -1681,6 +1771,385 @@ mod tests {
.expect("only terminal close eventually cancels the current
worker");
}
+ #[test]
+ fn
listener_bus_reports_transport_and_decode_failures_once_before_clearing() {
+ use std::sync::atomic::{AtomicUsize, Ordering};
+
+ struct Listener {
+ bus: Weak<ListenerBus>,
+ events: AtomicUsize,
+ errors: Mutex<Vec<String>>,
+ }
+ impl StreamingQueryListener for Listener {
+ fn on_event(&self, _: &StreamingQueryListenerEvent) {
+ self.events.fetch_add(1, Ordering::SeqCst);
+ }
+
+ fn try_on_event(&self, event: &StreamingQueryListenerEvent) ->
Result<()> {
+ if event.event_json == "invalid" {
+ return Err(SparkError::connect_msg("event decode failed"));
+ }
+ self.on_event(event);
+ Ok(())
+ }
+
+ fn on_stream_error(&self, error: &SparkError) -> bool {
+ // Reporting runs before cleanup, without holding the bus lock.
+ let bus = self.bus.upgrade().unwrap();
+ let state = bus.state.lock().unwrap();
+ assert_eq!(state.listeners.len(), 2);
+ assert_eq!(state.dispatching, Some(0));
+ drop(state);
+ self.errors.lock().unwrap().push(error.message());
+ true
+ }
+ }
+
+ for decode_failure in [false, true] {
+ let bus = Arc::new(ListenerBus::default());
+ let listener = Arc::new(Listener {
+ bus: Arc::downgrade(&bus),
+ events: AtomicUsize::new(0),
+ errors: Mutex::new(vec![]),
+ });
+ {
+ let mut state = bus.state.lock().unwrap();
+ state
+ .listeners
+ .push(("first".to_string(), listener.clone()));
+ state
+ .listeners
+ .push(("second".to_string(), listener.clone()));
+ }
+ let failure = if decode_failure {
+ Ok((QUERY_IDLE_EVENT, "invalid".to_string()))
+ } else {
+ Err(SparkError::connect_msg("transport failed"))
+ };
+ run_listener_event_loop(
+ vec![failure, Ok((QUERY_IDLE_EVENT, "valid".to_string()))],
+ Arc::downgrade(&bus),
+ 0,
+ );
+ assert_eq!(listener.events.load(Ordering::SeqCst), 0);
+ assert_eq!(
+ *listener.errors.lock().unwrap(),
+ vec![if decode_failure {
+ "event decode failed"
+ } else {
+ "transport failed"
+ }],
+ );
+ let state = bus.state.lock().unwrap();
+ assert!(state.listeners.is_empty());
+ assert!(state.dispatching.is_none());
+ }
+ }
+
+ #[test]
+ fn
nonlast_removal_waits_for_transport_decode_and_batch_error_notifications() {
+ use std::sync::atomic::{AtomicUsize, Ordering};
+ use std::sync::mpsc;
+ use std::time::{Duration, Instant};
+
+ struct ReleaseOnDrop(mpsc::Sender<()>);
+ impl Drop for ReleaseOnDrop {
+ fn drop(&mut self) {
+ let _ = self.0.send(());
+ }
+ }
+
+ struct Listener {
+ batch_failure: bool,
+ entered: mpsc::Sender<()>,
+ release: Mutex<mpsc::Receiver<()>>,
+ order: Arc<Mutex<Vec<&'static str>>>,
+ events: AtomicUsize,
+ errors: AtomicUsize,
+ }
+ impl StreamingQueryListener for Listener {
+ fn on_event(&self, _: &StreamingQueryListenerEvent) {
+ self.events.fetch_add(1, Ordering::SeqCst);
+ }
+
+ fn try_post_to_all(
+ &self,
+ _: &[Arc<dyn StreamingQueryListener>],
+ _: &StreamingQueryListenerEvent,
+ ) -> Option<Result<()>> {
+ self.batch_failure
+ .then(|| Err(SparkError::connect_msg("batch decode
failed")))
+ }
+
+ fn try_on_event(&self, _: &StreamingQueryListenerEvent) ->
Result<()> {
+ Err(SparkError::connect_msg("event decode failed"))
+ }
+
+ fn on_stream_error(&self, _: &SparkError) -> bool {
+ self.errors.fetch_add(1, Ordering::SeqCst);
+ self.order.lock().unwrap().push("reporting");
+ self.entered.send(()).unwrap();
+ self.release
+ .lock()
+ .unwrap()
+ .recv_timeout(Duration::from_secs(15))
+ .expect("the test releases the terminal notification");
+ self.order.lock().unwrap().push("reported");
+ true
+ }
+ }
+
+ for failure_kind in ["transport", "decode", "batch"] {
+ let spark = session();
+ let manager = spark.streams();
+ let (entered, reporting) = mpsc::channel();
+ let (release, released) = mpsc::channel();
+ let order = Arc::new(Mutex::new(vec![]));
+ let listener = Arc::new(Listener {
+ batch_failure: failure_kind == "batch",
+ entered,
+ release: Mutex::new(released),
+ order: Arc::clone(&order),
+ events: AtomicUsize::new(0),
+ errors: AtomicUsize::new(0),
+ });
+ {
+ let mut state = manager.bus.state.lock().unwrap();
+ state
+ .listeners
+ .push(("first".to_string(), listener.clone()));
+ state
+ .listeners
+ .push(("second".to_string(), listener.clone()));
+ }
+ let failure = if failure_kind == "transport" {
+ Err(SparkError::connect_msg("transport failed"))
+ } else {
+ Ok((QUERY_IDLE_EVENT, "event".to_string()))
+ };
+ let bus = Arc::downgrade(&manager.bus);
+ let (start, started) = mpsc::channel();
+ let (removed, removal) = mpsc::channel();
+ std::thread::scope(|scope| {
+ // Release the hook before scoped threads are joined even if a
+ // main-thread assertion fails while it is blocked.
+ let _release_on_drop = ReleaseOnDrop(release.clone());
+ let worker = scope.spawn(move || {
+ started.recv_timeout(Duration::from_secs(5)).unwrap();
+ run_listener_event_loop(vec![failure], bus, 0);
+ });
+ manager.bus.state.lock().unwrap().worker_id =
Some(worker.thread().id());
+ start.send(()).unwrap();
+ reporting.recv_timeout(Duration::from_secs(5)).unwrap();
+ let removal_order = Arc::clone(&order);
+ let manager = &manager;
+ let remover = scope.spawn(move || {
+ assert!(manager.remove_listener_found("second").unwrap());
+ removal_order.lock().unwrap().push("removed");
+ removed.send(()).unwrap();
+ });
+ let deadline = Instant::now() + Duration::from_secs(5);
+ loop {
+ if manager.bus.state.lock().unwrap().listeners.len() == 1 {
+ break;
+ }
+ assert!(Instant::now() < deadline, "remover did not reach
the bus");
+ std::thread::yield_now();
+ }
+ assert!(matches!(
+ removal.recv_timeout(Duration::from_millis(100)),
+ Err(mpsc::RecvTimeoutError::Timeout)
+ ));
+ release.send(()).unwrap();
+ let removal_result =
removal.recv_timeout(Duration::from_secs(5));
+ if removal_result.is_err() {
+ // Keep a broken guard-release regression from stranding
the
+ // scoped remover while the assertion reports its failure.
+ let mut state = manager.bus.state.lock().unwrap();
+ if state.dispatching == Some(0) {
+ state.dispatching = None;
+ manager.bus.changed.notify_all();
+ }
+ }
+ worker.join().unwrap();
+ remover.join().unwrap();
+ assert!(
+ removal_result.is_ok(),
+ "removal did not finish after reporting"
+ );
+ });
+ assert_eq!(*order.lock().unwrap(), ["reporting", "reported",
"removed"]);
+ assert_eq!(listener.events.load(Ordering::SeqCst), 0);
+ assert_eq!(listener.errors.load(Ordering::SeqCst), 1);
+ let state = manager.bus.state.lock().unwrap();
+ assert!(state.listeners.is_empty());
+ assert!(state.dispatching.is_none());
+ }
+ }
+
+ #[test]
+ fn a_terminal_error_notification_does_not_keep_the_listener_bus_alive() {
+ use std::sync::mpsc;
+ use std::time::Duration;
+
+ struct Listener {
+ entered: mpsc::Sender<()>,
+ release: Mutex<mpsc::Receiver<()>>,
+ }
+ impl StreamingQueryListener for Listener {
+ fn on_event(&self, _: &StreamingQueryListenerEvent) {}
+
+ fn on_stream_error(&self, _: &SparkError) -> bool {
+ self.entered.send(()).unwrap();
+ self.release
+ .lock()
+ .unwrap()
+ .recv_timeout(Duration::from_secs(5))
+ .expect("the test releases the terminal notification");
+ true
+ }
+ }
+
+ let bus = Arc::new(ListenerBus::default());
+ let weak = Arc::downgrade(&bus);
+ let (entered, reporting) = mpsc::channel();
+ let (release, released) = mpsc::channel();
+ bus.state.lock().unwrap().listeners.push((
+ "listener".to_string(),
+ Arc::new(Listener {
+ entered,
+ release: Mutex::new(released),
+ }),
+ ));
+ let worker_bus = weak.clone();
+ let worker = std::thread::spawn(move || {
+ run_listener_event_loop(
+ vec![Err(SparkError::connect_msg("transport failed"))],
+ worker_bus,
+ 0,
+ );
+ });
+ reporting.recv_timeout(Duration::from_secs(5)).unwrap();
+ drop(bus);
+ assert!(weak.upgrade().is_none());
+ release.send(()).unwrap();
+ worker.join().unwrap();
+ }
+
+ #[test]
+ fn a_stale_worker_does_not_report_errors_to_a_new_generation() {
+ use std::sync::atomic::{AtomicUsize, Ordering};
+
+ struct Listener(AtomicUsize);
+ impl StreamingQueryListener for Listener {
+ fn on_event(&self, _: &StreamingQueryListenerEvent) {
+ panic!("a stale worker must not dispatch events");
+ }
+
+ fn on_stream_error(&self, _: &SparkError) -> bool {
+ self.0.fetch_add(1, Ordering::SeqCst);
+ true
+ }
+ }
+ let bus = Arc::new(ListenerBus::default());
+ let listener = Arc::new(Listener(AtomicUsize::new(0)));
+ {
+ let mut state = bus.state.lock().unwrap();
+ state.generation = 2;
+ state
+ .listeners
+ .push(("current".to_string(), listener.clone()));
+ }
+ run_listener_event_loop(
+ vec![Err(SparkError::connect_msg("old transport failed"))],
+ Arc::downgrade(&bus),
+ 1,
+ );
+ assert_eq!(listener.0.load(Ordering::SeqCst), 0);
+ assert_eq!(bus.state.lock().unwrap().listeners.len(), 1);
+ }
+
+ #[test]
+ fn listener_bus_batch_hook_runs_once_per_event_before_any_callback() {
+ use std::sync::atomic::{AtomicUsize, Ordering};
+
+ struct NativeListener(AtomicUsize);
+ impl StreamingQueryListener for NativeListener {
+ fn on_event(&self, _: &StreamingQueryListenerEvent) {
+ self.0.fetch_add(1, Ordering::SeqCst);
+ }
+ }
+ struct BatchListener {
+ fail: bool,
+ prepared: AtomicUsize,
+ callbacks: AtomicUsize,
+ errors: AtomicUsize,
+ }
+ impl StreamingQueryListener for BatchListener {
+ fn on_event(&self, _: &StreamingQueryListenerEvent) {
+ self.callbacks.fetch_add(1, Ordering::SeqCst);
+ }
+
+ fn try_post_to_all(
+ &self,
+ listeners: &[Arc<dyn StreamingQueryListener>],
+ event: &StreamingQueryListenerEvent,
+ ) -> Option<Result<()>> {
+ self.prepared.fetch_add(1, Ordering::SeqCst);
+ if self.fail {
+ return Some(Err(SparkError::connect_msg("batch decode
failed")));
+ }
+ for listener in listeners {
+ listener.on_event(event);
+ }
+ Some(Ok(()))
+ }
+
+ fn on_stream_error(&self, _: &SparkError) -> bool {
+ self.errors.fetch_add(1, Ordering::SeqCst);
+ true
+ }
+ }
+
+ for fail in [false, true] {
+ let bus = Arc::new(ListenerBus::default());
+ let native = Arc::new(NativeListener(AtomicUsize::new(0)));
+ let batch = Arc::new(BatchListener {
+ fail,
+ prepared: AtomicUsize::new(0),
+ callbacks: AtomicUsize::new(0),
+ errors: AtomicUsize::new(0),
+ });
+ {
+ let mut state = bus.state.lock().unwrap();
+ // A native listener precedes duplicate adapter registrations.
Batch
+ // preparation must run before even that first native callback.
+ state.listeners.push(("native".to_string(), native.clone()));
+ state.listeners.push(("batch-1".to_string(), batch.clone()));
+ state.listeners.push(("batch-2".to_string(), batch.clone()));
+ }
+ run_listener_event_loop(
+ vec![
+ Ok((QUERY_IDLE_EVENT, "first".to_string())),
+ Ok((QUERY_IDLE_EVENT, "second".to_string())),
+ ],
+ Arc::downgrade(&bus),
+ 0,
+ );
+ assert_eq!(
+ batch.prepared.load(Ordering::SeqCst),
+ if fail { 1 } else { 2 }
+ );
+ assert_eq!(native.0.load(Ordering::SeqCst), if fail { 0 } else { 2
});
+ assert_eq!(
+ batch.callbacks.load(Ordering::SeqCst),
+ if fail { 0 } else { 4 }
+ );
+ assert_eq!(batch.errors.load(Ordering::SeqCst), usize::from(fail));
+ assert!(bus.state.lock().unwrap().listeners.is_empty());
+ }
+ }
+
#[test]
fn stream_reader_format_option() {
let spark = session();
diff --git a/docs/streaming.md b/docs/streaming.md
index f05c7d49..a144537c 100644
--- a/docs/streaming.md
+++ b/docs/streaming.md
@@ -113,3 +113,32 @@ for q in spark.streams().active()? {
!!! warning
Checkpoints are mandatory for fault tolerance. Always set
`checkpointLocation` for production queries.
+
+### Listener failures
+
+For native Rust listeners, a terminal stream end or error clears the
subscription's
+registrations. The default `StreamingQueryListener::on_stream_error` hook is
silent;
+override it to report failures, and re-add listeners to open a new
subscription. A
+normal stream end clears registrations without calling the error hook.
Adapters can
+return errors through `try_on_event` or provide shared asynchronous dispatch
through
+`try_post_to_all`. Use `DataStreamWriter::start_with_event_handler` or
+`to_table_with_event_handler` for shared synchronous start-event decoding.
+
+`SparkSession::stop()` and `close()` cancel listener dispatch and clear
registrations
+locally, without sending a listener-removal RPC or joining callbacks already
running.
+Reference PySpark leaves listener-bus cleanup to its handler thread during
session stop.
+
+The PySpark drop-in listener bus deserializes each event once before invoking
callbacks and
+shares that event object across all listener registrations. A malformed start
event
+raises from `start()` or `toTable()`, even when no listeners are registered. A
+transport failure, malformed asynchronous event or unknown event type warns and
+clears the subscription's listeners. Add the listeners again to open a
subscription.
+A server-side end of the event stream also clears registrations without a
warning;
+reference PySpark retains them after a normal stream end.
+
+Callback exceptions derived from `Exception` produce per-listener warnings and
+dispatch continues. An uncaught `BaseException`, or a callback warning
promoted to
+an error by Python's warning filters, terminates the native subscription and
clears
+its registrations. This cleanup is deliberate: reference PySpark can retain
stale
+registrations when its handler exits through `BaseException`. The terminal
handler
+warning is best effort when warning filters themselves raise errors.
diff --git a/python/pyspark/sql/streaming/query.py
b/python/pyspark/sql/streaming/query.py
index c0ae0ae9..4ba55391 100644
--- a/python/pyspark/sql/streaming/query.py
+++ b/python/pyspark/sql/streaming/query.py
@@ -41,6 +41,8 @@ from pyspark._pyspark import (
StreamingQuery as _StreamingQuery,
StreamingQueryManager as _StreamingQueryManager,
)
+from pyspark.errors import PySparkValueError
+
# As PySpark's, the exception of a failed query is that of pyspark.errors.
from pyspark.errors import StreamingQueryException # noqa: F401
@@ -60,24 +62,40 @@ _QUERY_IDLE_EVENT = 3
StreamingQuery = _StreamingQuery
-def _dispatch_listener_event(listener, event_type, event_json):
- """Deserialize a listener event and invoke the matching callback on
``listener``.
+def _deserialize_listener_event(event_type, event_json, *, started=False):
+ """Decode a protocol event before entering callback exception isolation.
- Called from the native (Rust) listener bus for each event. Mirrors the
reference
- ``StreamingQueryListenerBus.post_to_all``: build the typed event and route
it to
- onQueryStarted / onQueryProgress / onQueryIdle / onQueryTerminated.
Callback exceptions
- are surfaced as warnings, as PySpark's, so one listener cannot break the
dispatch thread.
+ Started events belong to the synchronous start response, not the
asynchronous
+ protocol enum. Malformed JSON and missing fields propagate to the caller,
as in
+ reference PySpark's writer and listener-bus deserializers.
"""
+ if started and event_type == _QUERY_STARTED_EVENT:
+ return QueryStartedEvent.fromJson(json.loads(event_json))
+ if event_type == _QUERY_PROGRESS_EVENT:
+ return QueryProgressEvent.fromJson(json.loads(event_json))
+ if event_type == _QUERY_TERMINATED_EVENT:
+ return QueryTerminatedEvent.fromJson(json.loads(event_json))
+ if event_type == _QUERY_IDLE_EVENT:
+ return QueryIdleEvent.fromJson(json.loads(event_json))
+ raise PySparkValueError(
+ errorClass="UNKNOWN_VALUE_FOR",
+ messageParameters={"var": f"proto.StreamingQueryEventType:
{event_type}"},
+ )
+
+
+def _dispatch_listener_event(listener, event):
+ """Warn on callback Exception; BaseException and warning-as-error escape
to the bus."""
try:
- j = json.loads(event_json)
- if event_type == _QUERY_STARTED_EVENT:
- listener.onQueryStarted(QueryStartedEvent.fromJson(j))
- elif event_type == _QUERY_PROGRESS_EVENT:
- listener.onQueryProgress(QueryProgressEvent.fromJson(j))
- elif event_type == _QUERY_IDLE_EVENT:
- listener.onQueryIdle(QueryIdleEvent.fromJson(j))
- elif event_type == _QUERY_TERMINATED_EVENT:
- listener.onQueryTerminated(QueryTerminatedEvent.fromJson(j))
+ if isinstance(event, QueryStartedEvent):
+ listener.onQueryStarted(event)
+ elif isinstance(event, QueryProgressEvent):
+ listener.onQueryProgress(event)
+ elif isinstance(event, QueryIdleEvent):
+ listener.onQueryIdle(event)
+ elif isinstance(event, QueryTerminatedEvent):
+ listener.onQueryTerminated(event)
+ else:
+ warnings.warn(f"Unknown StreamingQueryListener event: {event}")
except Exception as e: # noqa: BLE001
warnings.warn(f"Listener {str(listener)} threw an exception\n{e}")
diff --git a/python/tests/_mock_connect_server.py
b/python/tests/_mock_connect_server.py
index a2d27402..3e09303e 100644
--- a/python/tests/_mock_connect_server.py
+++ b/python/tests/_mock_connect_server.py
@@ -793,7 +793,11 @@ class RemoteService:
self.connection.recv()
def listener_event(self, session_id, event_type, event_json):
- self.connection.send(("listener_event", session_id, event_type,
json.dumps(event_json)))
+ self.listener_event_json(session_id, event_type,
json.dumps(event_json))
+
+ def listener_event_json(self, session_id, event_type, event_json):
+ """Send raw event JSON, including malformed payloads for protocol
error tests."""
+ self.connection.send(("listener_event", session_id, event_type,
event_json))
assert self.connection.poll(10), "Mock server did not respond"
self.connection.recv()
diff --git a/python/tests/streaming_listener_cases.py
b/python/tests/streaming_listener_cases.py
index 88cf3d47..0daf541a 100644
--- a/python/tests/streaming_listener_cases.py
+++ b/python/tests/streaming_listener_cases.py
@@ -16,6 +16,7 @@
#
"""Streaming listener lifecycle tests against the separately hosted mock
service."""
+import json
import subprocess
import sys
import tempfile
@@ -26,7 +27,7 @@ from pathlib import Path
import pytest
-from pyspark.sql.streaming.listener import StreamingQueryListener
+from pyspark.sql.streaming.listener import QueryIdleEvent,
StreamingQueryListener
class IdleListener(StreamingQueryListener):
@@ -73,18 +74,304 @@ def run_listener_case(*arguments):
)
except subprocess.TimeoutExpired:
output.seek(0)
- pytest.fail("Concurrent listener shutdown timed out:\n" +
output.read())
+ pytest.fail(f"Listener case {arguments} timed out:\n" +
output.read())
output.seek(0)
assert result.returncode == 0, output.read()
-def listener_warnings(caught):
- prefixes = ("Failed to add the listener", "Failed to remove the listener")
+_LIFECYCLE_WARNING_PREFIXES = ("Failed to add the listener", "Failed to remove
the listener")
+_HANDLER_WARNING_PREFIX = (
+ "StreamingQueryListenerBus Handler thread received exception, all client
side "
+ "listeners are removed and handler thread is terminated. The error is: "
+)
+
+
+def warnings_with_prefixes(caught, prefixes):
return [str(item.message) for item in caught
if issubclass(item.category, UserWarning) and
str(item.message).startswith(prefixes)]
+def listener_warnings(caught):
+ return warnings_with_prefixes(caught, _LIFECYCLE_WARNING_PREFIXES)
+
+
+def listener_event_warnings(caught):
+ return warnings_with_prefixes(
+ caught, (_HANDLER_WARNING_PREFIX, "Listener ",
*_LIFECYCLE_WARNING_PREFIXES)
+ )
+
+
+def emit_unrelated_warnings():
+ warnings.warn("unrelated streaming test deprecation", DeprecationWarning)
+ warnings.warn("unrelated streaming test user warning", UserWarning)
+
+
+def wait_for_empty_bus(manager, message="failed stream kept its
registrations"):
+ deadline = time.monotonic() + 5
+ while manager._sqlb._listener_bus:
+ assert time.monotonic() < deadline, message
+ time.sleep(0.01)
+
+
+def start_response(event_json):
+ from _mock_connect_server import message_field
+
+ # WriteStreamOperationStartResult: query_id=1, query_started_event_json=3.
+ return message_field(
+ 8,
+ message_field(1, message_field(1, "q") + message_field(2, "r"))
+ + message_field(3, event_json),
+ )
+
+
class TestStreamingListenerLifecycle:
+ @pytest.mark.parametrize("method", ["start", "toTable"])
+ @pytest.mark.parametrize("registered", [False, True])
+ @pytest.mark.parametrize("event_json, exception", [
+ ("", json.JSONDecodeError), ("{", json.JSONDecodeError), ("{}",
KeyError),
+ ], ids=["empty-json", "invalid-json", "missing-fields"])
+ def test_streaming_started_event_decode_error_propagates(
+ self, mock_connect, method, registered, event_json, exception,
+ ):
+ spark, service = mock_connect
+ service.listener_bus_enabled = True
+ listener = IdleListener()
+ try:
+ if registered:
+ spark.streams.addListener(listener)
+ service.execute_payload = start_response(event_json)
+ writer = spark.readStream.format("rate").load().writeStream
+ with warnings.catch_warnings(record=True) as caught:
+ warnings.simplefilter("always")
+ emit_unrelated_warnings()
+ with pytest.raises(exception):
+ getattr(writer, method)(*(["table"] if method == "toTable"
else []))
+ assert not listener_event_warnings(caught)
+ assert spark.streams._sqlb._listener_bus == ([listener] if
registered else [])
+ finally:
+ spark.streams.close()
+
+ @pytest.mark.parametrize("method", ["start", "toTable"])
+ def
test_streaming_started_callbacks_share_typed_event_and_isolate_exceptions(
+ self, mock_connect, method,
+ ):
+ spark, service = mock_connect
+ service.listener_bus_enabled = True
+ events = []
+
+ class StartedListener(IdleListener):
+ def onQueryStarted(self, event):
+ events.append(event)
+ if len(events) == 1:
+ raise ValueError("started callback failed")
+
+ listener = StartedListener()
+ try:
+ # Duplicate registrations receive the same deserialized event,
synchronously.
+ spark.streams.addListener(listener)
+ spark.streams.addListener(listener)
+ service.execute_payload = start_response(json.dumps({
+ "id": "00000000-0000-4000-8000-000000000001",
+ "runId": "00000000-0000-4000-8000-000000000002",
+ "name": "query", "timestamp": "started", "jobTags": ["tag"],
+ }))
+ writer = spark.readStream.format("rate").load().writeStream
+ with warnings.catch_warnings(record=True) as caught:
+ warnings.simplefilter("always")
+ emit_unrelated_warnings()
+ query = getattr(writer, method)(*(["table"] if method ==
"toTable" else []))
+ assert query.id == "q"
+ relevant = listener_event_warnings(caught)
+ assert len(relevant) == 1 and "started callback failed" in
relevant[0]
+ assert len(events) == 2 and events[0] is events[1]
+ assert events[0].timestamp == "started" and events[0].jobTags ==
{"tag"}
+ assert spark.streams._sqlb._listener_bus == [listener, listener]
+ finally:
+ spark.streams.close()
+
+ @pytest.mark.parametrize("event_type, event_json", [
+ (None, ""),
+ (3, ""),
+ (3, "{"),
+ (3, "{}"),
+ (99, "{}"),
+ (0, "{}"),
+ (-1, "{}"),
+ ], ids=["transport", "empty-json", "invalid-json", "missing-fields",
"unknown-type",
+ "unspecified-type", "async-started-type"])
+ def test_streaming_bus_errors_warn_once_clear_all_and_allow_reregistration(
+ self, mock_connect, event_type, event_json,
+ ):
+ spark, service = mock_connect
+ service.listener_bus_enabled = True
+ manager = spark.streams
+ first, second = IdleListener(), IdleListener()
+ try:
+ manager.addListener(first)
+ manager.addListener(first)
+ manager.addListener(second)
+ emit_idle(service, spark, "before-error")
+ first.wait_for("before-error")
+ second.wait_for("before-error")
+ if event_type is None:
+ from pyspark.errors.exceptions.connect import
SparkConnectGrpcException
+
+ # Compare the entire error rendering with a normal RPC abort
carrying
+ # the same status and details as the listener-stream abort.
+ previous_failure = service.execute_failure
+ service.execute_failure = "listener event stream failed"
+ try:
+ with pytest.raises(SparkConnectGrpcException) as error:
+ spark.range(1).collect()
+ expected_error = str(error.value)
+ finally:
+ service.execute_failure = previous_failure
+ elif event_type == 3:
+ # Derive the complete reference string from the vendored event
decoder,
+ # independently of the native bus and its exception round trip.
+ with pytest.raises((json.JSONDecodeError, KeyError)) as error:
+ QueryIdleEvent.fromJson(json.loads(event_json))
+ expected_error = str(error.value)
+ else:
+ from pyspark.errors import PySparkValueError
+
+ expected_error = str(PySparkValueError(
+ errorClass="UNKNOWN_VALUE_FOR",
+ messageParameters={"var": f"proto.StreamingQueryEventType:
{event_type}"},
+ ))
+ with warnings.catch_warnings(record=True) as caught:
+ warnings.simplefilter("always")
+ emit_unrelated_warnings()
+ if event_type is None:
+ assert service.end_listener_stream(spark.session_id,
failure=True)
+ else:
+ service.listener_event_json(spark.session_id, event_type,
event_json)
+ wait_for_empty_bus(manager)
+ messages = listener_event_warnings(caught)
+ assert len(messages) == 1
+ assert messages[0] == _HANDLER_WARNING_PREFIX + expected_error
+ assert first.timestamps == ["before-error", "before-error"]
+ assert second.timestamps == ["before-error"]
+ assert service.wait_listener_streams_finished(1)
+ assert service.listener_removals == 0
+
+ manager.addListener(second)
+ assert service.listener_subscriptions == 2
+ emit_idle(service, spark, "after-error")
+ second.wait_for("after-error")
+ assert manager._sqlb._listener_bus == [second]
+ assert first.timestamps == ["before-error", "before-error"]
+ manager.removeListener(second)
+ assert service.wait_listener_streams_finished(2)
+ finally:
+ manager.close()
+
+ def
test_streaming_callback_exceptions_warn_and_leave_the_bus_running(self,
mock_connect):
+ spark, service = mock_connect
+ service.listener_bus_enabled = True
+
+ class FailingListener(IdleListener):
+ def onQueryIdle(self, event):
+ super().onQueryIdle(event)
+ raise ValueError("idle callback failed")
+
+ first, second = FailingListener(), IdleListener()
+ try:
+ spark.streams.addListener(first)
+ spark.streams.addListener(second)
+ with warnings.catch_warnings(record=True) as caught:
+ warnings.simplefilter("always")
+ emit_unrelated_warnings()
+ for timestamp in ("first-callback", "second-callback"):
+ emit_idle(service, spark, timestamp)
+ second.wait_for(timestamp)
+ relevant = listener_event_warnings(caught)
+ assert len(relevant) == 2
+ assert relevant == [f"Listener {first} threw an exception\nidle
callback failed"] * 2
+ assert first.timestamps == second.timestamps == ["first-callback",
"second-callback"]
+ assert spark.streams._sqlb._listener_bus == [first, second]
+ assert service.listener_subscriptions == 1
+ finally:
+ spark.streams.close()
+
+ def test_streaming_duplicate_async_listeners_share_one_decoded_event(
+ self, mock_connect, monkeypatch,
+ ):
+ from pyspark.sql.streaming import query as query_module
+
+ spark, service = mock_connect
+ service.listener_bus_enabled = True
+ decoded, received = [], []
+ deserialize = query_module._deserialize_listener_event
+
+ def count_decode(*args, **kwargs):
+ event = deserialize(*args, **kwargs)
+ decoded.append(event)
+ return event
+
+ monkeypatch.setattr(query_module, "_deserialize_listener_event",
count_decode)
+
+ class SharedListener(IdleListener):
+ def onQueryIdle(self, event):
+ with self.condition:
+ received.append(event)
+ self.condition.notify_all()
+
+ listener = SharedListener()
+ try:
+ spark.streams.addListener(listener)
+ spark.streams.addListener(listener)
+ with warnings.catch_warnings(record=True) as caught:
+ warnings.simplefilter("always")
+ emit_unrelated_warnings()
+ emit_idle(service, spark, "shared-event")
+ with listener.condition:
+ assert listener.condition.wait_for(lambda: len(received)
== 2, timeout=5)
+ assert not listener_event_warnings(caught)
+ assert len(decoded) == 1
+ assert received[0] is received[1] is decoded[0]
+ assert received[0].timestamp == "shared-event"
+ finally:
+ spark.streams.close()
+
+ @pytest.mark.parametrize("failure", ["base-exception", "warning-as-error"])
+ def test_streaming_unisolated_callback_errors_clear_the_bus(self,
mock_connect, failure):
+ spark, service = mock_connect
+ service.listener_bus_enabled = True
+
+ class CallbackExit(BaseException):
+ pass
+
+ class FailingListener(IdleListener):
+ def onQueryIdle(self, event):
+ super().onQueryIdle(event)
+ if failure == "base-exception":
+ raise CallbackExit("uncaught callback exit")
+ raise ValueError("callback warning becomes an error")
+
+ first, second = FailingListener(), IdleListener()
+ try:
+ spark.streams.addListener(first)
+ spark.streams.addListener(second)
+ with warnings.catch_warnings(record=True) as caught:
+ warnings.simplefilter("always")
+ warnings.filterwarnings("error", message="Listener ",
category=UserWarning)
+ emit_unrelated_warnings()
+ emit_idle(service, spark, "unisolated-error")
+ wait_for_empty_bus(spark.streams)
+ relevant = listener_event_warnings(caught)
+ assert len(relevant) == 1
+ expected_error = (
+ "uncaught callback exit" if failure == "base-exception"
+ else f"Listener {first} threw an exception\ncallback warning
becomes an error"
+ )
+ assert relevant[0] == _HANDLER_WARNING_PREFIX + expected_error
+ assert first.timestamps == ["unisolated-error"]
+ assert second.timestamps == []
+ assert service.wait_listener_streams_finished(1)
+ finally:
+ spark.streams.close()
+
@pytest.mark.parametrize("remove_failure, callback_action", [
(False, "remove"), (True, "remove"), (False, "stop"), (True, "stop"),
(False, "add"),
(False, "remove_duplicate"), (True, "remove_duplicate"),
@@ -725,10 +1012,7 @@ def ended_stream_case(termination):
# or with a nonretryable server error. Server completion precedes
native cleanup.
assert service.end_listener_stream(spark.session_id,
failure=termination == "error")
assert service.wait_listener_streams_finished(1)
- deadline = time.monotonic() + 5
- while manager._sqlb._listener_bus:
- assert time.monotonic() < deadline, "ended stream kept its
registrations"
- time.sleep(0.01)
+ wait_for_empty_bus(manager, "ended stream kept its registrations")
assert listener._rust_listener_ids == stale_ids
manager.removeListener(listener)
assert listener._rust_listener_ids == stale_ids
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]