This is an automated email from the ASF dual-hosted git repository.

HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git


The following commit(s) were added to refs/heads/master by this push:
     new 93d2d07a [SPARK-61215][CONNECT][RUST] Match streaming listener parsing 
and failure handling
93d2d07a is described below

commit 93d2d07a3fde156f10d98197d0d49795640bb8c7
Author: DB Tsai <[email protected]>
AuthorDate: Mon Nov 9 15:50:35 2026 +0900

    [SPARK-61215][CONNECT][RUST] Match streaming listener parsing and failure 
handling
    
    ### What changes were proposed in this pull request?
    
    Propagate synchronous streaming query-start decoding errors from start() 
and toTable(), including when no listeners are registered. Report asynchronous 
transport, decoding and unknown-event errors once before clearing the affected 
subscription. Decode each event once and share it across the listener snapshot, 
including duplicate registrations.
    
    Native Rust listeners retain a silent default error hook and can opt into 
error reporting. Callback and warning hooks run without the listener-bus state 
lock. Concurrent removal of a non-last listener waits through its event 
callback or terminal error notification.
    
    This independent patch is based on master 
925dc876f63f1f779fe176f62575a0aa96cc6886, which already contains the listener 
lifecycle and generation-scoped removal behavior.
    
    ### Why are the changes needed?
    
    Malformed listener events can disappear or leave registrations active, and 
terminal transport failures can clear listeners without PySpark's handler 
warning. The caller should receive a malformed synchronous start event's 
original exception, and asynchronous failures should emit the corresponding 
handler warning while allowing listeners to be registered again.
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes. Start-event decoding failures now reach the caller, and asynchronous 
failures warn once and clear registrations. Individual callback Exceptions 
continue to warn per listener while dispatch proceeds.
    
    The documented limits remain:
    
    - Normal stream EOF clears registrations silently; reference PySpark 
retains them.
    - An uncaught asynchronous callback BaseException, or a callback warning 
promoted to an error, ends the subscription and clears registrations. Reference 
PySpark can retain stale registrations. The terminal handler warning is best 
effort if warning filters raise.
    - Session stop() and close() cancel and clear listeners locally, without 
listener-removal RPCs or waiting for active callbacks. The current callback 
snapshot can finish.
    - A registered listener can retain its session through listener.spark. 
Initial listener RPC establishment follows the transport deadline/retry policy. 
The inherited single-generation dispatch-marker boundary is unchanged.
    
    ### How was this patch tested?
    
    - Formatting, release core/API library tests, integration-test compilation 
with and without wasm-udf, and the extension build passed. Offline Python 
recorded **2,605 passed and 8 inherited live-server skips**, including all **53 
required listener cases**. All five new native unit tests passed. These were 
executed on the pre-comment revision and carried to the current source through 
exactly three documented comment-only replacements; they are not a fresh 
complete execution on the comme [...]
    - A fresh strict run on the current artifact passed **25 distinct Python 
cases once each**, with all **75 setup/call/teardown reports** passing and no 
required skips. Three selected native error regression unit cases also passed 
separately. The strict run attested the loaded extension, shim and test 
sources; it is not a stress-repeat result.
    - Against a live Spark server, **7 native Python cases passed / 0 skipped** 
and all three saved native streaming e2e callers passed, each **1 passed / 0 
ignored**. The live Python selection deselected 194 other cases; its import 
provenance is bound by the staging harness.
    - Introspection across 12 API groups found no missing public names; Row had 
no missing names and remained a tuple subclass. This checks names, not all 
signatures or behavior. The no-stubs audit passed. Rustdoc exited 0 with two 
inherited warnings in types.rs (unresolved from_json link and unclosed int 
tag). Complete source review round 12 reported no findings.
    - Workspace/all-target Clippy exited **101**. All 472 emitted source 
diagnostics and 721 primary/secondary spans map to unchanged saved-master 
statements and preserved current statements. This is source attribution, not an 
executed master lint pass or successful Clippy run. These scoped results do not 
certify the README's broad parity claim.
    
    - This artifact's official configuration and DataFrame-creation modules ran 
through its raw-transport plugin against a live JVM: **23 passed / 1 skipped / 
0 failures or errors**. The current source, saved extension and reviewed source 
match.
    - Common transport regression evidence is carried from a separate pandas 
artifact because all **53 shared transport/build/dependency inputs** match. Its 
full ordinary phase covered **87 modules: 2,586 passed / 641 skipped / 1 
manifest deselection**, with **one expected missing-catalog drift-check failure 
and no unexpected failures**. Its separate cluster/resource phase covered **4 
modules: 15 passed / 14 skipped**, including 10 functional passes and 5 guard 
passes. Only that donor art [...]
    
    The completed transport evidence and scoped native results were assessed 
together. Saved harness, source, artifact and endpoint evidence are retained; 
complete official/dependency whole-tree start/end attestations, per-case skip 
provenance and raw per-module pytest exit codes were not all recorded. The 
reported totals count test executions and do not establish whole-client parity.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: OpenAI Codex
    
    This pull request and its description were written by OpenAI Codex.
    
    Closes #398 from dbtsai/codex/residual-streaming-event-errors-master-925dc.
    
    Authored-by: DB Tsai <[email protected]>
    Signed-off-by: Hyukjin Kwon <[email protected]>
---
 crates/pyspark-rs/src/streaming.rs       | 172 +++++++++--
 crates/spark-connect/src/streaming.rs    | 513 +++++++++++++++++++++++++++++--
 docs/streaming.md                        |  29 ++
 python/pyspark/sql/streaming/query.py    |  48 ++-
 python/tests/_mock_connect_server.py     |   6 +-
 python/tests/streaming_listener_cases.py | 300 +++++++++++++++++-
 6 files changed, 1003 insertions(+), 65 deletions(-)

diff --git a/crates/pyspark-rs/src/streaming.rs 
b/crates/pyspark-rs/src/streaming.rs
index 32a5b309..50cda3d8 100644
--- a/crates/pyspark-rs/src/streaming.rs
+++ b/crates/pyspark-rs/src/streaming.rs
@@ -7,43 +7,171 @@ use spark_connect::streaming::{
     StreamingQueryListener, StreamingQueryListenerEvent, 
StreamingQueryManager, Trigger,
 };
 use spark_connect::udf::PythonUDFPayload;
+use spark_connect_core::error::{Result as SparkResult, SparkError};
 use std::collections::HashMap;
 use std::sync::Arc;
 
 use crate::dataframe::{PyDataFrame, WriterState};
 use crate::errors::ResultExt;
 
-/// Adapts a Python `StreamingQueryListener` to the native Rust listener 
trait: on each
-/// event it acquires the GIL and calls the Python-side dispatch helper (which 
builds the
-/// typed event object and invokes the right `onQuery*` callback).
+/// Adapts Python listeners to the native bus: deserialize one typed event for 
the whole
+/// snapshot, then dispatch it through Python's callback-exception isolation 
helper. Decode
+/// errors return to the bus, and terminal failures produce the reference 
handler warning.
 struct PyListenerAdapter {
     listener: Py<PyAny>,
 }
 
+/// An unisolated dispatch failure. Keep Python errors intact so synchronous 
starts
+/// can re-raise their original type; native listener errors remain typed 
SparkErrors.
+enum ListenerDispatchError {
+    Python(PyErr),
+    Native(Box<SparkError>),
+}
+
+impl From<PyErr> for ListenerDispatchError {
+    fn from(error: PyErr) -> Self {
+        Self::Python(error)
+    }
+}
+
+/// Decode before any callback, sharing one Python object across every 
registration.
+/// The same path handles synchronous starts and asynchronous protocol events, 
including
+/// mixed native/Python snapshots. Keep native errors typed until the 
reporting boundary.
+fn post_listener_event<'a>(
+    py: Python<'_>,
+    listeners: impl IntoIterator<Item = &'a dyn StreamingQueryListener>,
+    event: &StreamingQueryListenerEvent,
+    started: bool,
+) -> std::result::Result<(), ListenerDispatchError> {
+    let module = py.import("pyspark.sql.streaming.query")?;
+    let kwargs = PyDict::new(py);
+    kwargs.set_item("started", started)?;
+    let decoded = module.call_method(
+        "_deserialize_listener_event",
+        (event.event_type, event.event_json.as_str()),
+        Some(&kwargs),
+    )?;
+    for listener in listeners {
+        match listener
+            .as_any()
+            .and_then(|any| any.downcast_ref::<PyListenerAdapter>())
+        {
+            Some(adapter) => {
+                module.call_method1(
+                    "_dispatch_listener_event",
+                    (adapter.listener.bind(py), &decoded),
+                )?;
+            }
+            // Native callbacks keep their usual ability to wait for Python 
work.
+            None if started => py.detach(|| listener.on_event(event)),
+            None => py
+                .detach(|| listener.try_on_event(event).map_err(Box::new))
+                .map_err(ListenerDispatchError::Native)?,
+        }
+    }
+    Ok(())
+}
+
+/// Any Python exception not isolated by the helper terminates the 
subscription: this
+/// includes decoding errors, callback BaseException, and warnings promoted to 
errors.
+/// Async reporting keeps only str(), without allocating an unconsumed 
callback-registry id.
+fn async_dispatch_error(py: Python<'_>, error: ListenerDispatchError) -> 
Box<SparkError> {
+    match error {
+        ListenerDispatchError::Python(error) => {
+            Box::new(SparkError::connect_msg(error.value(py).to_string()))
+        }
+        ListenerDispatchError::Native(error) => error,
+    }
+}
+
 impl StreamingQueryListener for PyListenerAdapter {
     fn on_event(&self, event: &StreamingQueryListenerEvent) {
+        // This infallible entry point is also used by the core's default 
synchronous
+        // start handler. Python writer calls use post_started_event to 
propagate errors.
         Python::attach(|py| {
-            let dispatch = py
-                .import("pyspark.sql.streaming.query")
-                .and_then(|m| m.getattr("_dispatch_listener_event"));
-            if let Ok(dispatch) = dispatch {
-                // Swallow listener/callback errors so one bad listener cannot 
kill the
-                // dispatch thread (reference pyspark also isolates callback 
exceptions).
-                let _ = dispatch.call1((
-                    self.listener.bind(py),
-                    event.event_type,
-                    event.event_json.as_str(),
-                ));
-            }
+            let _ = post_listener_event(
+                py,
+                std::iter::once(self as &dyn StreamingQueryListener),
+                event,
+                true,
+            );
         });
     }
 
+    fn try_on_event(&self, event: &StreamingQueryListenerEvent) -> 
SparkResult<()> {
+        Python::attach(|py| {
+            post_listener_event(
+                py,
+                std::iter::once(self as &dyn StreamingQueryListener),
+                event,
+                false,
+            )
+            .map_err(|error| async_dispatch_error(py, error))
+        })
+        .map_err(|error| *error)
+    }
+
+    fn try_post_to_all(
+        &self,
+        listeners: &[Arc<dyn StreamingQueryListener>],
+        event: &StreamingQueryListenerEvent,
+    ) -> Option<SparkResult<()>> {
+        Some(
+            Python::attach(|py| {
+                post_listener_event(py, listeners.iter().map(Arc::as_ref), 
event, false)
+                    .map_err(|error| async_dispatch_error(py, error))
+            })
+            .map_err(|error| *error),
+        )
+    }
+
+    fn on_stream_error(&self, error: &SparkError) -> bool {
+        Python::attach(|py| {
+            let message = 
crate::errors::spark_error_to_py_exception(error.clone())
+                .value(py)
+                .str()
+                .and_then(|text| Ok(text.to_cow()?.into_owned()))
+                .unwrap_or_else(|_| error.message());
+            let _ = warn(
+                py,
+                format!(
+                    "StreamingQueryListenerBus Handler thread received 
exception, all client side \
+                     listeners are removed and handler thread is terminated. 
The error is: {}",
+                    message
+                ),
+            );
+        });
+        true
+    }
+
     /// The adapter, of which `StreamingQueryManager._sqlb` gives back the 
Python listener.
     fn as_any(&self) -> Option<&dyn std::any::Any> {
         Some(self)
     }
 }
 
+/// Parse a synchronous started event once, including when there are no 
listeners, and
+/// dispatch its typed object before start/toTable returns. Preserve the 
original Python
+/// decoding exception across the detached core call.
+#[allow(clippy::result_large_err)] // The core started-event handler requires 
SparkResult.
+fn post_started_event(
+    manager: &StreamingQueryManager,
+    event: &StreamingQueryListenerEvent,
+) -> SparkResult<()> {
+    Python::attach(|py| {
+        let listeners = manager.listeners();
+        post_listener_event(py, listeners.iter().map(Arc::as_ref), event, 
true).map_err(|error| {
+            match error {
+                ListenerDispatchError::Python(error) => {
+                    Box::new(crate::errors::callback_error(error))
+                }
+                ListenerDispatchError::Native(error) => error,
+            }
+        })
+    })
+    .map_err(|error| *error)
+}
+
 /// Apply a named read option to a streaming reader when supplied (mirrors the 
batch
 /// `set_opt` in readwriter.rs): `None` leaves the option unset (not the 
string "None");
 /// booleans lowercase to "true"/"false" via `coerce_option_value`.
@@ -856,8 +984,11 @@ impl PyDataStreamWriter {
     ) -> PyResult<PyStreamingQuery> {
         let writer = self.apply_write_opts(format, outputMode, partitionBy, 
queryName, options)?;
         // Memory/console/foreach sinks take no path; the core treats "" as 
unset.
-        let query =
-            crate::dataframe::detached(py, || 
writer.start(path.unwrap_or("")).map_err(Box::new))?;
+        let query = crate::dataframe::detached(py, || {
+            writer
+                .start_with_event_handler(path.unwrap_or(""), 
post_started_event)
+                .map_err(Box::new)
+        })?;
         Ok(PyStreamingQuery::new(query))
     }
 
@@ -874,8 +1005,11 @@ impl PyDataStreamWriter {
         options: Option<&Bound<'_, PyDict>>,
     ) -> PyResult<PyStreamingQuery> {
         let writer = self.apply_write_opts(format, outputMode, partitionBy, 
queryName, options)?;
-        let query =
-            crate::dataframe::detached(py, || 
writer.to_table(tableName).map_err(Box::new))?;
+        let query = crate::dataframe::detached(py, || {
+            writer
+                .to_table_with_event_handler(tableName, post_started_event)
+                .map_err(Box::new)
+        })?;
         Ok(PyStreamingQuery::new(query))
     }
 
diff --git a/crates/spark-connect/src/streaming.rs 
b/crates/spark-connect/src/streaming.rs
index a736b0be..67e7dfb6 100644
--- a/crates/spark-connect/src/streaming.rs
+++ b/crates/spark-connect/src/streaming.rs
@@ -384,19 +384,49 @@ impl DataStreamWriter {
     }
 
     /// Start the streaming query writing to a path, returning a 
StreamingQuery handle.
-    pub fn start(mut self, path: &str) -> Result<StreamingQuery> {
+    pub fn start(self, path: &str) -> Result<StreamingQuery> {
+        self.start_with_event_handler(path, |manager, event| {
+            manager.post_to_all(event);
+            Ok(())
+        })
+    }
+
+    /// Start a query with a language-specific handler for its synchronous 
started event.
+    /// The handler runs even when no listeners are registered and must 
deserialize the event
+    /// before dispatching it. A handler error is returned instead of the 
query handle.
+    pub fn start_with_event_handler(
+        mut self,
+        path: &str,
+        on_started: impl FnOnce(&StreamingQueryManager, 
&StreamingQueryListenerEvent) -> Result<()>,
+    ) -> Result<StreamingQuery> {
         self.path = Some(path.to_string());
-        self._start_internal()
+        self._start_internal(on_started)
     }
 
     /// Start the streaming query writing to a table, returning a 
StreamingQuery handle.
-    pub fn to_table(mut self, table_name: &str) -> Result<StreamingQuery> {
+    pub fn to_table(self, table_name: &str) -> Result<StreamingQuery> {
+        self.to_table_with_event_handler(table_name, |manager, event| {
+            manager.post_to_all(event);
+            Ok(())
+        })
+    }
+
+    /// The table-writing counterpart of
+    /// [`start_with_event_handler`](Self::start_with_event_handler).
+    pub fn to_table_with_event_handler(
+        mut self,
+        table_name: &str,
+        on_started: impl FnOnce(&StreamingQueryManager, 
&StreamingQueryListenerEvent) -> Result<()>,
+    ) -> Result<StreamingQuery> {
         self.table_name = Some(table_name.to_string());
-        self._start_internal()
+        self._start_internal(on_started)
     }
 
     /// Internal method to build and execute the write stream command.
-    fn _start_internal(self) -> Result<StreamingQuery> {
+    fn _start_internal(
+        self,
+        on_started: impl FnOnce(&StreamingQueryManager, 
&StreamingQueryListenerEvent) -> Result<()>,
+    ) -> Result<StreamingQuery> {
         let mut write_op = proto::WriteStreamOperationStart::default();
         write_op.input = Some(crate::plan::relation_with_ids(
             &self.plan,
@@ -490,12 +520,13 @@ impl DataStreamWriter {
         // with the result of its start, as the session has client-side 
listeners, and the client
         // posts it to them before it gives back the query.
         if let Some(event_json) = started_event {
-            self.session
-                .streams()
-                .post_to_all(&StreamingQueryListenerEvent {
+            on_started(
+                &self.session.streams(),
+                &StreamingQueryListenerEvent {
                     event_type: QUERY_STARTED_EVENT,
                     event_json,
-                });
+                },
+            )?;
         }
 
         Ok(StreamingQuery {
@@ -910,6 +941,39 @@ pub struct StreamingQueryListenerEvent {
 pub trait StreamingQueryListener: Send + Sync {
     fn on_event(&self, event: &StreamingQueryListenerEvent);
 
+    /// Optionally handle an asynchronous bus event for the whole listener 
snapshot, for
+    /// adapters that deserialize a shared event object once before invoking 
any callback.
+    /// Synchronous started events instead use 
[`DataStreamWriter::start_with_event_handler`] or
+    /// [`DataStreamWriter::to_table_with_event_handler`] for shared 
deserialization.
+    /// The first adapter returning `Some` owns dispatch for every entry, 
including native
+    /// listeners and duplicates. Returning `None` must not invoke callbacks; 
the bus then
+    /// tries the next adapter or falls back to each listener's
+    /// [`try_on_event`](Self::try_on_event).
+    fn try_post_to_all(
+        &self,
+        _listeners: &[Arc<dyn StreamingQueryListener>],
+        _event: &StreamingQueryListenerEvent,
+    ) -> Option<Result<()>> {
+        None
+    }
+
+    /// Deserialize and dispatch an asynchronous event. An error is any 
failure the adapter
+    /// does not isolate, including protocol/deserialization errors and 
adapter-defined
+    /// unisolated callback failures. It terminates this subscription and 
clears its registrations.
+    fn try_on_event(&self, event: &StreamingQueryListenerEvent) -> Result<()> {
+        self.on_event(event);
+        Ok(())
+    }
+
+    /// Report a terminal subscription failure from transport, deserialization 
or dispatch.
+    /// Return true when reported, so the bus emits only one notification even 
if a listener
+    /// is registered more than once. The bus invokes this without its state 
lock, before
+    /// clearing the failed generation. Like an event callback, a concurrent 
removal
+    /// of a non-last listener from another thread waits for this hook to 
return.
+    fn on_stream_error(&self, _error: &SparkError) -> bool {
+        false
+    }
+
     /// This listener as `Any`, for a client that wraps its own listeners to 
find them among
     /// [`StreamingQueryManager::listeners`]. `None` by default.
     fn as_any(&self) -> Option<&dyn std::any::Any> {
@@ -934,13 +998,13 @@ struct ListenerBusState {
     phase: ListenerBusPhase,
     worker_id: Option<std::thread::ThreadId>,
     generation: u64,
-    /// The generation whose worker is delivering an event to its snapshot of 
the listeners.
-    /// PySpark's bus holds its lock while it delivers an event, so a removal 
from another
-    /// thread waits for it.
+    /// The generation whose worker is delivering an event, or reporting a 
terminal
+    /// failure, to its snapshot of the listeners. PySpark's bus holds its 
lock during
+    /// delivery, so a removal from another thread waits for it.
     dispatching: Option<u64>,
 }
 
-/// The delivery of one event by the worker of `generation`, until it is 
dropped.
+/// An event delivery or terminal failure notification by the worker of 
`generation`.
 struct Dispatch<'a> {
     bus: &'a Weak<ListenerBus>,
     generation: u64,
@@ -995,14 +1059,13 @@ impl ListenerBus {
 /// The worker retains neither the session nor its bus. The stream owns only 
transport
 /// and progress-handler state; dropping the last session/manager cancels the 
reader.
 fn run_listener_event_loop(
-    mut stream: ListenerEventStream,
+    stream: impl IntoIterator<Item = Result<(i32, String)>>,
     bus: Weak<ListenerBus>,
     generation: u64,
 ) {
+    let mut stream = stream.into_iter();
+    let mut failure = None;
     for item in &mut stream {
-        let Ok((event_type, event_json)) = item else {
-            break;
-        };
         let Some(owner) = bus.upgrade() else {
             break;
         };
@@ -1023,12 +1086,39 @@ fn run_listener_event_loop(
             bus: &bus,
             generation,
         };
+        let (event_type, event_json) = match item {
+            Ok(event) => event,
+            Err(error) => {
+                failure = Some((listeners, error, dispatch));
+                break;
+            }
+        };
         let event = StreamingQueryListenerEvent {
             event_type,
             event_json,
         };
+        let dispatched = listeners
+            .iter()
+            .find_map(|listener| listener.try_post_to_all(&listeners, &event))
+            .unwrap_or_else(|| {
+                listeners
+                    .iter()
+                    .try_for_each(|listener| listener.try_on_event(&event))
+            });
+        if let Err(error) = dispatched {
+            failure = Some((listeners, error, dispatch));
+            break;
+        }
+        drop(dispatch);
+    }
+    if let Some((listeners, error, dispatch)) = failure {
+        // A removal from another thread waits for this terminal notification, 
just
+        // as it waits for an ordinary event. No state lock or strong bus 
owner is
+        // held across either callback path; the worker can still remove 
itself.
         for listener in listeners {
-            listener.on_event(&event);
+            if listener.on_stream_error(&error) {
+                break;
+            }
         }
         drop(dispatch);
     }
@@ -1176,10 +1266,10 @@ impl StreamingQueryManager {
             };
             if state.listeners.len() > 1 {
                 state.listeners.remove(index);
-                // As PySpark's bus, whose delivery of an event holds the lock 
of its removal:
-                // the removal returns once the event in delivery has been 
delivered, also to
-                // the removed listener, which gets no events after it. A 
callback of the bus
-                // does not wait, as it would wait for its own delivery.
+                // As in PySpark's bus, a removal waits for delivery to the 
snapshot,
+                // including the removed listener, to finish. This also covers 
a
+                // terminal error notification. A callback of the bus does not 
wait,
+                // as it would wait for its own delivery.
                 if state.worker_id != Some(std::thread::current().id()) {
                     let generation = state.generation;
                     while state.dispatching == Some(generation) {
@@ -1681,6 +1771,385 @@ mod tests {
             .expect("only terminal close eventually cancels the current 
worker");
     }
 
+    #[test]
+    fn 
listener_bus_reports_transport_and_decode_failures_once_before_clearing() {
+        use std::sync::atomic::{AtomicUsize, Ordering};
+
+        struct Listener {
+            bus: Weak<ListenerBus>,
+            events: AtomicUsize,
+            errors: Mutex<Vec<String>>,
+        }
+        impl StreamingQueryListener for Listener {
+            fn on_event(&self, _: &StreamingQueryListenerEvent) {
+                self.events.fetch_add(1, Ordering::SeqCst);
+            }
+
+            fn try_on_event(&self, event: &StreamingQueryListenerEvent) -> 
Result<()> {
+                if event.event_json == "invalid" {
+                    return Err(SparkError::connect_msg("event decode failed"));
+                }
+                self.on_event(event);
+                Ok(())
+            }
+
+            fn on_stream_error(&self, error: &SparkError) -> bool {
+                // Reporting runs before cleanup, without holding the bus lock.
+                let bus = self.bus.upgrade().unwrap();
+                let state = bus.state.lock().unwrap();
+                assert_eq!(state.listeners.len(), 2);
+                assert_eq!(state.dispatching, Some(0));
+                drop(state);
+                self.errors.lock().unwrap().push(error.message());
+                true
+            }
+        }
+
+        for decode_failure in [false, true] {
+            let bus = Arc::new(ListenerBus::default());
+            let listener = Arc::new(Listener {
+                bus: Arc::downgrade(&bus),
+                events: AtomicUsize::new(0),
+                errors: Mutex::new(vec![]),
+            });
+            {
+                let mut state = bus.state.lock().unwrap();
+                state
+                    .listeners
+                    .push(("first".to_string(), listener.clone()));
+                state
+                    .listeners
+                    .push(("second".to_string(), listener.clone()));
+            }
+            let failure = if decode_failure {
+                Ok((QUERY_IDLE_EVENT, "invalid".to_string()))
+            } else {
+                Err(SparkError::connect_msg("transport failed"))
+            };
+            run_listener_event_loop(
+                vec![failure, Ok((QUERY_IDLE_EVENT, "valid".to_string()))],
+                Arc::downgrade(&bus),
+                0,
+            );
+            assert_eq!(listener.events.load(Ordering::SeqCst), 0);
+            assert_eq!(
+                *listener.errors.lock().unwrap(),
+                vec![if decode_failure {
+                    "event decode failed"
+                } else {
+                    "transport failed"
+                }],
+            );
+            let state = bus.state.lock().unwrap();
+            assert!(state.listeners.is_empty());
+            assert!(state.dispatching.is_none());
+        }
+    }
+
+    #[test]
+    fn 
nonlast_removal_waits_for_transport_decode_and_batch_error_notifications() {
+        use std::sync::atomic::{AtomicUsize, Ordering};
+        use std::sync::mpsc;
+        use std::time::{Duration, Instant};
+
+        struct ReleaseOnDrop(mpsc::Sender<()>);
+        impl Drop for ReleaseOnDrop {
+            fn drop(&mut self) {
+                let _ = self.0.send(());
+            }
+        }
+
+        struct Listener {
+            batch_failure: bool,
+            entered: mpsc::Sender<()>,
+            release: Mutex<mpsc::Receiver<()>>,
+            order: Arc<Mutex<Vec<&'static str>>>,
+            events: AtomicUsize,
+            errors: AtomicUsize,
+        }
+        impl StreamingQueryListener for Listener {
+            fn on_event(&self, _: &StreamingQueryListenerEvent) {
+                self.events.fetch_add(1, Ordering::SeqCst);
+            }
+
+            fn try_post_to_all(
+                &self,
+                _: &[Arc<dyn StreamingQueryListener>],
+                _: &StreamingQueryListenerEvent,
+            ) -> Option<Result<()>> {
+                self.batch_failure
+                    .then(|| Err(SparkError::connect_msg("batch decode 
failed")))
+            }
+
+            fn try_on_event(&self, _: &StreamingQueryListenerEvent) -> 
Result<()> {
+                Err(SparkError::connect_msg("event decode failed"))
+            }
+
+            fn on_stream_error(&self, _: &SparkError) -> bool {
+                self.errors.fetch_add(1, Ordering::SeqCst);
+                self.order.lock().unwrap().push("reporting");
+                self.entered.send(()).unwrap();
+                self.release
+                    .lock()
+                    .unwrap()
+                    .recv_timeout(Duration::from_secs(15))
+                    .expect("the test releases the terminal notification");
+                self.order.lock().unwrap().push("reported");
+                true
+            }
+        }
+
+        for failure_kind in ["transport", "decode", "batch"] {
+            let spark = session();
+            let manager = spark.streams();
+            let (entered, reporting) = mpsc::channel();
+            let (release, released) = mpsc::channel();
+            let order = Arc::new(Mutex::new(vec![]));
+            let listener = Arc::new(Listener {
+                batch_failure: failure_kind == "batch",
+                entered,
+                release: Mutex::new(released),
+                order: Arc::clone(&order),
+                events: AtomicUsize::new(0),
+                errors: AtomicUsize::new(0),
+            });
+            {
+                let mut state = manager.bus.state.lock().unwrap();
+                state
+                    .listeners
+                    .push(("first".to_string(), listener.clone()));
+                state
+                    .listeners
+                    .push(("second".to_string(), listener.clone()));
+            }
+            let failure = if failure_kind == "transport" {
+                Err(SparkError::connect_msg("transport failed"))
+            } else {
+                Ok((QUERY_IDLE_EVENT, "event".to_string()))
+            };
+            let bus = Arc::downgrade(&manager.bus);
+            let (start, started) = mpsc::channel();
+            let (removed, removal) = mpsc::channel();
+            std::thread::scope(|scope| {
+                // Release the hook before scoped threads are joined even if a
+                // main-thread assertion fails while it is blocked.
+                let _release_on_drop = ReleaseOnDrop(release.clone());
+                let worker = scope.spawn(move || {
+                    started.recv_timeout(Duration::from_secs(5)).unwrap();
+                    run_listener_event_loop(vec![failure], bus, 0);
+                });
+                manager.bus.state.lock().unwrap().worker_id = 
Some(worker.thread().id());
+                start.send(()).unwrap();
+                reporting.recv_timeout(Duration::from_secs(5)).unwrap();
+                let removal_order = Arc::clone(&order);
+                let manager = &manager;
+                let remover = scope.spawn(move || {
+                    assert!(manager.remove_listener_found("second").unwrap());
+                    removal_order.lock().unwrap().push("removed");
+                    removed.send(()).unwrap();
+                });
+                let deadline = Instant::now() + Duration::from_secs(5);
+                loop {
+                    if manager.bus.state.lock().unwrap().listeners.len() == 1 {
+                        break;
+                    }
+                    assert!(Instant::now() < deadline, "remover did not reach 
the bus");
+                    std::thread::yield_now();
+                }
+                assert!(matches!(
+                    removal.recv_timeout(Duration::from_millis(100)),
+                    Err(mpsc::RecvTimeoutError::Timeout)
+                ));
+                release.send(()).unwrap();
+                let removal_result = 
removal.recv_timeout(Duration::from_secs(5));
+                if removal_result.is_err() {
+                    // Keep a broken guard-release regression from stranding 
the
+                    // scoped remover while the assertion reports its failure.
+                    let mut state = manager.bus.state.lock().unwrap();
+                    if state.dispatching == Some(0) {
+                        state.dispatching = None;
+                        manager.bus.changed.notify_all();
+                    }
+                }
+                worker.join().unwrap();
+                remover.join().unwrap();
+                assert!(
+                    removal_result.is_ok(),
+                    "removal did not finish after reporting"
+                );
+            });
+            assert_eq!(*order.lock().unwrap(), ["reporting", "reported", 
"removed"]);
+            assert_eq!(listener.events.load(Ordering::SeqCst), 0);
+            assert_eq!(listener.errors.load(Ordering::SeqCst), 1);
+            let state = manager.bus.state.lock().unwrap();
+            assert!(state.listeners.is_empty());
+            assert!(state.dispatching.is_none());
+        }
+    }
+
+    #[test]
+    fn a_terminal_error_notification_does_not_keep_the_listener_bus_alive() {
+        use std::sync::mpsc;
+        use std::time::Duration;
+
+        struct Listener {
+            entered: mpsc::Sender<()>,
+            release: Mutex<mpsc::Receiver<()>>,
+        }
+        impl StreamingQueryListener for Listener {
+            fn on_event(&self, _: &StreamingQueryListenerEvent) {}
+
+            fn on_stream_error(&self, _: &SparkError) -> bool {
+                self.entered.send(()).unwrap();
+                self.release
+                    .lock()
+                    .unwrap()
+                    .recv_timeout(Duration::from_secs(5))
+                    .expect("the test releases the terminal notification");
+                true
+            }
+        }
+
+        let bus = Arc::new(ListenerBus::default());
+        let weak = Arc::downgrade(&bus);
+        let (entered, reporting) = mpsc::channel();
+        let (release, released) = mpsc::channel();
+        bus.state.lock().unwrap().listeners.push((
+            "listener".to_string(),
+            Arc::new(Listener {
+                entered,
+                release: Mutex::new(released),
+            }),
+        ));
+        let worker_bus = weak.clone();
+        let worker = std::thread::spawn(move || {
+            run_listener_event_loop(
+                vec![Err(SparkError::connect_msg("transport failed"))],
+                worker_bus,
+                0,
+            );
+        });
+        reporting.recv_timeout(Duration::from_secs(5)).unwrap();
+        drop(bus);
+        assert!(weak.upgrade().is_none());
+        release.send(()).unwrap();
+        worker.join().unwrap();
+    }
+
+    #[test]
+    fn a_stale_worker_does_not_report_errors_to_a_new_generation() {
+        use std::sync::atomic::{AtomicUsize, Ordering};
+
+        struct Listener(AtomicUsize);
+        impl StreamingQueryListener for Listener {
+            fn on_event(&self, _: &StreamingQueryListenerEvent) {
+                panic!("a stale worker must not dispatch events");
+            }
+
+            fn on_stream_error(&self, _: &SparkError) -> bool {
+                self.0.fetch_add(1, Ordering::SeqCst);
+                true
+            }
+        }
+        let bus = Arc::new(ListenerBus::default());
+        let listener = Arc::new(Listener(AtomicUsize::new(0)));
+        {
+            let mut state = bus.state.lock().unwrap();
+            state.generation = 2;
+            state
+                .listeners
+                .push(("current".to_string(), listener.clone()));
+        }
+        run_listener_event_loop(
+            vec![Err(SparkError::connect_msg("old transport failed"))],
+            Arc::downgrade(&bus),
+            1,
+        );
+        assert_eq!(listener.0.load(Ordering::SeqCst), 0);
+        assert_eq!(bus.state.lock().unwrap().listeners.len(), 1);
+    }
+
+    #[test]
+    fn listener_bus_batch_hook_runs_once_per_event_before_any_callback() {
+        use std::sync::atomic::{AtomicUsize, Ordering};
+
+        struct NativeListener(AtomicUsize);
+        impl StreamingQueryListener for NativeListener {
+            fn on_event(&self, _: &StreamingQueryListenerEvent) {
+                self.0.fetch_add(1, Ordering::SeqCst);
+            }
+        }
+        struct BatchListener {
+            fail: bool,
+            prepared: AtomicUsize,
+            callbacks: AtomicUsize,
+            errors: AtomicUsize,
+        }
+        impl StreamingQueryListener for BatchListener {
+            fn on_event(&self, _: &StreamingQueryListenerEvent) {
+                self.callbacks.fetch_add(1, Ordering::SeqCst);
+            }
+
+            fn try_post_to_all(
+                &self,
+                listeners: &[Arc<dyn StreamingQueryListener>],
+                event: &StreamingQueryListenerEvent,
+            ) -> Option<Result<()>> {
+                self.prepared.fetch_add(1, Ordering::SeqCst);
+                if self.fail {
+                    return Some(Err(SparkError::connect_msg("batch decode 
failed")));
+                }
+                for listener in listeners {
+                    listener.on_event(event);
+                }
+                Some(Ok(()))
+            }
+
+            fn on_stream_error(&self, _: &SparkError) -> bool {
+                self.errors.fetch_add(1, Ordering::SeqCst);
+                true
+            }
+        }
+
+        for fail in [false, true] {
+            let bus = Arc::new(ListenerBus::default());
+            let native = Arc::new(NativeListener(AtomicUsize::new(0)));
+            let batch = Arc::new(BatchListener {
+                fail,
+                prepared: AtomicUsize::new(0),
+                callbacks: AtomicUsize::new(0),
+                errors: AtomicUsize::new(0),
+            });
+            {
+                let mut state = bus.state.lock().unwrap();
+                // A native listener precedes duplicate adapter registrations. 
Batch
+                // preparation must run before even that first native callback.
+                state.listeners.push(("native".to_string(), native.clone()));
+                state.listeners.push(("batch-1".to_string(), batch.clone()));
+                state.listeners.push(("batch-2".to_string(), batch.clone()));
+            }
+            run_listener_event_loop(
+                vec![
+                    Ok((QUERY_IDLE_EVENT, "first".to_string())),
+                    Ok((QUERY_IDLE_EVENT, "second".to_string())),
+                ],
+                Arc::downgrade(&bus),
+                0,
+            );
+            assert_eq!(
+                batch.prepared.load(Ordering::SeqCst),
+                if fail { 1 } else { 2 }
+            );
+            assert_eq!(native.0.load(Ordering::SeqCst), if fail { 0 } else { 2 
});
+            assert_eq!(
+                batch.callbacks.load(Ordering::SeqCst),
+                if fail { 0 } else { 4 }
+            );
+            assert_eq!(batch.errors.load(Ordering::SeqCst), usize::from(fail));
+            assert!(bus.state.lock().unwrap().listeners.is_empty());
+        }
+    }
+
     #[test]
     fn stream_reader_format_option() {
         let spark = session();
diff --git a/docs/streaming.md b/docs/streaming.md
index f05c7d49..a144537c 100644
--- a/docs/streaming.md
+++ b/docs/streaming.md
@@ -113,3 +113,32 @@ for q in spark.streams().active()? {
 
 !!! warning
     Checkpoints are mandatory for fault tolerance. Always set 
`checkpointLocation` for production queries.
+
+### Listener failures
+
+For native Rust listeners, a terminal stream end or error clears the 
subscription's
+registrations. The default `StreamingQueryListener::on_stream_error` hook is 
silent;
+override it to report failures, and re-add listeners to open a new 
subscription. A
+normal stream end clears registrations without calling the error hook. 
Adapters can
+return errors through `try_on_event` or provide shared asynchronous dispatch 
through
+`try_post_to_all`. Use `DataStreamWriter::start_with_event_handler` or
+`to_table_with_event_handler` for shared synchronous start-event decoding.
+
+`SparkSession::stop()` and `close()` cancel listener dispatch and clear 
registrations
+locally, without sending a listener-removal RPC or joining callbacks already 
running.
+Reference PySpark leaves listener-bus cleanup to its handler thread during 
session stop.
+
+The PySpark drop-in listener bus deserializes each event once before invoking 
callbacks and
+shares that event object across all listener registrations. A malformed start 
event
+raises from `start()` or `toTable()`, even when no listeners are registered. A
+transport failure, malformed asynchronous event or unknown event type warns and
+clears the subscription's listeners. Add the listeners again to open a 
subscription.
+A server-side end of the event stream also clears registrations without a 
warning;
+reference PySpark retains them after a normal stream end.
+
+Callback exceptions derived from `Exception` produce per-listener warnings and
+dispatch continues. An uncaught `BaseException`, or a callback warning 
promoted to
+an error by Python's warning filters, terminates the native subscription and 
clears
+its registrations. This cleanup is deliberate: reference PySpark can retain 
stale
+registrations when its handler exits through `BaseException`. The terminal 
handler
+warning is best effort when warning filters themselves raise errors.
diff --git a/python/pyspark/sql/streaming/query.py 
b/python/pyspark/sql/streaming/query.py
index c0ae0ae9..4ba55391 100644
--- a/python/pyspark/sql/streaming/query.py
+++ b/python/pyspark/sql/streaming/query.py
@@ -41,6 +41,8 @@ from pyspark._pyspark import (
     StreamingQuery as _StreamingQuery,
     StreamingQueryManager as _StreamingQueryManager,
 )
+from pyspark.errors import PySparkValueError
+
 # As PySpark's, the exception of a failed query is that of pyspark.errors.
 from pyspark.errors import StreamingQueryException  # noqa: F401
 
@@ -60,24 +62,40 @@ _QUERY_IDLE_EVENT = 3
 StreamingQuery = _StreamingQuery
 
 
-def _dispatch_listener_event(listener, event_type, event_json):
-    """Deserialize a listener event and invoke the matching callback on 
``listener``.
+def _deserialize_listener_event(event_type, event_json, *, started=False):
+    """Decode a protocol event before entering callback exception isolation.
 
-    Called from the native (Rust) listener bus for each event. Mirrors the 
reference
-    ``StreamingQueryListenerBus.post_to_all``: build the typed event and route 
it to
-    onQueryStarted / onQueryProgress / onQueryIdle / onQueryTerminated. 
Callback exceptions
-    are surfaced as warnings, as PySpark's, so one listener cannot break the 
dispatch thread.
+    Started events belong to the synchronous start response, not the 
asynchronous
+    protocol enum. Malformed JSON and missing fields propagate to the caller, 
as in
+    reference PySpark's writer and listener-bus deserializers.
     """
+    if started and event_type == _QUERY_STARTED_EVENT:
+        return QueryStartedEvent.fromJson(json.loads(event_json))
+    if event_type == _QUERY_PROGRESS_EVENT:
+        return QueryProgressEvent.fromJson(json.loads(event_json))
+    if event_type == _QUERY_TERMINATED_EVENT:
+        return QueryTerminatedEvent.fromJson(json.loads(event_json))
+    if event_type == _QUERY_IDLE_EVENT:
+        return QueryIdleEvent.fromJson(json.loads(event_json))
+    raise PySparkValueError(
+        errorClass="UNKNOWN_VALUE_FOR",
+        messageParameters={"var": f"proto.StreamingQueryEventType: 
{event_type}"},
+    )
+
+
+def _dispatch_listener_event(listener, event):
+    """Warn on callback Exception; BaseException and warning-as-error escape 
to the bus."""
     try:
-        j = json.loads(event_json)
-        if event_type == _QUERY_STARTED_EVENT:
-            listener.onQueryStarted(QueryStartedEvent.fromJson(j))
-        elif event_type == _QUERY_PROGRESS_EVENT:
-            listener.onQueryProgress(QueryProgressEvent.fromJson(j))
-        elif event_type == _QUERY_IDLE_EVENT:
-            listener.onQueryIdle(QueryIdleEvent.fromJson(j))
-        elif event_type == _QUERY_TERMINATED_EVENT:
-            listener.onQueryTerminated(QueryTerminatedEvent.fromJson(j))
+        if isinstance(event, QueryStartedEvent):
+            listener.onQueryStarted(event)
+        elif isinstance(event, QueryProgressEvent):
+            listener.onQueryProgress(event)
+        elif isinstance(event, QueryIdleEvent):
+            listener.onQueryIdle(event)
+        elif isinstance(event, QueryTerminatedEvent):
+            listener.onQueryTerminated(event)
+        else:
+            warnings.warn(f"Unknown StreamingQueryListener event: {event}")
     except Exception as e:  # noqa: BLE001
         warnings.warn(f"Listener {str(listener)} threw an exception\n{e}")
 
diff --git a/python/tests/_mock_connect_server.py 
b/python/tests/_mock_connect_server.py
index a2d27402..3e09303e 100644
--- a/python/tests/_mock_connect_server.py
+++ b/python/tests/_mock_connect_server.py
@@ -793,7 +793,11 @@ class RemoteService:
         self.connection.recv()
 
     def listener_event(self, session_id, event_type, event_json):
-        self.connection.send(("listener_event", session_id, event_type, 
json.dumps(event_json)))
+        self.listener_event_json(session_id, event_type, 
json.dumps(event_json))
+
+    def listener_event_json(self, session_id, event_type, event_json):
+        """Send raw event JSON, including malformed payloads for protocol 
error tests."""
+        self.connection.send(("listener_event", session_id, event_type, 
event_json))
         assert self.connection.poll(10), "Mock server did not respond"
         self.connection.recv()
 
diff --git a/python/tests/streaming_listener_cases.py 
b/python/tests/streaming_listener_cases.py
index 88cf3d47..0daf541a 100644
--- a/python/tests/streaming_listener_cases.py
+++ b/python/tests/streaming_listener_cases.py
@@ -16,6 +16,7 @@
 #
 """Streaming listener lifecycle tests against the separately hosted mock 
service."""
 
+import json
 import subprocess
 import sys
 import tempfile
@@ -26,7 +27,7 @@ from pathlib import Path
 
 import pytest
 
-from pyspark.sql.streaming.listener import StreamingQueryListener
+from pyspark.sql.streaming.listener import QueryIdleEvent, 
StreamingQueryListener
 
 
 class IdleListener(StreamingQueryListener):
@@ -73,18 +74,304 @@ def run_listener_case(*arguments):
             )
         except subprocess.TimeoutExpired:
             output.seek(0)
-            pytest.fail("Concurrent listener shutdown timed out:\n" + 
output.read())
+            pytest.fail(f"Listener case {arguments} timed out:\n" + 
output.read())
         output.seek(0)
         assert result.returncode == 0, output.read()
 
 
-def listener_warnings(caught):
-    prefixes = ("Failed to add the listener", "Failed to remove the listener")
+_LIFECYCLE_WARNING_PREFIXES = ("Failed to add the listener", "Failed to remove 
the listener")
+_HANDLER_WARNING_PREFIX = (
+    "StreamingQueryListenerBus Handler thread received exception, all client 
side "
+    "listeners are removed and handler thread is terminated. The error is: "
+)
+
+
+def warnings_with_prefixes(caught, prefixes):
     return [str(item.message) for item in caught
             if issubclass(item.category, UserWarning) and 
str(item.message).startswith(prefixes)]
 
 
+def listener_warnings(caught):
+    return warnings_with_prefixes(caught, _LIFECYCLE_WARNING_PREFIXES)
+
+
+def listener_event_warnings(caught):
+    return warnings_with_prefixes(
+        caught, (_HANDLER_WARNING_PREFIX, "Listener ", 
*_LIFECYCLE_WARNING_PREFIXES)
+    )
+
+
+def emit_unrelated_warnings():
+    warnings.warn("unrelated streaming test deprecation", DeprecationWarning)
+    warnings.warn("unrelated streaming test user warning", UserWarning)
+
+
+def wait_for_empty_bus(manager, message="failed stream kept its 
registrations"):
+    deadline = time.monotonic() + 5
+    while manager._sqlb._listener_bus:
+        assert time.monotonic() < deadline, message
+        time.sleep(0.01)
+
+
+def start_response(event_json):
+    from _mock_connect_server import message_field
+
+    # WriteStreamOperationStartResult: query_id=1, query_started_event_json=3.
+    return message_field(
+        8,
+        message_field(1, message_field(1, "q") + message_field(2, "r"))
+        + message_field(3, event_json),
+    )
+
+
 class TestStreamingListenerLifecycle:
+    @pytest.mark.parametrize("method", ["start", "toTable"])
+    @pytest.mark.parametrize("registered", [False, True])
+    @pytest.mark.parametrize("event_json, exception", [
+        ("", json.JSONDecodeError), ("{", json.JSONDecodeError), ("{}", 
KeyError),
+    ], ids=["empty-json", "invalid-json", "missing-fields"])
+    def test_streaming_started_event_decode_error_propagates(
+        self, mock_connect, method, registered, event_json, exception,
+    ):
+        spark, service = mock_connect
+        service.listener_bus_enabled = True
+        listener = IdleListener()
+        try:
+            if registered:
+                spark.streams.addListener(listener)
+            service.execute_payload = start_response(event_json)
+            writer = spark.readStream.format("rate").load().writeStream
+            with warnings.catch_warnings(record=True) as caught:
+                warnings.simplefilter("always")
+                emit_unrelated_warnings()
+                with pytest.raises(exception):
+                    getattr(writer, method)(*(["table"] if method == "toTable" 
else []))
+            assert not listener_event_warnings(caught)
+            assert spark.streams._sqlb._listener_bus == ([listener] if 
registered else [])
+        finally:
+            spark.streams.close()
+
+    @pytest.mark.parametrize("method", ["start", "toTable"])
+    def 
test_streaming_started_callbacks_share_typed_event_and_isolate_exceptions(
+        self, mock_connect, method,
+    ):
+        spark, service = mock_connect
+        service.listener_bus_enabled = True
+        events = []
+
+        class StartedListener(IdleListener):
+            def onQueryStarted(self, event):
+                events.append(event)
+                if len(events) == 1:
+                    raise ValueError("started callback failed")
+
+        listener = StartedListener()
+        try:
+            # Duplicate registrations receive the same deserialized event, 
synchronously.
+            spark.streams.addListener(listener)
+            spark.streams.addListener(listener)
+            service.execute_payload = start_response(json.dumps({
+                "id": "00000000-0000-4000-8000-000000000001",
+                "runId": "00000000-0000-4000-8000-000000000002",
+                "name": "query", "timestamp": "started", "jobTags": ["tag"],
+            }))
+            writer = spark.readStream.format("rate").load().writeStream
+            with warnings.catch_warnings(record=True) as caught:
+                warnings.simplefilter("always")
+                emit_unrelated_warnings()
+                query = getattr(writer, method)(*(["table"] if method == 
"toTable" else []))
+            assert query.id == "q"
+            relevant = listener_event_warnings(caught)
+            assert len(relevant) == 1 and "started callback failed" in 
relevant[0]
+            assert len(events) == 2 and events[0] is events[1]
+            assert events[0].timestamp == "started" and events[0].jobTags == 
{"tag"}
+            assert spark.streams._sqlb._listener_bus == [listener, listener]
+        finally:
+            spark.streams.close()
+
+    @pytest.mark.parametrize("event_type, event_json", [
+        (None, ""),
+        (3, ""),
+        (3, "{"),
+        (3, "{}"),
+        (99, "{}"),
+        (0, "{}"),
+        (-1, "{}"),
+    ], ids=["transport", "empty-json", "invalid-json", "missing-fields", 
"unknown-type",
+            "unspecified-type", "async-started-type"])
+    def test_streaming_bus_errors_warn_once_clear_all_and_allow_reregistration(
+        self, mock_connect, event_type, event_json,
+    ):
+        spark, service = mock_connect
+        service.listener_bus_enabled = True
+        manager = spark.streams
+        first, second = IdleListener(), IdleListener()
+        try:
+            manager.addListener(first)
+            manager.addListener(first)
+            manager.addListener(second)
+            emit_idle(service, spark, "before-error")
+            first.wait_for("before-error")
+            second.wait_for("before-error")
+            if event_type is None:
+                from pyspark.errors.exceptions.connect import 
SparkConnectGrpcException
+
+                # Compare the entire error rendering with a normal RPC abort 
carrying
+                # the same status and details as the listener-stream abort.
+                previous_failure = service.execute_failure
+                service.execute_failure = "listener event stream failed"
+                try:
+                    with pytest.raises(SparkConnectGrpcException) as error:
+                        spark.range(1).collect()
+                    expected_error = str(error.value)
+                finally:
+                    service.execute_failure = previous_failure
+            elif event_type == 3:
+                # Derive the complete reference string from the vendored event 
decoder,
+                # independently of the native bus and its exception round trip.
+                with pytest.raises((json.JSONDecodeError, KeyError)) as error:
+                    QueryIdleEvent.fromJson(json.loads(event_json))
+                expected_error = str(error.value)
+            else:
+                from pyspark.errors import PySparkValueError
+
+                expected_error = str(PySparkValueError(
+                    errorClass="UNKNOWN_VALUE_FOR",
+                    messageParameters={"var": f"proto.StreamingQueryEventType: 
{event_type}"},
+                ))
+            with warnings.catch_warnings(record=True) as caught:
+                warnings.simplefilter("always")
+                emit_unrelated_warnings()
+                if event_type is None:
+                    assert service.end_listener_stream(spark.session_id, 
failure=True)
+                else:
+                    service.listener_event_json(spark.session_id, event_type, 
event_json)
+                wait_for_empty_bus(manager)
+            messages = listener_event_warnings(caught)
+            assert len(messages) == 1
+            assert messages[0] == _HANDLER_WARNING_PREFIX + expected_error
+            assert first.timestamps == ["before-error", "before-error"]
+            assert second.timestamps == ["before-error"]
+            assert service.wait_listener_streams_finished(1)
+            assert service.listener_removals == 0
+
+            manager.addListener(second)
+            assert service.listener_subscriptions == 2
+            emit_idle(service, spark, "after-error")
+            second.wait_for("after-error")
+            assert manager._sqlb._listener_bus == [second]
+            assert first.timestamps == ["before-error", "before-error"]
+            manager.removeListener(second)
+            assert service.wait_listener_streams_finished(2)
+        finally:
+            manager.close()
+
+    def 
test_streaming_callback_exceptions_warn_and_leave_the_bus_running(self, 
mock_connect):
+        spark, service = mock_connect
+        service.listener_bus_enabled = True
+
+        class FailingListener(IdleListener):
+            def onQueryIdle(self, event):
+                super().onQueryIdle(event)
+                raise ValueError("idle callback failed")
+
+        first, second = FailingListener(), IdleListener()
+        try:
+            spark.streams.addListener(first)
+            spark.streams.addListener(second)
+            with warnings.catch_warnings(record=True) as caught:
+                warnings.simplefilter("always")
+                emit_unrelated_warnings()
+                for timestamp in ("first-callback", "second-callback"):
+                    emit_idle(service, spark, timestamp)
+                    second.wait_for(timestamp)
+            relevant = listener_event_warnings(caught)
+            assert len(relevant) == 2
+            assert relevant == [f"Listener {first} threw an exception\nidle 
callback failed"] * 2
+            assert first.timestamps == second.timestamps == ["first-callback", 
"second-callback"]
+            assert spark.streams._sqlb._listener_bus == [first, second]
+            assert service.listener_subscriptions == 1
+        finally:
+            spark.streams.close()
+
+    def test_streaming_duplicate_async_listeners_share_one_decoded_event(
+        self, mock_connect, monkeypatch,
+    ):
+        from pyspark.sql.streaming import query as query_module
+
+        spark, service = mock_connect
+        service.listener_bus_enabled = True
+        decoded, received = [], []
+        deserialize = query_module._deserialize_listener_event
+
+        def count_decode(*args, **kwargs):
+            event = deserialize(*args, **kwargs)
+            decoded.append(event)
+            return event
+
+        monkeypatch.setattr(query_module, "_deserialize_listener_event", 
count_decode)
+
+        class SharedListener(IdleListener):
+            def onQueryIdle(self, event):
+                with self.condition:
+                    received.append(event)
+                    self.condition.notify_all()
+
+        listener = SharedListener()
+        try:
+            spark.streams.addListener(listener)
+            spark.streams.addListener(listener)
+            with warnings.catch_warnings(record=True) as caught:
+                warnings.simplefilter("always")
+                emit_unrelated_warnings()
+                emit_idle(service, spark, "shared-event")
+                with listener.condition:
+                    assert listener.condition.wait_for(lambda: len(received) 
== 2, timeout=5)
+            assert not listener_event_warnings(caught)
+            assert len(decoded) == 1
+            assert received[0] is received[1] is decoded[0]
+            assert received[0].timestamp == "shared-event"
+        finally:
+            spark.streams.close()
+
+    @pytest.mark.parametrize("failure", ["base-exception", "warning-as-error"])
+    def test_streaming_unisolated_callback_errors_clear_the_bus(self, 
mock_connect, failure):
+        spark, service = mock_connect
+        service.listener_bus_enabled = True
+
+        class CallbackExit(BaseException):
+            pass
+
+        class FailingListener(IdleListener):
+            def onQueryIdle(self, event):
+                super().onQueryIdle(event)
+                if failure == "base-exception":
+                    raise CallbackExit("uncaught callback exit")
+                raise ValueError("callback warning becomes an error")
+
+        first, second = FailingListener(), IdleListener()
+        try:
+            spark.streams.addListener(first)
+            spark.streams.addListener(second)
+            with warnings.catch_warnings(record=True) as caught:
+                warnings.simplefilter("always")
+                warnings.filterwarnings("error", message="Listener ", 
category=UserWarning)
+                emit_unrelated_warnings()
+                emit_idle(service, spark, "unisolated-error")
+                wait_for_empty_bus(spark.streams)
+            relevant = listener_event_warnings(caught)
+            assert len(relevant) == 1
+            expected_error = (
+                "uncaught callback exit" if failure == "base-exception"
+                else f"Listener {first} threw an exception\ncallback warning 
becomes an error"
+            )
+            assert relevant[0] == _HANDLER_WARNING_PREFIX + expected_error
+            assert first.timestamps == ["unisolated-error"]
+            assert second.timestamps == []
+            assert service.wait_listener_streams_finished(1)
+        finally:
+            spark.streams.close()
+
     @pytest.mark.parametrize("remove_failure, callback_action", [
         (False, "remove"), (True, "remove"), (False, "stop"), (True, "stop"), 
(False, "add"),
         (False, "remove_duplicate"), (True, "remove_duplicate"),
@@ -725,10 +1012,7 @@ def ended_stream_case(termination):
             # or with a nonretryable server error. Server completion precedes 
native cleanup.
             assert service.end_listener_stream(spark.session_id, 
failure=termination == "error")
             assert service.wait_listener_streams_finished(1)
-            deadline = time.monotonic() + 5
-            while manager._sqlb._listener_bus:
-                assert time.monotonic() < deadline, "ended stream kept its 
registrations"
-                time.sleep(0.01)
+            wait_for_empty_bus(manager, "ended stream kept its registrations")
             assert listener._rust_listener_ids == stale_ids
             manager.removeListener(listener)
             assert listener._rust_listener_ids == stale_ids


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to