This is an automated email from the ASF dual-hosted git repository.

HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git


The following commit(s) were added to refs/heads/master by this push:
     new e8b1d428 [SPARK-61162][CONNECT][PYTHON] Share the streaming query 
listener bus of a session, post the started events of queries, and give the 
listeners the session, in the Rust and Python clients, as PySpark does
e8b1d428 is described below

commit e8b1d4284e11d119c4fda9563a913ece22f43e7c
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Sun Nov 8 15:07:28 2026 +0900

    [SPARK-61162][CONNECT][PYTHON] Share the streaming query listener bus of a 
session, post the started events of queries, and give the listeners the 
session, in the Rust and Python clients, as PySpark does
    
    ### What changes were proposed in this pull request?
    
    [SPARK-61162](https://issues.apache.org/jira/browse/SPARK-61162): PySpark 
Connect's `spark.streams` is one `StreamingQueryManager` of the session, whose 
`StreamingQueryListenerBus` holds the client-side listeners. This PR makes the 
clients' listeners work the same way.
    
    `addListener` calls `_set_spark_session` of PySpark's 
`StreamingQueryListener`, which #350 (SPARK-61151) vendored.
    
    **Rust core** (`spark-connect`):
    - The managers of a session (`session.streams()`) share the listener bus of 
the session, rather than each having one of its own.
    - `add_listener` of the first listener returns once the server has added 
its listener (`listener_bus_listener_added`), as PySpark's 
`_register_server_side_listener` waits for it. So a query started right after 
it sends its started event.
    - `DataStreamWriter` posts the started event of a query, which the server 
sends with the result of the start (`query_started_event_json`), to the 
listeners before `start` returns, as `QUERY_STARTED_EVENT`.
    - `remove_listener` of an id of no listener does nothing. For the last 
listener, it asks the server to remove its listener, waits for the events left, 
and should that request fail, keeps the listener and returns the error, as 
PySpark's.
    - `listeners()` gives the listeners, in their order. 
`StreamingQueryListener` gains `as_any`, `None` by default, for a client that 
wraps its listeners.
    - The events are posted to the listeners without any lock held, so a 
listener can add or remove listeners.
    
    **Python client**:
    - `spark.streams` is the same object at each access.
    - `addListener` gives the listener the session (`listener.spark`), as 
PySpark Connect's `_set_spark_session`.
    - A failure to add or remove the listener of the server warns, with 
PySpark's messages, rather than raising.
    - `StreamingQueryManager._sqlb._listener_bus` lists the listeners, as 
PySpark Connect's, which Apache Spark's tests read to remove them.
    - The callbacks get the started events (`onQueryStarted`). An exception of 
a callback warns with PySpark's message.
    
    ### Why are the changes needed?
    
    - `onQueryStarted` was never called: the client ignored 
`query_started_event_json`.
    - Listeners had no session (`self.spark` was `None`), so a listener that 
runs queries failed.
    - A listener added through one `spark.streams` was removed through another 
only because removing from an empty bus asked the server to remove the listener 
of the session, which also cut the other listeners off.
    - `addListener` returned before the server had added its listener, so a 
query started right after could miss its events.
    - `spark.streams == spark.streams` was `False`, and the manager had no 
`_sqlb`.
    
    Six tests of Apache Spark's `test_parity_listener.py`, and 
`test_query_manager_no_recreation` of `test_parity_streaming.py`, failed on 
this client.
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes. The listeners of streaming queries get their events, and their 
session, as with PySpark 4.3.0. For the Rust API, `QUERY_STARTED_EVENT`, 
`StreamingQueryManager::listeners` and `StreamingQueryListener::as_any` are new.
    
    ### How was this patch tested?
    
    - A new Rust e2e test adds a listener through one manager of a session, and 
checks:
      - that it gets the started event before `start` returns, and progress 
events
      - that the other managers of the session list it
      - that it gets no more events once removed through another manager
    - A new live test checks:
      - that `spark.streams` is the same object
      - the session of the listeners, and `_sqlb._listener_bus`
      - that two listeners get the started event before `start` returns
      - that a removed listener gets no more events, while the other gets the 
terminated event
    
      It passes with official PySpark 4.3.0, and fails without this change.
    - Apache Spark's `test_parity_listener.py` and `test_parity_streaming.py` 
pass on this client against a local server, all 9 and 30 tests. Before this 
change, 6 and 1 failed.
    - The lib tests of both crates (597 passed), the build of all tests, also 
with `wasm-udf`, the offline Python suites (2214 passed, 8 skipped), and the 
live drop-in and Arrow suites (200 passed) and live cases (10 passed) against a 
local Spark 4.3.0 server. Clippy reports no diagnostics in the changed lines.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Opus 5.5
    
    This pull request and its description were written by Isaac.
    
    Closes #353 from HyukjinKwon/SPARK-61162-listener-bus.
    
    Authored-by: Hyukjin Kwon <[email protected]>
    Signed-off-by: Hyukjin Kwon <[email protected]>
---
 crates/pyspark-rs/src/session.rs            |  26 ++-
 crates/pyspark-rs/src/streaming.rs          | 104 +++++++++--
 crates/spark-connect/src/session.rs         |  12 ++
 crates/spark-connect/src/streaming.rs       | 277 +++++++++++++++++++++-------
 crates/spark-connect/tests/e2e_streaming.rs |  70 +++++++
 python/pyspark/sql/streaming/query.py       |  17 +-
 python/tests/test_dropin_e2e.py             |  51 +++++
 7 files changed, 463 insertions(+), 94 deletions(-)

diff --git a/crates/pyspark-rs/src/session.rs b/crates/pyspark-rs/src/session.rs
index b19a8445..1bfe99b1 100644
--- a/crates/pyspark-rs/src/session.rs
+++ b/crates/pyspark-rs/src/session.rs
@@ -564,6 +564,8 @@ fn tag_of(py: Python<'_>, tag: &Bound<'_, PyAny>) -> 
PyResult<String> {
 #[pyclass(name = "SparkSession", module = "pyspark.sql.session")]
 pub struct PySparkSession {
     pub(crate) session: SparkSession,
+    /// The `StreamingQueryManager` of `spark.streams`, made at its first 
access.
+    streams: std::sync::OnceLock<Py<PyStreamingQueryManager>>,
 }
 
 impl PySparkSession {
@@ -576,7 +578,10 @@ impl PySparkSession {
                 error.write_unraisable(py, None);
             }
         });
-        PySparkSession { session }
+        PySparkSession {
+            session,
+            streams: std::sync::OnceLock::new(),
+        }
     }
 
     /// A DataFrame of local rows, as PySpark Connect creates one: the schema 
is given, or
@@ -1768,14 +1773,23 @@ impl PySparkSession {
         PyDataStreamReader::new(reader)
     }
 
-    /// Get the StreamingQueryManager for managing active streaming queries 
(`spark.streams`).
+    /// Get the StreamingQueryManager for managing active streaming queries 
(`spark.streams`):
+    /// the same object at each access, as PySpark Connect's.
     ///
-    /// The native manager owns a Rust-side listener bus, so 
`addListener`/`removeListener`/
-    /// `close` are implemented in the core (Rust clients get the feature too).
+    /// The native manager is of the Rust-side listener bus of the session, so
+    /// `addListener`/`removeListener`/`close` are implemented in the core 
(Rust clients get the
+    /// feature too).
     #[getter]
     #[pyo3(name = "streams")]
-    fn streams(&self) -> PyStreamingQueryManager {
-        PyStreamingQueryManager::new(self.session.streams())
+    fn streams(&self, py: Python<'_>) -> PyResult<Py<PyStreamingQueryManager>> 
{
+        if let Some(manager) = self.streams.get() {
+            return Ok(manager.clone_ref(py));
+        }
+        let manager = Py::new(
+            py,
+            PyStreamingQueryManager::new(self.session.streams(), 
self.session.clone()),
+        )?;
+        Ok(self.streams.get_or_init(|| manager).clone_ref(py))
     }
 
     /// Stop this Spark session. Best-effort and idempotent, mirroring 
pyspark: a failure
diff --git a/crates/pyspark-rs/src/streaming.rs 
b/crates/pyspark-rs/src/streaming.rs
index cc47079a..3a7c7df6 100644
--- a/crates/pyspark-rs/src/streaming.rs
+++ b/crates/pyspark-rs/src/streaming.rs
@@ -37,6 +37,11 @@ impl StreamingQueryListener for PyListenerAdapter {
             }
         });
     }
+
+    /// The adapter, of which `StreamingQueryManager._sqlb` gives back the 
Python listener.
+    fn as_any(&self) -> Option<&dyn std::any::Any> {
+        Some(self)
+    }
 }
 
 /// Apply a named read option to a streaming reader when supplied (mirrors the 
batch
@@ -1119,15 +1124,57 @@ impl PyStreamingQuery {
     }
 }
 
-/// Python wrapper for StreamingQueryManager.
+/// Python wrapper for StreamingQueryManager, of the listener bus of its 
session.
 #[pyclass(name = "StreamingQueryManager", module = 
"pyspark.sql.streaming.query")]
 pub struct PyStreamingQueryManager {
     inner: StreamingQueryManager,
+    /// The session, which the listeners get as their `spark`.
+    session: spark_connect::session::SparkSession,
 }
 
 impl PyStreamingQueryManager {
-    pub fn new(manager: StreamingQueryManager) -> Self {
-        PyStreamingQueryManager { inner: manager }
+    pub fn new(
+        manager: StreamingQueryManager,
+        session: spark_connect::session::SparkSession,
+    ) -> Self {
+        PyStreamingQueryManager {
+            inner: manager,
+            session,
+        }
+    }
+}
+
+/// Warn of `message`, as PySpark Connect's listener bus warns of a failure to 
add or remove a
+/// listener, rather than raising.
+fn warn(py: Python<'_>, message: String) -> PyResult<()> {
+    py.import("warnings")?.call_method1("warn", (message,))?;
+    Ok(())
+}
+
+/// `StreamingQueryManager._sqlb`, as PySpark Connect's 
`StreamingQueryListenerBus`, of its
+/// `_listener_bus`: the client-side listeners of the session, in the order 
that they were added,
+/// which Apache Spark's tests read to remove them.
+#[pyclass(
+    name = "StreamingQueryListenerBus",
+    module = "pyspark.sql.streaming.query",
+    frozen
+)]
+pub struct PyStreamingQueryListenerBus {
+    manager: StreamingQueryManager,
+}
+
+#[pymethods]
+impl PyStreamingQueryListenerBus {
+    #[getter]
+    fn _listener_bus(&self, py: Python<'_>) -> Vec<Py<PyAny>> {
+        self.manager
+            .listeners()
+            .iter()
+            .filter_map(|listener| {
+                let adapter = 
listener.as_any()?.downcast_ref::<PyListenerAdapter>()?;
+                Some(adapter.listener.clone_ref(py))
+            })
+            .collect()
     }
 }
 
@@ -1161,31 +1208,64 @@ impl PyStreamingQueryManager {
     }
 
     /// Register a client-side listener object (with 
onQueryStarted/Progress/Idle/
-    /// Terminated callbacks). Mirrors `StreamingQueryManager.addListener`. 
The native
-    /// Rust bus streams events and dispatches to this listener; the assigned 
id is
-    /// stashed on the listener so `removeListener` can find it.
+    /// Terminated callbacks). Mirrors `StreamingQueryManager.addListener`: 
the listener gets the
+    /// session as its `spark`, and the native Rust bus of the session 
dispatches its events to
+    /// it; the assigned id is stashed on the listener so `removeListener` can 
find it. Should the
+    /// server fail to add its listener, this warns, as PySpark Connect's does.
     fn addListener(&self, py: Python<'_>, listener: Py<PyAny>) -> PyResult<()> 
{
+        let session = 
crate::session::PySparkSession::new(self.session.clone());
+        listener
+            .bind(py)
+            .call_method1("_set_spark_session", (session,))?;
         let adapter = Arc::new(PyListenerAdapter {
             listener: listener.clone_ref(py),
         });
-        let id =
-            crate::dataframe::detached(py, || 
self.inner.add_listener(adapter).map_err(Box::new))?;
-        listener.bind(py).setattr("_rust_listener_id", id)?;
-        Ok(())
+        match crate::dataframe::detached(py, || 
self.inner.add_listener(adapter).map_err(Box::new))
+        {
+            Ok(id) => listener.bind(py).setattr("_rust_listener_id", id),
+            Err(error) => warn(
+                py,
+                format!(
+                    "Failed to add the listener because of exception: {}\n\
+                     The listener is not added, please add it again.",
+                    error.value(py)
+                ),
+            ),
+        }
     }
 
     /// Remove a previously-added client-side listener object. Mirrors
-    /// `StreamingQueryManager.removeListener`.
+    /// `StreamingQueryManager.removeListener`: should the server fail to 
remove its listener,
+    /// this warns, and the listener stays, as PySpark Connect's does.
     fn removeListener(&self, py: Python<'_>, listener: Py<PyAny>) -> 
PyResult<()> {
         let bound = listener.bind(py);
         if let Ok(id_obj) = bound.getattr("_rust_listener_id") {
             let id: String = id_obj.extract()?;
-            crate::dataframe::detached(py, || 
self.inner.remove_listener(&id).map_err(Box::new))?;
+            if let Err(error) =
+                crate::dataframe::detached(py, || 
self.inner.remove_listener(&id).map_err(Box::new))
+            {
+                return warn(
+                    py,
+                    format!(
+                        "Failed to remove the listener because of exception: 
{}\n\
+                         The listener is not removed, please remove it again.",
+                        error.value(py)
+                    ),
+                );
+            }
             let _ = bound.delattr("_rust_listener_id");
         }
         Ok(())
     }
 
+    /// The listener bus of the session, as PySpark Connect's `_sqlb`.
+    #[getter]
+    fn _sqlb(&self) -> PyStreamingQueryListenerBus {
+        PyStreamingQueryListenerBus {
+            manager: self.session.streams(),
+        }
+    }
+
     /// Remove all client-side listeners and stop the dispatch thread. Mirrors
     /// `StreamingQueryManager.close`.
     fn close(&self, py: Python<'_>) -> PyResult<()> {
diff --git a/crates/spark-connect/src/session.rs 
b/crates/spark-connect/src/session.rs
index cbbbf63f..95b3eab4 100644
--- a/crates/spark-connect/src/session.rs
+++ b/crates/spark-connect/src/session.rs
@@ -96,6 +96,10 @@ pub struct SparkSession {
     stopped: Arc<AtomicBool>,
     /// Successful server version, shared by handles for this session only.
     server_version: Arc<Mutex<Option<String>>>,
+    /// The client-side listener bus of the streaming queries of this session 
(shared across
+    /// clones), which each of its `StreamingQueryManager`s adds to, removes 
from and posts to,
+    /// as the one `StreamingQueryListenerBus` of PySpark Connect's 
`spark.streams`.
+    listener_bus: Arc<crate::streaming::ListenerBus>,
 }
 
 impl SparkSession {
@@ -111,6 +115,7 @@ impl SparkSession {
             python_accumulators: PythonAccumulators::default(),
             stopped: Arc::new(AtomicBool::new(false)),
             server_version: Arc::new(Mutex::new(None)),
+            listener_bus: Arc::default(),
         }
     }
 
@@ -276,6 +281,7 @@ impl SparkSession {
             python_accumulators: PythonAccumulators::default(),
             stopped: Arc::new(AtomicBool::new(false)),
             server_version: Arc::new(Mutex::new(None)),
+            listener_bus: Arc::default(),
         }
     }
 
@@ -656,6 +662,11 @@ impl SparkSession {
         crate::streaming::StreamingQueryManager::new(self.clone())
     }
 
+    /// The client-side listener bus of the streaming queries of this session.
+    pub(crate) fn listener_bus(&self) -> Arc<crate::streaming::ListenerBus> {
+        Arc::clone(&self.listener_bus)
+    }
+
     /// Get the catalog for this session.
     ///
     /// Mirrors `pyspark.sql.SparkSession.catalog`.
@@ -1106,6 +1117,7 @@ impl Clone for SparkSession {
             python_accumulators: self.python_accumulators.clone(),
             stopped: Arc::clone(&self.stopped),
             server_version: Arc::clone(&self.server_version),
+            listener_bus: Arc::clone(&self.listener_bus),
         }
     }
 }
diff --git a/crates/spark-connect/src/streaming.rs 
b/crates/spark-connect/src/streaming.rs
index 9e17be0e..f8a72ec0 100644
--- a/crates/spark-connect/src/streaming.rs
+++ b/crates/spark-connect/src/streaming.rs
@@ -465,6 +465,7 @@ impl DataStreamWriter {
         let mut query_id = String::new();
         let mut run_id = String::new();
         let mut name = self.query_name.clone();
+        let mut started_event = None;
         for resp in responses {
             if let Some(
                 
proto::execute_plan_response::ResponseType::WriteStreamOperationStartResult(res),
@@ -477,6 +478,7 @@ impl DataStreamWriter {
                 if !res.name.is_empty() {
                     name = Some(res.name);
                 }
+                started_event = res.query_started_event_json;
             }
         }
         if query_id.is_empty() {
@@ -484,6 +486,17 @@ impl DataStreamWriter {
                 "writeStream.start: server returned no 
WriteStreamOperationStartResult",
             ));
         }
+        // As PySpark Connect's DataStreamWriter: the server sends the started 
event of the query
+        // with the result of its start, as the session has client-side 
listeners, and the client
+        // posts it to them before it gives back the query.
+        if let Some(event_json) = started_event {
+            self.session
+                .streams()
+                .post_to_all(&StreamingQueryListenerEvent {
+                    event_type: QUERY_STARTED_EVENT,
+                    event_json,
+                });
+        }
 
         Ok(StreamingQuery {
             session: self.session,
@@ -744,6 +757,46 @@ pub struct ListenerEventStream {
     done: bool,
 }
 
+impl ListenerEventStream {
+    /// Read the responses of the server until the one that tells that it 
added the listener of
+    /// the session (`listener_bus_listener_added`), keeping the events of 
them, which the
+    /// iterator gives first, as PySpark Connect's 
`_register_server_side_listener` waits for it.
+    fn wait_until_listener_added(&mut self) -> Result<()> {
+        loop {
+            match block_on(self.stream.message()) {
+                Ok(Some(resp)) => {
+                    if let Some(
+                        
proto::execute_plan_response::ResponseType::StreamingQueryListenerEventsResult(res),
+                    ) = resp.response_type
+                    {
+                        if !res.events.is_empty() {
+                            let mut events: Vec<(i32, String)> =
+                                std::mem::take(&mut 
self.buffered_events).collect();
+                            events.extend(
+                                res.events
+                                    .into_iter()
+                                    .map(|event| (event.event_type, 
event.event_json)),
+                            );
+                            self.buffered_events = events.into_iter();
+                        }
+                        if res.listener_bus_listener_added.is_some() {
+                            return Ok(());
+                        }
+                    }
+                }
+                Ok(None) => {
+                    self.done = true;
+                    return Ok(());
+                }
+                Err(e) => {
+                    self.done = true;
+                    return Err(e);
+                }
+            }
+        }
+    }
+}
+
 impl Iterator for ListenerEventStream {
     type Item = Result<(i32, String)>;
 
@@ -796,10 +849,18 @@ pub const QUERY_PROGRESS_EVENT: i32 = 1;
 pub const QUERY_TERMINATED_EVENT: i32 = 2;
 pub const QUERY_IDLE_EVENT: i32 = 3;
 
+/// The started event of a query. The server sends it with the result of the 
start of the query
+/// (`query_started_event_json`), when the session has a client-side listener, 
rather than on the
+/// listener bus, and the client posts it to its listeners as the query 
starts, as PySpark
+/// Connect's `DataStreamWriter.start` does: so it is not of the server's
+/// `StreamingQueryEventType`.
+pub const QUERY_STARTED_EVENT: i32 = -1;
+
 /// A streaming-query listener event delivered by the client-side listener bus.
 ///
 /// `event_json` is the server-provided JSON for the event; `event_type` is 
one of
-/// [`QUERY_PROGRESS_EVENT`], [`QUERY_TERMINATED_EVENT`], [`QUERY_IDLE_EVENT`].
+/// [`QUERY_STARTED_EVENT`], [`QUERY_PROGRESS_EVENT`], 
[`QUERY_TERMINATED_EVENT`],
+/// [`QUERY_IDLE_EVENT`].
 #[derive(Debug, Clone)]
 pub struct StreamingQueryListenerEvent {
     pub event_type: i32,
@@ -811,43 +872,64 @@ pub struct StreamingQueryListenerEvent {
 /// runs a background bus that streams events from the server and dispatches 
them here.
 pub trait StreamingQueryListener: Send + Sync {
     fn on_event(&self, event: &StreamingQueryListenerEvent);
+
+    /// This listener as `Any`, for a client that wraps its own listeners to 
find them among
+    /// [`StreamingQueryManager::listeners`]. `None` by default.
+    fn as_any(&self) -> Option<&dyn std::any::Any> {
+        None
+    }
+}
+
+/// The client-side listener bus of the streaming queries of a session, which 
all of its
+/// `StreamingQueryManager`s share, as PySpark Connect's 
`StreamingQueryListenerBus` of
+/// `spark.streams`: the registered listeners, in their order, and the thread 
that posts the
+/// events of the server to them, from the first listener to the removal of 
the last. `ops`
+/// serializes the additions and removals, which talk to the server, while the 
thread takes
+/// `state` only to read the listeners, and calls them without any lock.
+#[derive(Default)]
+pub(crate) struct ListenerBus {
+    ops: Mutex<()>,
+    state: Mutex<ListenerBusState>,
 }
 
-/// Shared state of the client-side listener bus: the registered listeners and 
the
-/// background dispatch thread (started with the first listener, stopped with 
the last).
 #[derive(Default)]
 struct ListenerBusState {
     listeners: Vec<(String, Arc<dyn StreamingQueryListener>)>,
     thread: Option<std::thread::JoinHandle<()>>,
 }
 
-/// The background dispatch loop: opens the server event stream and forwards 
each event
-/// to every currently-registered listener. Ends when the stream closes (the 
server was
-/// asked to stop via `RemoveListenerBusListener`) or all listeners are 
removed.
-fn run_listener_event_loop(session: SparkSession, bus: 
Arc<Mutex<ListenerBusState>>) {
-    let stream = match 
StreamingQueryManager::new(session).listener_event_stream() {
-        Ok(s) => s,
-        Err(_) => return,
-    };
+impl ListenerBus {
+    /// Post `event` to every listener, as PySpark Connect's `post_to_all`.
+    fn post_to_all(&self, event: &StreamingQueryListenerEvent) {
+        let listeners: Vec<Arc<dyn StreamingQueryListener>> = self
+            .state
+            .lock()
+            .unwrap()
+            .listeners
+            .iter()
+            .map(|(_, listener)| Arc::clone(listener))
+            .collect();
+        for listener in &listeners {
+            listener.on_event(event);
+        }
+    }
+}
+
+/// The background dispatch loop: posts each event of the stream of the 
server, on which the
+/// server has added the listener of the session, to the listeners. It ends 
when the stream does,
+/// as when the server was asked to remove its listener 
(`RemoveListenerBusListener`). An error
+/// of the stream removes all the listeners, as PySpark Connect's handler 
thread does.
+fn run_listener_event_loop(stream: ListenerEventStream, bus: Arc<ListenerBus>) 
{
     for item in stream {
-        let listeners: Vec<Arc<dyn StreamingQueryListener>> = {
-            let st = bus.lock().unwrap();
-            if st.listeners.is_empty() {
-                break;
-            }
-            st.listeners.iter().map(|(_, l)| l.clone()).collect()
-        };
         match item {
-            Ok((event_type, event_json)) => {
-                let ev = StreamingQueryListenerEvent {
-                    event_type,
-                    event_json,
-                };
-                for l in &listeners {
-                    l.on_event(&ev);
-                }
+            Ok((event_type, event_json)) => 
bus.post_to_all(&StreamingQueryListenerEvent {
+                event_type,
+                event_json,
+            }),
+            Err(_) => {
+                bus.state.lock().unwrap().listeners.clear();
+                return;
             }
-            Err(_) => break,
         }
     }
 }
@@ -855,81 +937,136 @@ fn run_listener_event_loop(session: SparkSession, bus: 
Arc<Mutex<ListenerBusStat
 /// Manager for active streaming queries.
 ///
 /// Mirrors `pyspark.sql.connect.streaming.StreamingQueryManager`, including a 
native
-/// client-side listener bus (so Rust clients get the listener feature too).
+/// client-side listener bus (so Rust clients get the listener feature too), 
which is that of
+/// its session.
 pub struct StreamingQueryManager {
     session: SparkSession,
-    bus: Arc<Mutex<ListenerBusState>>,
+    bus: Arc<ListenerBus>,
 }
 
 impl StreamingQueryManager {
-    /// Create a new StreamingQueryManager.
+    /// Create a new StreamingQueryManager, of the listener bus of `session`.
     pub(crate) fn new(session: SparkSession) -> Self {
-        StreamingQueryManager {
-            session,
-            bus: Arc::new(Mutex::new(ListenerBusState::default())),
-        }
+        let bus = session.listener_bus();
+        StreamingQueryManager { session, bus }
     }
 
     /// Register a client-side listener. Returns an id that can be passed to
-    /// [`StreamingQueryManager::remove_listener`]. Starts the background 
dispatch
-    /// thread when it is the first listener. Mirrors 
`StreamingQueryManager.addListener`.
+    /// [`StreamingQueryManager::remove_listener`]. Mirrors 
`StreamingQueryManager.addListener`:
+    /// for the first listener of the session, the server adds its listener 
before this returns,
+    /// so that the queries started then send their started events, and a 
thread posts the events
+    /// of the server to the listeners. Should the server fail to add it, the 
listener is not
+    /// added.
     pub fn add_listener(&self, listener: Arc<dyn StreamingQueryListener>) -> 
Result<String> {
         let id = uuid::Uuid::new_v4().to_string();
-        let mut st = self.bus.lock().unwrap();
-        st.listeners.push((id.clone(), listener));
-        if st.listeners.len() == 1 {
-            let session = self.session.clone();
-            let bus = self.bus.clone();
-            st.thread = Some(std::thread::spawn(move || {
-                run_listener_event_loop(session, bus);
-            }));
+        let _ops = self.bus.ops.lock().unwrap();
+        let first = {
+            let mut state = self.bus.state.lock().unwrap();
+            state.listeners.push((id.clone(), listener));
+            state.listeners.len() == 1
+        };
+        if first {
+            let registered = self.listener_event_stream().and_then(|mut 
stream| {
+                stream.wait_until_listener_added()?;
+                Ok(stream)
+            });
+            let stream = match registered {
+                Ok(stream) => stream,
+                Err(e) => {
+                    self.bus.state.lock().unwrap().listeners.clear();
+                    return Err(e);
+                }
+            };
+            let bus = Arc::clone(&self.bus);
+            let thread = std::thread::spawn(move || 
run_listener_event_loop(stream, bus));
+            self.bus.state.lock().unwrap().thread = Some(thread);
         }
         Ok(id)
     }
 
-    /// Remove a client-side listener by id. Stops the background dispatch 
thread when
-    /// the last listener is removed. Mirrors 
`StreamingQueryManager.removeListener`.
+    /// Remove a client-side listener by id, which does nothing for an id of 
no listener. Mirrors
+    /// `StreamingQueryManager.removeListener`: for the last listener, the 
server is asked to
+    /// remove its listener, which ends its stream, and this returns once the 
thread has posted
+    /// the events left. Should that request fail, the listener stays, and the 
error is returned.
     pub fn remove_listener(&self, id: &str) -> Result<()> {
-        let now_empty = {
-            let mut st = self.bus.lock().unwrap();
-            st.listeners.retain(|(lid, _)| lid != id);
-            st.listeners.is_empty()
-        };
-        if now_empty {
-            self.stop_listener_bus();
+        let _ops = self.bus.ops.lock().unwrap();
+        {
+            let mut state = self.bus.state.lock().unwrap();
+            let Some(index) = state.listeners.iter().position(|(lid, _)| lid 
== id) else {
+                return Ok(());
+            };
+            if state.listeners.len() > 1 {
+                state.listeners.remove(index);
+                return Ok(());
+            }
         }
+        self.remove_server_listener()?;
+        self.join_listener_thread();
+        self.bus
+            .state
+            .lock()
+            .unwrap()
+            .listeners
+            .retain(|(lid, _)| lid != id);
         Ok(())
     }
 
     /// Remove all client-side listeners and stop the dispatch thread. Mirrors
-    /// `StreamingQueryManager.close`.
+    /// `StreamingQueryManager.close`. Should the server fail to remove its 
listener, the thread
+    /// is left to end with its stream.
     pub fn close(&self) -> Result<()> {
-        let had_listeners = {
-            let mut st = self.bus.lock().unwrap();
-            let had = !st.listeners.is_empty();
-            st.listeners.clear();
-            had
-        };
-        if had_listeners {
-            self.stop_listener_bus();
+        let _ops = self.bus.ops.lock().unwrap();
+        if self.bus.state.lock().unwrap().listeners.is_empty() {
+            return Ok(());
+        }
+        if self.remove_server_listener().is_ok() {
+            self.join_listener_thread();
         }
+        let mut state = self.bus.state.lock().unwrap();
+        state.listeners.clear();
+        state.thread = None;
         Ok(())
     }
 
-    /// Ask the server to stop streaming listener events (which ends the 
background
-    /// thread's stream), then join the thread.
-    fn stop_listener_bus(&self) {
+    /// The client-side listeners of the session, in the order that they were 
added, as the
+    /// `_listener_bus` of PySpark Connect's `StreamingQueryListenerBus`.
+    pub fn listeners(&self) -> Vec<Arc<dyn StreamingQueryListener>> {
+        self.bus
+            .state
+            .lock()
+            .unwrap()
+            .listeners
+            .iter()
+            .map(|(_, listener)| Arc::clone(listener))
+            .collect()
+    }
+
+    /// Post `event` to the client-side listeners of the session.
+    pub(crate) fn post_to_all(&self, event: &StreamingQueryListenerEvent) {
+        self.bus.post_to_all(event);
+    }
+
+    /// Ask the server to remove the listener of the session, which ends its 
stream of events.
+    fn remove_server_listener(&self) -> Result<()> {
         let mut lb = proto::StreamingQueryListenerBusCommand::default();
         lb.command = Some(
             
proto::streaming_query_listener_bus_command::Command::RemoveListenerBusListener(true),
         );
-        let _ = crate::dataframe::execute_command_collect(
+        crate::dataframe::execute_command_collect(
             &self.session,
             proto::command::CommandType::StreamingQueryListenerBusCommand(lb),
-        );
-        let handle = self.bus.lock().unwrap().thread.take();
-        if let Some(h) = handle {
-            let _ = h.join();
+        )
+        .map(|_| ())
+    }
+
+    /// Wait for the dispatch thread to post the events left of its ended 
stream, unless this is
+    /// that thread, as when a listener removes the last listener.
+    fn join_listener_thread(&self) {
+        let thread = self.bus.state.lock().unwrap().thread.take();
+        if let Some(thread) = thread {
+            if thread.thread().id() != std::thread::current().id() {
+                let _ = thread.join();
+            }
         }
     }
 
diff --git a/crates/spark-connect/tests/e2e_streaming.rs 
b/crates/spark-connect/tests/e2e_streaming.rs
index b2283a7a..8092457b 100644
--- a/crates/spark-connect/tests/e2e_streaming.rs
+++ b/crates/spark-connect/tests/e2e_streaming.rs
@@ -168,3 +168,73 @@ fn streaming_native_listener_bus() {
         "expected at least one native listener event, got {got}"
     );
 }
+
+/// As PySpark Connect's `spark.streams`: the managers of a session share its 
listener bus, the
+/// started event of a query reaches the listeners before `start` returns, and 
a listener removed
+/// through another manager of the session gets no more events.
+#[test]
+fn streaming_listeners_are_those_of_the_session() {
+    if !should_run() {
+        return;
+    }
+    use spark_connect::streaming::{
+        StreamingQueryListener, StreamingQueryListenerEvent, 
QUERY_PROGRESS_EVENT,
+        QUERY_STARTED_EVENT,
+    };
+    use std::sync::{Arc, Mutex};
+    use std::time::Duration;
+
+    struct Recording {
+        events: Mutex<Vec<i32>>,
+    }
+    impl StreamingQueryListener for Recording {
+        fn on_event(&self, event: &StreamingQueryListenerEvent) {
+            self.events.lock().unwrap().push(event.event_type);
+        }
+    }
+
+    let spark = session();
+    let listener = Arc::new(Recording {
+        events: Mutex::new(Vec::new()),
+    });
+    let id = spark
+        .streams()
+        .add_listener(listener.clone())
+        .expect("add_listener");
+    assert_eq!(spark.streams().listeners().len(), 1);
+    let query = spark
+        .read_stream()
+        .format("rate")
+        .load(None)
+        .write_stream()
+        .format("noop")
+        .query_name("e2e_session_listener_bus")
+        .start("")
+        .expect("start streaming query");
+    assert_eq!(
+        listener.events.lock().unwrap().first(),
+        Some(&QUERY_STARTED_EVENT)
+    );
+    for _ in 0..60 {
+        if listener
+            .events
+            .lock()
+            .unwrap()
+            .contains(&QUERY_PROGRESS_EVENT)
+        {
+            break;
+        }
+        std::thread::sleep(Duration::from_millis(500));
+    }
+    spark
+        .streams()
+        .remove_listener(&id)
+        .expect("remove_listener");
+    assert!(spark.streams().listeners().is_empty());
+    let seen = listener.events.lock().unwrap().len();
+    std::thread::sleep(Duration::from_secs(2));
+    query.stop().expect("stop");
+    let events = listener.events.lock().unwrap();
+    assert!(events.contains(&QUERY_PROGRESS_EVENT), "{events:?}");
+    assert_eq!(events.len(), seen, "{events:?}");
+}
diff --git a/python/pyspark/sql/streaming/query.py 
b/python/pyspark/sql/streaming/query.py
index 919fcb83..c0ae0ae9 100644
--- a/python/pyspark/sql/streaming/query.py
+++ b/python/pyspark/sql/streaming/query.py
@@ -48,7 +48,10 @@ if TYPE_CHECKING:
     from pyspark.sql.session import SparkSession
 
 # StreamingQueryEventType values (spark.connect.StreamingQueryEventType); the 
Rust
-# listener stream yields these as the event_type int alongside each event's 
JSON.
+# listener stream yields these as the event_type int alongside each event's 
JSON. The started
+# event of a query comes with the result of its start, which the Rust core 
posts as
+# spark_connect::streaming::QUERY_STARTED_EVENT, of no StreamingQueryEventType.
+_QUERY_STARTED_EVENT = -1
 _QUERY_PROGRESS_EVENT = 1
 _QUERY_TERMINATED_EVENT = 2
 _QUERY_IDLE_EVENT = 3
@@ -61,20 +64,22 @@ def _dispatch_listener_event(listener, event_type, 
event_json):
     """Deserialize a listener event and invoke the matching callback on 
``listener``.
 
     Called from the native (Rust) listener bus for each event. Mirrors the 
reference
-    ``StreamingQueryListenerBus`` dispatch: build the typed event and route it 
to
-    onQueryProgress / onQueryIdle / onQueryTerminated. Callback exceptions are 
surfaced
-    as warnings so one listener cannot break the dispatch thread.
+    ``StreamingQueryListenerBus.post_to_all``: build the typed event and route 
it to
+    onQueryStarted / onQueryProgress / onQueryIdle / onQueryTerminated. 
Callback exceptions
+    are surfaced as warnings, as PySpark's, so one listener cannot break the 
dispatch thread.
     """
     try:
         j = json.loads(event_json)
-        if event_type == _QUERY_PROGRESS_EVENT:
+        if event_type == _QUERY_STARTED_EVENT:
+            listener.onQueryStarted(QueryStartedEvent.fromJson(j))
+        elif event_type == _QUERY_PROGRESS_EVENT:
             listener.onQueryProgress(QueryProgressEvent.fromJson(j))
         elif event_type == _QUERY_IDLE_EVENT:
             listener.onQueryIdle(QueryIdleEvent.fromJson(j))
         elif event_type == _QUERY_TERMINATED_EVENT:
             listener.onQueryTerminated(QueryTerminatedEvent.fromJson(j))
     except Exception as e:  # noqa: BLE001
-        warnings.warn(f"Listener callback raised exception: {e}")
+        warnings.warn(f"Listener {str(listener)} threw an exception\n{e}")
 
 
 # ``spark.streams`` returns the native (Rust) StreamingQueryManager, which 
owns the
diff --git a/python/tests/test_dropin_e2e.py b/python/tests/test_dropin_e2e.py
index 61d746fc..90ee3602 100644
--- a/python/tests/test_dropin_e2e.py
+++ b/python/tests/test_dropin_e2e.py
@@ -3071,6 +3071,57 @@ def 
test_dataframes_of_a_stream_are_streaming_as_pyspark(session):
     assert session.range(1).isStreaming is False
 
 
+def test_streaming_listeners_are_those_of_the_session_as_pyspark(session):
+    # As PySpark Connect's spark.streams: one manager, of the listener bus of 
the session, whose
+    # listeners get the session, are listed by _sqlb, get the started event of 
a query before
+    # start returns, and get no events once removed.
+    from pyspark.sql.streaming import StreamingQueryListener
+
+    class Listener(StreamingQueryListener):
+        def __init__(self):
+            self.events = []
+
+        def onQueryStarted(self, event):
+            self.events.append(("started", str(event.id)))
+
+        def onQueryProgress(self, event):
+            self.events.append(("progress", str(event.progress.id)))
+
+        def onQueryIdle(self, event):
+            pass
+
+        def onQueryTerminated(self, event):
+            self.events.append(("terminated", str(event.id)))
+
+    assert session.streams is session.streams
+    first, second = Listener(), Listener()
+    session.streams.addListener(first)
+    session.streams.addListener(second)
+    query = None
+    try:
+        assert first.spark.range(3).count() == 3
+        assert session.streams._sqlb._listener_bus == [first, second]
+        query = 
session.readStream.format("rate").load().writeStream.format("noop").start()
+        assert first.events[:1] == second.events[:1] == [("started", query.id)]
+        deadline = time.time() + 60
+        while ("progress", query.id) not in first.events and time.time() < 
deadline:
+            time.sleep(0.5)
+        session.streams.removeListener(first)
+        assert session.streams._sqlb._listener_bus == [second]
+        seen = list(first.events)
+        query.stop()
+        deadline = time.time() + 30
+        while ("terminated", query.id) not in second.events and time.time() < 
deadline:
+            time.sleep(0.5)
+        assert ("progress", query.id) in seen and first.events == seen
+        assert ("terminated", query.id) in second.events
+    finally:
+        for listener in session.streams._sqlb._listener_bus:
+            session.streams.removeListener(listener)
+        if query is not None:
+            query.stop()
+
+
 def 
test_streaming_queries_return_the_status_and_the_exception_of_pyspark(session):
     # As PySpark's: the status of a query is a dict, and the exception of a 
failed query is
     # the StreamingQueryException of pyspark.errors, of the message of the 
server and its JVM


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to