This is an automated email from the ASF dual-hosted git repository.

HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git


The following commit(s) were added to refs/heads/master by this push:
     new ef43037d [SPARK-61185][CONNECT][PYTHON] Remove each registration of a 
streaming query listener exactly once, also of a listener added twice or 
removed through another session, in the Python client as PySpark does
ef43037d is described below

commit ef43037ddc63177324805e342bd588f8c347d69e
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Sun Nov 8 18:55:29 2026 +0900

    [SPARK-61185][CONNECT][PYTHON] Remove each registration of a streaming 
query listener exactly once, also of a listener added twice or removed through 
another session, in the Python client as PySpark does
    
    ### What changes were proposed in this pull request?
    
    [SPARK-61185](https://issues.apache.org/jira/browse/SPARK-61185): Remove 
each registration of a streaming query listener exactly once, also of a 
listener added twice or removed through another session, in the Python client 
as PySpark does.
    
    The Python client kept a single `_rust_listener_id` attribute on the 
listener object, so it could hold only one registration of a listener:
    
    - `addListener` of the same listener object twice overwrote the id. The two 
`removeListener` calls then removed the same entry of the bus, and one entry 
stayed on the bus, which kept delivering query events to the listener.
    - `removeListener` deleted the id even when the bus of the session did not 
have the listener, for example when the listener was added through 
`spark.streams` and removed through `spark2.streams`. The later 
`spark.streams.removeListener` then found no id and did nothing.
    
    Now each `addListener` appends its id to the listener 
(`_rust_listener_ids`), and `removeListener` removes the first registration 
that the bus of the session has, as the bus removes the first occurrence in 
PySpark. The core `StreamingQueryManager` gets `remove_listener_found`, which 
reports whether the bus had the id; `remove_listener` keeps its signature and 
delegates to it. The Python client forgets an id only when the bus removed it.
    
    Official PySpark 4.3.0 behaves the same on both cases: the bus is empty 
after the removals, and no event reaches the listener after the removal.
    
    ### Why are the changes needed?
    
    A listener that the user removed kept receiving the events of the streaming 
queries. The repro, add twice and remove twice:
    
    ```python
    l = L()
    spark.streams.addListener(l)
    spark.streams.addListener(l)
    spark.streams.removeListener(l)
    spark.streams.removeListener(l)
    # ours: len(spark.streams._sqlb._listener_bus) == 1, and the query events 
still reach l
    # official 4.3.0: 0, and no event reaches l
    ```
    
    The cross-session case, `spark.streams.addListener(l)`, 
`spark2.streams.removeListener(l)`, `spark.streams.removeListener(l)`, leaves 
one entry in the bus of `spark` and delivers the events, while official 4.3.0 
leaves none and delivers none.
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes. `removeListener` now removes a listener added twice once per call, and 
a removal through another session no longer forgets the listener of the 
session. A new Rust method, `StreamingQueryManager::remove_listener_found`, 
reports whether the bus had the listener.
    
    ### How was this patch tested?
    
    - `test_streaming_listeners_are_removed_once_per_registration_as_pyspark` 
(live, `test_dropin_e2e.py`): add twice and remove twice leaves an empty bus; a 
removal through another session leaves the listener in this bus, and the next 
removal empties it; after the removals a query delivers no events. Official 
PySpark 4.3.0 gives the same results for the same probes (`probe_dup`: 2 adds, 
1 remove, 0 after the second removal, 0 events; `probe_xsess`: 1 entry after 
the removal through `spar [...]
    - `remove_listener_of_an_id_the_bus_does_not_have_is_not_found` (Rust unit 
test): an id that the bus does not have is not found and is a no-op.
    - The lib tests of both crates (600 passed), the build of all tests, also 
with `wasm-udf`, and the offline Python suites (2227 passed, 8 skipped), and 
the live tests of streaming queries and listeners (15 passed) against a local 
Spark 4.3.0 server; rustfmt passes, and clippy reports no diagnostics in the 
changed lines.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Opus 5.5
    
    This pull request and its description were written by Isaac.
    
    Closes #363 from HyukjinKwon/SPARK-61185-listener-registrations-once.
    
    Authored-by: Hyukjin Kwon <[email protected]>
    Signed-off-by: Hyukjin Kwon <[email protected]>
---
 crates/pyspark-rs/src/streaming.rs    | 59 ++++++++++++++++++++++++-----------
 crates/spark-connect/src/streaming.rs | 21 +++++++++++--
 python/tests/test_dropin_e2e.py       | 56 +++++++++++++++++++++++++++++++++
 3 files changed, 114 insertions(+), 22 deletions(-)

diff --git a/crates/pyspark-rs/src/streaming.rs 
b/crates/pyspark-rs/src/streaming.rs
index 42ecba13..58a3c9c4 100644
--- a/crates/pyspark-rs/src/streaming.rs
+++ b/crates/pyspark-rs/src/streaming.rs
@@ -1241,8 +1241,9 @@ impl PyStreamingQueryManager {
     /// Register a client-side listener object (with 
onQueryStarted/Progress/Idle/
     /// Terminated callbacks). Mirrors `StreamingQueryManager.addListener`: 
the listener gets the
     /// session as its `spark`, and the native Rust bus of the session 
dispatches its events to
-    /// it; the assigned id is stashed on the listener so `removeListener` can 
find it. Should the
-    /// server fail to add its listener, this warns, as PySpark Connect's does.
+    /// it. The id of each registration is appended to the listener's 
`_rust_listener_ids`, as a
+    /// listener added twice is in the bus twice. Should the server fail to 
add its listener, this
+    /// warns, as PySpark Connect's does.
     fn addListener(&self, py: Python<'_>, listener: Py<PyAny>) -> PyResult<()> 
{
         let session = 
crate::session::PySparkSession::new(self.session.clone());
         listener
@@ -1253,7 +1254,13 @@ impl PyStreamingQueryManager {
         });
         match crate::dataframe::detached(py, || 
self.inner.add_listener(adapter).map_err(Box::new))
         {
-            Ok(id) => listener.bind(py).setattr("_rust_listener_id", id),
+            Ok(id) => {
+                let bound = listener.bind(py);
+                match bound.getattr("_rust_listener_ids") {
+                    Ok(ids) => ids.call_method1("append", (id,)).map(|_| ()),
+                    Err(_) => bound.setattr("_rust_listener_ids", vec![id]),
+                }
+            }
             Err(error) => warn(
                 py,
                 format!(
@@ -1266,25 +1273,39 @@ impl PyStreamingQueryManager {
     }
 
     /// Remove a previously-added client-side listener object. Mirrors
-    /// `StreamingQueryManager.removeListener`: should the server fail to 
remove its listener,
-    /// this warns, and the listener stays, as PySpark Connect's does.
+    /// `StreamingQueryManager.removeListener`: one registration of the 
listener is removed from
+    /// the bus of the session, the first one that the bus has (as the bus 
removes the first
+    /// occurrence), and the others, which are of other buses, stay. Should 
the server fail to
+    /// remove its listener, this warns, and the listener stays, as PySpark 
Connect's does.
     fn removeListener(&self, py: Python<'_>, listener: Py<PyAny>) -> 
PyResult<()> {
         let bound = listener.bind(py);
-        if let Ok(id_obj) = bound.getattr("_rust_listener_id") {
-            let id: String = id_obj.extract()?;
-            if let Err(error) =
-                crate::dataframe::detached(py, || 
self.inner.remove_listener(&id).map_err(Box::new))
-            {
-                return warn(
-                    py,
-                    format!(
-                        "Failed to remove the listener because of exception: 
{}\n\
-                         The listener is not removed, please remove it again.",
-                        error.value(py)
-                    ),
-                );
+        let Ok(ids) = bound.getattr("_rust_listener_ids") else {
+            return Ok(());
+        };
+        let registered: Vec<String> = ids.extract()?;
+        for id in registered {
+            match crate::dataframe::detached(py, || {
+                self.inner.remove_listener_found(&id).map_err(Box::new)
+            }) {
+                Ok(false) => continue,
+                Ok(true) => {
+                    ids.call_method1("remove", (&id,))?;
+                    if ids.len()? == 0 {
+                        let _ = bound.delattr("_rust_listener_ids");
+                    }
+                    return Ok(());
+                }
+                Err(error) => {
+                    return warn(
+                        py,
+                        format!(
+                            "Failed to remove the listener because of 
exception: {}\n\
+                             The listener is not removed, please remove it 
again.",
+                            error.value(py)
+                        ),
+                    );
+                }
             }
-            let _ = bound.delattr("_rust_listener_id");
         }
         Ok(())
     }
diff --git a/crates/spark-connect/src/streaming.rs 
b/crates/spark-connect/src/streaming.rs
index f8a72ec0..77ee15e2 100644
--- a/crates/spark-connect/src/streaming.rs
+++ b/crates/spark-connect/src/streaming.rs
@@ -989,15 +989,22 @@ impl StreamingQueryManager {
     /// remove its listener, which ends its stream, and this returns once the 
thread has posted
     /// the events left. Should that request fail, the listener stays, and the 
error is returned.
     pub fn remove_listener(&self, id: &str) -> Result<()> {
+        self.remove_listener_found(id).map(|_| ())
+    }
+
+    /// As [`remove_listener`](Self::remove_listener), which returns whether 
the listener of `id`
+    /// was in this bus (and so is removed now), for a client that keeps the 
ids of its listeners
+    /// per bus, as the ids of another session's bus are not in this one.
+    pub fn remove_listener_found(&self, id: &str) -> Result<bool> {
         let _ops = self.bus.ops.lock().unwrap();
         {
             let mut state = self.bus.state.lock().unwrap();
             let Some(index) = state.listeners.iter().position(|(lid, _)| lid 
== id) else {
-                return Ok(());
+                return Ok(false);
             };
             if state.listeners.len() > 1 {
                 state.listeners.remove(index);
-                return Ok(());
+                return Ok(true);
             }
         }
         self.remove_server_listener()?;
@@ -1008,7 +1015,7 @@ impl StreamingQueryManager {
             .unwrap()
             .listeners
             .retain(|(lid, _)| lid != id);
-        Ok(())
+        Ok(true)
     }
 
     /// Remove all client-side listeners and stop the dispatch thread. Mirrors
@@ -1471,4 +1478,12 @@ mod tests {
         let writer = writer.options(opts);
         assert_eq!(writer.options.get("key2"), Some(&"val2".to_string()));
     }
+
+    #[test]
+    fn remove_listener_of_an_id_the_bus_does_not_have_is_not_found() {
+        let spark = session();
+        let streams = spark.streams();
+        assert!(!streams.remove_listener_found("no-such-id").unwrap());
+        streams.remove_listener("no-such-id").unwrap();
+    }
 }
diff --git a/python/tests/test_dropin_e2e.py b/python/tests/test_dropin_e2e.py
index 67a4fed7..157284eb 100644
--- a/python/tests/test_dropin_e2e.py
+++ b/python/tests/test_dropin_e2e.py
@@ -1635,6 +1635,62 @@ def 
test_writer_v2_takes_its_arguments_as_pyspark(session):
         session.sql(f"DROP TABLE IF EXISTS {name}")
 
 
+def 
test_streaming_listeners_are_removed_once_per_registration_as_pyspark(session):
+    # As PySpark's StreamingQueryManager: a listener added twice is in the bus 
twice, and each
+    # removeListener removes one of its registrations; a removal through 
another session, whose
+    # bus does not have the listener, leaves it in this bus. Once removed, it 
gets no events.
+    from pyspark.sql.streaming import StreamingQueryListener
+
+    class Listener(StreamingQueryListener):
+        def __init__(self):
+            self.events = []
+
+        def onQueryStarted(self, event):
+            self.events.append("started")
+
+        def onQueryProgress(self, event):
+            self.events.append("progress")
+
+        def onQueryIdle(self, event):
+            pass
+
+        def onQueryTerminated(self, event):
+            self.events.append("terminated")
+
+    listener = Listener()
+    other = session.newSession()
+    try:
+        session.streams.addListener(listener)
+        session.streams.addListener(listener)
+        assert session.streams._sqlb._listener_bus == [listener, listener]
+        session.streams.removeListener(listener)
+        assert session.streams._sqlb._listener_bus == [listener]
+        session.streams.removeListener(listener)
+        assert session.streams._sqlb._listener_bus == []
+        session.streams.removeListener(listener)
+        assert session.streams._sqlb._listener_bus == []
+
+        session.streams.addListener(listener)
+        other.streams.removeListener(listener)
+        assert session.streams._sqlb._listener_bus == [listener]
+        assert other.streams._sqlb._listener_bus == []
+        session.streams.removeListener(listener)
+        assert session.streams._sqlb._listener_bus == []
+
+        query = 
session.readStream.format("rate").load().writeStream.format("noop")
+        query = query.trigger(processingTime="200 milliseconds").start()
+        try:
+            time.sleep(2)
+        finally:
+            query.stop()
+        time.sleep(1)
+        assert listener.events == []
+    finally:
+        for registered in list(session.streams._sqlb._listener_bus):
+            session.streams.removeListener(registered)
+        other.stop()
+
+
 def 
test_collected_rows_take_the_names_of_the_schema_of_the_result_as_pyspark(session):
     # As PySpark Connect's: the fields of nested structs have the names of the 
schema, which keep
     # duplicate names, and collect raises for a calendar interval.


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to