This is an automated email from the ASF dual-hosted git repository.

HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git


The following commit(s) were added to refs/heads/master by this push:
     new 5efdd8a5 [SPARK-61208][CONNECT] Return from removeListener only after 
the event in delivery reached its listener, so that a removed listener gets no 
more events, in the Rust and Python clients as PySpark does
5efdd8a5 is described below

commit 5efdd8a5175c8148ea4d7850193f2100e884c239
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Mon Nov 9 12:15:43 2026 +0900

    [SPARK-61208][CONNECT] Return from removeListener only after the event in 
delivery reached its listener, so that a removed listener gets no more events, 
in the Rust and Python clients as PySpark does
    
    ### What changes were proposed in this pull request?
    
    [SPARK-61208](https://issues.apache.org/jira/browse/SPARK-61208): PySpark's 
`StreamingQueryListenerBus` delivers each event to all its listeners in 
`post_to_all`, while it holds `_lock`, and `remove` takes the same lock. So 
`removeListener` returns only after the event in delivery has reached every 
listener, also the removed one, and the removed listener gets no event after 
that.
    
    The bus of the Rust client (`crates/spark-connect/src/streaming.rs`) takes 
a snapshot of its listeners for each event and delivers it without its lock, 
and it removes a listener that is not the last at once. Now:
    
    - The worker records which generation it is delivering an event for 
(`ListenerBusState::dispatching`), and a guard clears that when the delivery 
ends, also on a panic, and wakes the waiters.
    - A removal of a listener that is not the last, from another thread than 
the worker, waits for the delivery in progress before it returns, as PySpark's 
lock makes it wait.
    - A removal from a callback of the bus does not wait, as it would wait for 
its own delivery. PySpark's `_lock` is not reentrant, so PySpark hangs there.
    - The removal of the last listener, and `close`, already wait for the 
worker to end.
    
    ### Why are the changes needed?
    
    When a listener was removed while another listener's callback ran, the 
removed listener got the event after `removeListener` had returned. Listener A 
is busy in its first progress callback, and another thread removes listener B:
    
    ```
    official 4.3.0: A progress (busy), B progress, removeListener(B) returned
    before:         A progress (busy), removeListener(B) returned, B progress
    after:          A progress (busy), B progress, removeListener(B) returned
    ```
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes. `removeListener` of a listener that is not the last, called while an 
event is being delivered, returns after the delivery, so the removed listener 
gets no event after it returns. The call can now wait for a callback of another 
listener, as it does in PySpark.
    
    ### How was this patch tested?
    
    - `test_streaming_listener_removal_waits_for_the_event_in_delivery` in 
`python/tests/streaming_listener_cases.py`, against the mock server's listener 
bus:
      - The first listener blocks in its callback for an event, and another 
thread removes the second listener.
      - The removal does not return while the first listener blocks.
      - Once the first listener is released, the second listener gets the event 
and then the removal returns.
      - A later event reaches only the first listener.
      - The test fails without the change, where the removal returns at once.
    - The probe above, against a Spark 4.3.0 Connect server: the order of this 
build is the order of official PySpark 4.3.0.
    - The offline streaming listener tests (`-k StreamingListener`, 28 passed).
    - The lib tests of both crates (636 passed), the build of all tests, also 
with `wasm-udf`, the offline Python suites (2610 passed, 8 skipped), and the 
live drop-in and Arrow suites (223 passed) and live cases (12 passed) against a 
local Spark 4.3.0 server. Clippy reports no diagnostics in the changed lines.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Opus 5.5
    
    This pull request and its description were written by Isaac.
    
    Closes #394 from HyukjinKwon/SPARK-61208-listener-in-flight-event.
    
    Authored-by: Hyukjin Kwon <[email protected]>
    Signed-off-by: Hyukjin Kwon <[email protected]>
---
 crates/spark-connect/src/streaming.rs    | 44 +++++++++++++++++++++++++++--
 python/tests/streaming_listener_cases.py | 48 ++++++++++++++++++++++++++++++++
 2 files changed, 89 insertions(+), 3 deletions(-)

diff --git a/crates/spark-connect/src/streaming.rs 
b/crates/spark-connect/src/streaming.rs
index 20f45903..a736b0be 100644
--- a/crates/spark-connect/src/streaming.rs
+++ b/crates/spark-connect/src/streaming.rs
@@ -934,6 +934,28 @@ struct ListenerBusState {
     phase: ListenerBusPhase,
     worker_id: Option<std::thread::ThreadId>,
     generation: u64,
+    /// The generation whose worker is delivering an event to its snapshot of 
the listeners.
+    /// PySpark's bus holds its lock while it delivers an event, so a removal 
from another
+    /// thread waits for it.
+    dispatching: Option<u64>,
+}
+
+/// The delivery of one event by the worker of `generation`, until it is 
dropped.
+struct Dispatch<'a> {
+    bus: &'a Weak<ListenerBus>,
+    generation: u64,
+}
+
+impl Drop for Dispatch<'_> {
+    fn drop(&mut self) {
+        if let Some(bus) = self.bus.upgrade() {
+            let mut state = bus.state.lock().unwrap();
+            if state.dispatching == Some(self.generation) {
+                state.dispatching = None;
+                bus.changed.notify_all();
+            }
+        }
+    }
 }
 
 #[derive(Debug, Default, PartialEq, Eq)]
@@ -981,21 +1003,26 @@ fn run_listener_event_loop(
         let Ok((event_type, event_json)) = item else {
             break;
         };
-        let Some(bus) = bus.upgrade() else {
+        let Some(owner) = bus.upgrade() else {
             break;
         };
         let listeners: Vec<Arc<dyn StreamingQueryListener>> = {
-            let state = bus.state.lock().unwrap();
+            let mut state = owner.state.lock().unwrap();
             if state.generation != generation || state.listeners.is_empty() {
                 break;
             }
+            state.dispatching = Some(generation);
             state
                 .listeners
                 .iter()
                 .map(|(_, listener)| Arc::clone(listener))
                 .collect()
         };
-        drop(bus);
+        drop(owner);
+        let dispatch = Dispatch {
+            bus: &bus,
+            generation,
+        };
         let event = StreamingQueryListenerEvent {
             event_type,
             event_json,
@@ -1003,6 +1030,7 @@ fn run_listener_event_loop(
         for listener in listeners {
             listener.on_event(&event);
         }
+        drop(dispatch);
     }
     if let Some(bus) = bus.upgrade() {
         bus.finish_generation(generation);
@@ -1148,6 +1176,16 @@ impl StreamingQueryManager {
             };
             if state.listeners.len() > 1 {
                 state.listeners.remove(index);
+                // As PySpark's bus, whose delivery of an event holds the lock 
of its removal:
+                // the removal returns once the event in delivery has been 
delivered, also to
+                // the removed listener, which gets no events after it. A 
callback of the bus
+                // does not wait, as it would wait for its own delivery.
+                if state.worker_id != Some(std::thread::current().id()) {
+                    let generation = state.generation;
+                    while state.dispatching == Some(generation) {
+                        state = self.bus.changed.wait(state).unwrap();
+                    }
+                }
                 return Ok(true);
             }
         }
diff --git a/python/tests/streaming_listener_cases.py 
b/python/tests/streaming_listener_cases.py
index a5750312..88cf3d47 100644
--- a/python/tests/streaming_listener_cases.py
+++ b/python/tests/streaming_listener_cases.py
@@ -345,6 +345,54 @@ class TestStreamingListenerLifecycle:
         assert service.release_session_calls == 0
 
 
+    def test_streaming_listener_removal_waits_for_the_event_in_delivery(self, 
mock_connect):
+        # As PySpark's bus, whose delivery of an event holds the lock of its 
removal: a
+        # listener removed by another thread while an event is delivered gets 
that event
+        # before removeListener returns, and no event after it.
+        spark, service = mock_connect
+        service.listener_bus_enabled = True
+        order = []
+        entered, release = threading.Event(), threading.Event()
+
+        class Blocking(IdleListener):
+            def onQueryIdle(self, event):
+                order.append("first " + event.timestamp)
+                entered.set()
+                assert release.wait(5)
+                super().onQueryIdle(event)
+
+        class Recording(IdleListener):
+            def onQueryIdle(self, event):
+                order.append("second " + event.timestamp)
+                super().onQueryIdle(event)
+
+        def remove():
+            spark.streams.removeListener(second)
+            order.append("removed")
+
+        first, second = Blocking(), Recording()
+        try:
+            spark.streams.addListener(first)
+            spark.streams.addListener(second)
+            emit_idle(service, spark, "in-delivery")
+            assert entered.wait(5)
+            removal = threading.Thread(target=remove)
+            removal.start()
+            # The first listener holds the delivery, so the removal waits for 
it.
+            removal.join(0.5)
+            assert removal.is_alive()
+            release.set()
+            removal.join(5)
+            assert not removal.is_alive()
+            assert order == ["first in-delivery", "second in-delivery", 
"removed"]
+            emit_idle(service, spark, "after")
+            first.wait_for("after")
+            assert second.timestamps == ["in-delivery"]
+        finally:
+            release.set()
+            spark.streams.close()
+
+
 def concurrent_close_case(remove_failure, callback_action):
     from _mock_connect_server import mock_connect_session
 


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to