This is an automated email from the ASF dual-hosted git repository.
HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git
The following commit(s) were added to refs/heads/master by this push:
new ef43037d [SPARK-61185][CONNECT][PYTHON] Remove each registration of a
streaming query listener exactly once, also of a listener added twice or
removed through another session, in the Python client as PySpark does
ef43037d is described below
commit ef43037ddc63177324805e342bd588f8c347d69e
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Sun Nov 8 18:55:29 2026 +0900
[SPARK-61185][CONNECT][PYTHON] Remove each registration of a streaming
query listener exactly once, also of a listener added twice or removed through
another session, in the Python client as PySpark does
### What changes were proposed in this pull request?
[SPARK-61185](https://issues.apache.org/jira/browse/SPARK-61185): Remove
each registration of a streaming query listener exactly once, also of a
listener added twice or removed through another session, in the Python client
as PySpark does.
The Python client kept a single `_rust_listener_id` attribute on the
listener object, so it could hold only one registration of a listener:
- `addListener` of the same listener object twice overwrote the id. The two
`removeListener` calls then removed the same entry of the bus, and one entry
stayed on the bus, which kept delivering query events to the listener.
- `removeListener` deleted the id even when the bus of the session did not
have the listener, for example when the listener was added through
`spark.streams` and removed through `spark2.streams`. The later
`spark.streams.removeListener` then found no id and did nothing.
Now each `addListener` appends its id to the listener
(`_rust_listener_ids`), and `removeListener` removes the first registration
that the bus of the session has, as the bus removes the first occurrence in
PySpark. The core `StreamingQueryManager` gets `remove_listener_found`, which
reports whether the bus had the id; `remove_listener` keeps its signature and
delegates to it. The Python client forgets an id only when the bus removed it.
Official PySpark 4.3.0 behaves the same on both cases: the bus is empty
after the removals, and no event reaches the listener after the removal.
### Why are the changes needed?
A listener that the user removed kept receiving the events of the streaming
queries. The repro, add twice and remove twice:
```python
l = L()
spark.streams.addListener(l)
spark.streams.addListener(l)
spark.streams.removeListener(l)
spark.streams.removeListener(l)
# ours: len(spark.streams._sqlb._listener_bus) == 1, and the query events
still reach l
# official 4.3.0: 0, and no event reaches l
```
The cross-session case, `spark.streams.addListener(l)`,
`spark2.streams.removeListener(l)`, `spark.streams.removeListener(l)`, leaves
one entry in the bus of `spark` and delivers the events, while official 4.3.0
leaves none and delivers none.
### Does this PR introduce _any_ user-facing change?
Yes. `removeListener` now removes a listener added twice once per call, and
a removal through another session no longer forgets the listener of the
session. A new Rust method, `StreamingQueryManager::remove_listener_found`,
reports whether the bus had the listener.
### How was this patch tested?
- `test_streaming_listeners_are_removed_once_per_registration_as_pyspark`
(live, `test_dropin_e2e.py`): add twice and remove twice leaves an empty bus; a
removal through another session leaves the listener in this bus, and the next
removal empties it; after the removals a query delivers no events. Official
PySpark 4.3.0 gives the same results for the same probes (`probe_dup`: 2 adds,
1 remove, 0 after the second removal, 0 events; `probe_xsess`: 1 entry after
the removal through `spar [...]
- `remove_listener_of_an_id_the_bus_does_not_have_is_not_found` (Rust unit
test): an id that the bus does not have is not found and is a no-op.
- The lib tests of both crates (600 passed), the build of all tests, also
with `wasm-udf`, and the offline Python suites (2227 passed, 8 skipped), and
the live tests of streaming queries and listeners (15 passed) against a local
Spark 4.3.0 server; rustfmt passes, and clippy reports no diagnostics in the
changed lines.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
This pull request and its description were written by Isaac.
Closes #363 from HyukjinKwon/SPARK-61185-listener-registrations-once.
Authored-by: Hyukjin Kwon <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
---
crates/pyspark-rs/src/streaming.rs | 59 ++++++++++++++++++++++++-----------
crates/spark-connect/src/streaming.rs | 21 +++++++++++--
python/tests/test_dropin_e2e.py | 56 +++++++++++++++++++++++++++++++++
3 files changed, 114 insertions(+), 22 deletions(-)
diff --git a/crates/pyspark-rs/src/streaming.rs
b/crates/pyspark-rs/src/streaming.rs
index 42ecba13..58a3c9c4 100644
--- a/crates/pyspark-rs/src/streaming.rs
+++ b/crates/pyspark-rs/src/streaming.rs
@@ -1241,8 +1241,9 @@ impl PyStreamingQueryManager {
/// Register a client-side listener object (with
onQueryStarted/Progress/Idle/
/// Terminated callbacks). Mirrors `StreamingQueryManager.addListener`:
the listener gets the
/// session as its `spark`, and the native Rust bus of the session
dispatches its events to
- /// it; the assigned id is stashed on the listener so `removeListener` can
find it. Should the
- /// server fail to add its listener, this warns, as PySpark Connect's does.
+ /// it. The id of each registration is appended to the listener's
`_rust_listener_ids`, as a
+ /// listener added twice is in the bus twice. Should the server fail to
add its listener, this
+ /// warns, as PySpark Connect's does.
fn addListener(&self, py: Python<'_>, listener: Py<PyAny>) -> PyResult<()>
{
let session =
crate::session::PySparkSession::new(self.session.clone());
listener
@@ -1253,7 +1254,13 @@ impl PyStreamingQueryManager {
});
match crate::dataframe::detached(py, ||
self.inner.add_listener(adapter).map_err(Box::new))
{
- Ok(id) => listener.bind(py).setattr("_rust_listener_id", id),
+ Ok(id) => {
+ let bound = listener.bind(py);
+ match bound.getattr("_rust_listener_ids") {
+ Ok(ids) => ids.call_method1("append", (id,)).map(|_| ()),
+ Err(_) => bound.setattr("_rust_listener_ids", vec![id]),
+ }
+ }
Err(error) => warn(
py,
format!(
@@ -1266,25 +1273,39 @@ impl PyStreamingQueryManager {
}
/// Remove a previously-added client-side listener object. Mirrors
- /// `StreamingQueryManager.removeListener`: should the server fail to
remove its listener,
- /// this warns, and the listener stays, as PySpark Connect's does.
+ /// `StreamingQueryManager.removeListener`: one registration of the
listener is removed from
+ /// the bus of the session, the first one that the bus has (as the bus
removes the first
+ /// occurrence), and the others, which are of other buses, stay. Should
the server fail to
+ /// remove its listener, this warns, and the listener stays, as PySpark
Connect's does.
fn removeListener(&self, py: Python<'_>, listener: Py<PyAny>) ->
PyResult<()> {
let bound = listener.bind(py);
- if let Ok(id_obj) = bound.getattr("_rust_listener_id") {
- let id: String = id_obj.extract()?;
- if let Err(error) =
- crate::dataframe::detached(py, ||
self.inner.remove_listener(&id).map_err(Box::new))
- {
- return warn(
- py,
- format!(
- "Failed to remove the listener because of exception:
{}\n\
- The listener is not removed, please remove it again.",
- error.value(py)
- ),
- );
+ let Ok(ids) = bound.getattr("_rust_listener_ids") else {
+ return Ok(());
+ };
+ let registered: Vec<String> = ids.extract()?;
+ for id in registered {
+ match crate::dataframe::detached(py, || {
+ self.inner.remove_listener_found(&id).map_err(Box::new)
+ }) {
+ Ok(false) => continue,
+ Ok(true) => {
+ ids.call_method1("remove", (&id,))?;
+ if ids.len()? == 0 {
+ let _ = bound.delattr("_rust_listener_ids");
+ }
+ return Ok(());
+ }
+ Err(error) => {
+ return warn(
+ py,
+ format!(
+ "Failed to remove the listener because of
exception: {}\n\
+ The listener is not removed, please remove it
again.",
+ error.value(py)
+ ),
+ );
+ }
}
- let _ = bound.delattr("_rust_listener_id");
}
Ok(())
}
diff --git a/crates/spark-connect/src/streaming.rs
b/crates/spark-connect/src/streaming.rs
index f8a72ec0..77ee15e2 100644
--- a/crates/spark-connect/src/streaming.rs
+++ b/crates/spark-connect/src/streaming.rs
@@ -989,15 +989,22 @@ impl StreamingQueryManager {
/// remove its listener, which ends its stream, and this returns once the
thread has posted
/// the events left. Should that request fail, the listener stays, and the
error is returned.
pub fn remove_listener(&self, id: &str) -> Result<()> {
+ self.remove_listener_found(id).map(|_| ())
+ }
+
+ /// As [`remove_listener`](Self::remove_listener), which returns whether
the listener of `id`
+ /// was in this bus (and so is removed now), for a client that keeps the
ids of its listeners
+ /// per bus, as the ids of another session's bus are not in this one.
+ pub fn remove_listener_found(&self, id: &str) -> Result<bool> {
let _ops = self.bus.ops.lock().unwrap();
{
let mut state = self.bus.state.lock().unwrap();
let Some(index) = state.listeners.iter().position(|(lid, _)| lid
== id) else {
- return Ok(());
+ return Ok(false);
};
if state.listeners.len() > 1 {
state.listeners.remove(index);
- return Ok(());
+ return Ok(true);
}
}
self.remove_server_listener()?;
@@ -1008,7 +1015,7 @@ impl StreamingQueryManager {
.unwrap()
.listeners
.retain(|(lid, _)| lid != id);
- Ok(())
+ Ok(true)
}
/// Remove all client-side listeners and stop the dispatch thread. Mirrors
@@ -1471,4 +1478,12 @@ mod tests {
let writer = writer.options(opts);
assert_eq!(writer.options.get("key2"), Some(&"val2".to_string()));
}
+
+ #[test]
+ fn remove_listener_of_an_id_the_bus_does_not_have_is_not_found() {
+ let spark = session();
+ let streams = spark.streams();
+ assert!(!streams.remove_listener_found("no-such-id").unwrap());
+ streams.remove_listener("no-such-id").unwrap();
+ }
}
diff --git a/python/tests/test_dropin_e2e.py b/python/tests/test_dropin_e2e.py
index 67a4fed7..157284eb 100644
--- a/python/tests/test_dropin_e2e.py
+++ b/python/tests/test_dropin_e2e.py
@@ -1635,6 +1635,62 @@ def
test_writer_v2_takes_its_arguments_as_pyspark(session):
session.sql(f"DROP TABLE IF EXISTS {name}")
+def
test_streaming_listeners_are_removed_once_per_registration_as_pyspark(session):
+ # As PySpark's StreamingQueryManager: a listener added twice is in the bus
twice, and each
+ # removeListener removes one of its registrations; a removal through
another session, whose
+ # bus does not have the listener, leaves it in this bus. Once removed, it
gets no events.
+ from pyspark.sql.streaming import StreamingQueryListener
+
+ class Listener(StreamingQueryListener):
+ def __init__(self):
+ self.events = []
+
+ def onQueryStarted(self, event):
+ self.events.append("started")
+
+ def onQueryProgress(self, event):
+ self.events.append("progress")
+
+ def onQueryIdle(self, event):
+ pass
+
+ def onQueryTerminated(self, event):
+ self.events.append("terminated")
+
+ listener = Listener()
+ other = session.newSession()
+ try:
+ session.streams.addListener(listener)
+ session.streams.addListener(listener)
+ assert session.streams._sqlb._listener_bus == [listener, listener]
+ session.streams.removeListener(listener)
+ assert session.streams._sqlb._listener_bus == [listener]
+ session.streams.removeListener(listener)
+ assert session.streams._sqlb._listener_bus == []
+ session.streams.removeListener(listener)
+ assert session.streams._sqlb._listener_bus == []
+
+ session.streams.addListener(listener)
+ other.streams.removeListener(listener)
+ assert session.streams._sqlb._listener_bus == [listener]
+ assert other.streams._sqlb._listener_bus == []
+ session.streams.removeListener(listener)
+ assert session.streams._sqlb._listener_bus == []
+
+ query =
session.readStream.format("rate").load().writeStream.format("noop")
+ query = query.trigger(processingTime="200 milliseconds").start()
+ try:
+ time.sleep(2)
+ finally:
+ query.stop()
+ time.sleep(1)
+ assert listener.events == []
+ finally:
+ for registered in list(session.streams._sqlb._listener_bus):
+ session.streams.removeListener(registered)
+ other.stop()
+
+
def
test_collected_rows_take_the_names_of_the_schema_of_the_result_as_pyspark(session):
# As PySpark Connect's: the fields of nested structs have the names of the
schema, which keep
# duplicate names, and collect raises for a calendar interval.
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]