This is an automated email from the ASF dual-hosted git repository.
HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git
The following commit(s) were added to refs/heads/master by this push:
new e8b1d428 [SPARK-61162][CONNECT][PYTHON] Share the streaming query
listener bus of a session, post the started events of queries, and give the
listeners the session, in the Rust and Python clients, as PySpark does
e8b1d428 is described below
commit e8b1d4284e11d119c4fda9563a913ece22f43e7c
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Sun Nov 8 15:07:28 2026 +0900
[SPARK-61162][CONNECT][PYTHON] Share the streaming query listener bus of a
session, post the started events of queries, and give the listeners the
session, in the Rust and Python clients, as PySpark does
### What changes were proposed in this pull request?
[SPARK-61162](https://issues.apache.org/jira/browse/SPARK-61162): PySpark
Connect's `spark.streams` is one `StreamingQueryManager` of the session, whose
`StreamingQueryListenerBus` holds the client-side listeners. This PR makes the
clients' listeners work the same way.
`addListener` calls `_set_spark_session` of PySpark's
`StreamingQueryListener`, which #350 (SPARK-61151) vendored.
**Rust core** (`spark-connect`):
- The managers of a session (`session.streams()`) share the listener bus of
the session, rather than each having one of its own.
- `add_listener` of the first listener returns once the server has added
its listener (`listener_bus_listener_added`), as PySpark's
`_register_server_side_listener` waits for it. So a query started right after
it sends its started event.
- `DataStreamWriter` posts the started event of a query, which the server
sends with the result of the start (`query_started_event_json`), to the
listeners before `start` returns, as `QUERY_STARTED_EVENT`.
- `remove_listener` of an id of no listener does nothing. For the last
listener, it asks the server to remove its listener, waits for the events left,
and should that request fail, keeps the listener and returns the error, as
PySpark's.
- `listeners()` gives the listeners, in their order.
`StreamingQueryListener` gains `as_any`, `None` by default, for a client that
wraps its listeners.
- The events are posted to the listeners without any lock held, so a
listener can add or remove listeners.
**Python client**:
- `spark.streams` is the same object at each access.
- `addListener` gives the listener the session (`listener.spark`), as
PySpark Connect's `_set_spark_session`.
- A failure to add or remove the listener of the server warns, with
PySpark's messages, rather than raising.
- `StreamingQueryManager._sqlb._listener_bus` lists the listeners, as
PySpark Connect's, which Apache Spark's tests read to remove them.
- The callbacks get the started events (`onQueryStarted`). An exception of
a callback warns with PySpark's message.
### Why are the changes needed?
- `onQueryStarted` was never called: the client ignored
`query_started_event_json`.
- Listeners had no session (`self.spark` was `None`), so a listener that
runs queries failed.
- A listener added through one `spark.streams` was removed through another
only because removing from an empty bus asked the server to remove the listener
of the session, which also cut the other listeners off.
- `addListener` returned before the server had added its listener, so a
query started right after could miss its events.
- `spark.streams == spark.streams` was `False`, and the manager had no
`_sqlb`.
Six tests of Apache Spark's `test_parity_listener.py`, and
`test_query_manager_no_recreation` of `test_parity_streaming.py`, failed on
this client.
### Does this PR introduce _any_ user-facing change?
Yes. The listeners of streaming queries get their events, and their
session, as with PySpark 4.3.0. For the Rust API, `QUERY_STARTED_EVENT`,
`StreamingQueryManager::listeners` and `StreamingQueryListener::as_any` are new.
### How was this patch tested?
- A new Rust e2e test adds a listener through one manager of a session, and
checks:
- that it gets the started event before `start` returns, and progress
events
- that the other managers of the session list it
- that it gets no more events once removed through another manager
- A new live test checks:
- that `spark.streams` is the same object
- the session of the listeners, and `_sqlb._listener_bus`
- that two listeners get the started event before `start` returns
- that a removed listener gets no more events, while the other gets the
terminated event
It passes with official PySpark 4.3.0, and fails without this change.
- Apache Spark's `test_parity_listener.py` and `test_parity_streaming.py`
pass on this client against a local server, all 9 and 30 tests. Before this
change, 6 and 1 failed.
- The lib tests of both crates (597 passed), the build of all tests, also
with `wasm-udf`, the offline Python suites (2214 passed, 8 skipped), and the
live drop-in and Arrow suites (200 passed) and live cases (10 passed) against a
local Spark 4.3.0 server. Clippy reports no diagnostics in the changed lines.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
This pull request and its description were written by Isaac.
Closes #353 from HyukjinKwon/SPARK-61162-listener-bus.
Authored-by: Hyukjin Kwon <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
---
crates/pyspark-rs/src/session.rs | 26 ++-
crates/pyspark-rs/src/streaming.rs | 104 +++++++++--
crates/spark-connect/src/session.rs | 12 ++
crates/spark-connect/src/streaming.rs | 277 +++++++++++++++++++++-------
crates/spark-connect/tests/e2e_streaming.rs | 70 +++++++
python/pyspark/sql/streaming/query.py | 17 +-
python/tests/test_dropin_e2e.py | 51 +++++
7 files changed, 463 insertions(+), 94 deletions(-)
diff --git a/crates/pyspark-rs/src/session.rs b/crates/pyspark-rs/src/session.rs
index b19a8445..1bfe99b1 100644
--- a/crates/pyspark-rs/src/session.rs
+++ b/crates/pyspark-rs/src/session.rs
@@ -564,6 +564,8 @@ fn tag_of(py: Python<'_>, tag: &Bound<'_, PyAny>) ->
PyResult<String> {
#[pyclass(name = "SparkSession", module = "pyspark.sql.session")]
pub struct PySparkSession {
pub(crate) session: SparkSession,
+ /// The `StreamingQueryManager` of `spark.streams`, made at its first
access.
+ streams: std::sync::OnceLock<Py<PyStreamingQueryManager>>,
}
impl PySparkSession {
@@ -576,7 +578,10 @@ impl PySparkSession {
error.write_unraisable(py, None);
}
});
- PySparkSession { session }
+ PySparkSession {
+ session,
+ streams: std::sync::OnceLock::new(),
+ }
}
/// A DataFrame of local rows, as PySpark Connect creates one: the schema
is given, or
@@ -1768,14 +1773,23 @@ impl PySparkSession {
PyDataStreamReader::new(reader)
}
- /// Get the StreamingQueryManager for managing active streaming queries
(`spark.streams`).
+ /// Get the StreamingQueryManager for managing active streaming queries
(`spark.streams`):
+ /// the same object at each access, as PySpark Connect's.
///
- /// The native manager owns a Rust-side listener bus, so
`addListener`/`removeListener`/
- /// `close` are implemented in the core (Rust clients get the feature too).
+ /// The native manager is of the Rust-side listener bus of the session, so
+ /// `addListener`/`removeListener`/`close` are implemented in the core
(Rust clients get the
+ /// feature too).
#[getter]
#[pyo3(name = "streams")]
- fn streams(&self) -> PyStreamingQueryManager {
- PyStreamingQueryManager::new(self.session.streams())
+ fn streams(&self, py: Python<'_>) -> PyResult<Py<PyStreamingQueryManager>>
{
+ if let Some(manager) = self.streams.get() {
+ return Ok(manager.clone_ref(py));
+ }
+ let manager = Py::new(
+ py,
+ PyStreamingQueryManager::new(self.session.streams(),
self.session.clone()),
+ )?;
+ Ok(self.streams.get_or_init(|| manager).clone_ref(py))
}
/// Stop this Spark session. Best-effort and idempotent, mirroring
pyspark: a failure
diff --git a/crates/pyspark-rs/src/streaming.rs
b/crates/pyspark-rs/src/streaming.rs
index cc47079a..3a7c7df6 100644
--- a/crates/pyspark-rs/src/streaming.rs
+++ b/crates/pyspark-rs/src/streaming.rs
@@ -37,6 +37,11 @@ impl StreamingQueryListener for PyListenerAdapter {
}
});
}
+
+ /// The adapter, of which `StreamingQueryManager._sqlb` gives back the
Python listener.
+ fn as_any(&self) -> Option<&dyn std::any::Any> {
+ Some(self)
+ }
}
/// Apply a named read option to a streaming reader when supplied (mirrors the
batch
@@ -1119,15 +1124,57 @@ impl PyStreamingQuery {
}
}
-/// Python wrapper for StreamingQueryManager.
+/// Python wrapper for StreamingQueryManager, of the listener bus of its
session.
#[pyclass(name = "StreamingQueryManager", module =
"pyspark.sql.streaming.query")]
pub struct PyStreamingQueryManager {
inner: StreamingQueryManager,
+ /// The session, which the listeners get as their `spark`.
+ session: spark_connect::session::SparkSession,
}
impl PyStreamingQueryManager {
- pub fn new(manager: StreamingQueryManager) -> Self {
- PyStreamingQueryManager { inner: manager }
+ pub fn new(
+ manager: StreamingQueryManager,
+ session: spark_connect::session::SparkSession,
+ ) -> Self {
+ PyStreamingQueryManager {
+ inner: manager,
+ session,
+ }
+ }
+}
+
+/// Warn of `message`, as PySpark Connect's listener bus warns of a failure to
add or remove a
+/// listener, rather than raising.
+fn warn(py: Python<'_>, message: String) -> PyResult<()> {
+ py.import("warnings")?.call_method1("warn", (message,))?;
+ Ok(())
+}
+
+/// `StreamingQueryManager._sqlb`, as PySpark Connect's
`StreamingQueryListenerBus`, of its
+/// `_listener_bus`: the client-side listeners of the session, in the order
that they were added,
+/// which Apache Spark's tests read to remove them.
+#[pyclass(
+ name = "StreamingQueryListenerBus",
+ module = "pyspark.sql.streaming.query",
+ frozen
+)]
+pub struct PyStreamingQueryListenerBus {
+ manager: StreamingQueryManager,
+}
+
+#[pymethods]
+impl PyStreamingQueryListenerBus {
+ #[getter]
+ fn _listener_bus(&self, py: Python<'_>) -> Vec<Py<PyAny>> {
+ self.manager
+ .listeners()
+ .iter()
+ .filter_map(|listener| {
+ let adapter =
listener.as_any()?.downcast_ref::<PyListenerAdapter>()?;
+ Some(adapter.listener.clone_ref(py))
+ })
+ .collect()
}
}
@@ -1161,31 +1208,64 @@ impl PyStreamingQueryManager {
}
/// Register a client-side listener object (with
onQueryStarted/Progress/Idle/
- /// Terminated callbacks). Mirrors `StreamingQueryManager.addListener`.
The native
- /// Rust bus streams events and dispatches to this listener; the assigned
id is
- /// stashed on the listener so `removeListener` can find it.
+ /// Terminated callbacks). Mirrors `StreamingQueryManager.addListener`:
the listener gets the
+ /// session as its `spark`, and the native Rust bus of the session
dispatches its events to
+ /// it; the assigned id is stashed on the listener so `removeListener` can
find it. Should the
+ /// server fail to add its listener, this warns, as PySpark Connect's does.
fn addListener(&self, py: Python<'_>, listener: Py<PyAny>) -> PyResult<()>
{
+ let session =
crate::session::PySparkSession::new(self.session.clone());
+ listener
+ .bind(py)
+ .call_method1("_set_spark_session", (session,))?;
let adapter = Arc::new(PyListenerAdapter {
listener: listener.clone_ref(py),
});
- let id =
- crate::dataframe::detached(py, ||
self.inner.add_listener(adapter).map_err(Box::new))?;
- listener.bind(py).setattr("_rust_listener_id", id)?;
- Ok(())
+ match crate::dataframe::detached(py, ||
self.inner.add_listener(adapter).map_err(Box::new))
+ {
+ Ok(id) => listener.bind(py).setattr("_rust_listener_id", id),
+ Err(error) => warn(
+ py,
+ format!(
+ "Failed to add the listener because of exception: {}\n\
+ The listener is not added, please add it again.",
+ error.value(py)
+ ),
+ ),
+ }
}
/// Remove a previously-added client-side listener object. Mirrors
- /// `StreamingQueryManager.removeListener`.
+ /// `StreamingQueryManager.removeListener`: should the server fail to
remove its listener,
+ /// this warns, and the listener stays, as PySpark Connect's does.
fn removeListener(&self, py: Python<'_>, listener: Py<PyAny>) ->
PyResult<()> {
let bound = listener.bind(py);
if let Ok(id_obj) = bound.getattr("_rust_listener_id") {
let id: String = id_obj.extract()?;
- crate::dataframe::detached(py, ||
self.inner.remove_listener(&id).map_err(Box::new))?;
+ if let Err(error) =
+ crate::dataframe::detached(py, ||
self.inner.remove_listener(&id).map_err(Box::new))
+ {
+ return warn(
+ py,
+ format!(
+ "Failed to remove the listener because of exception:
{}\n\
+ The listener is not removed, please remove it again.",
+ error.value(py)
+ ),
+ );
+ }
let _ = bound.delattr("_rust_listener_id");
}
Ok(())
}
+ /// The listener bus of the session, as PySpark Connect's `_sqlb`.
+ #[getter]
+ fn _sqlb(&self) -> PyStreamingQueryListenerBus {
+ PyStreamingQueryListenerBus {
+ manager: self.session.streams(),
+ }
+ }
+
/// Remove all client-side listeners and stop the dispatch thread. Mirrors
/// `StreamingQueryManager.close`.
fn close(&self, py: Python<'_>) -> PyResult<()> {
diff --git a/crates/spark-connect/src/session.rs
b/crates/spark-connect/src/session.rs
index cbbbf63f..95b3eab4 100644
--- a/crates/spark-connect/src/session.rs
+++ b/crates/spark-connect/src/session.rs
@@ -96,6 +96,10 @@ pub struct SparkSession {
stopped: Arc<AtomicBool>,
/// Successful server version, shared by handles for this session only.
server_version: Arc<Mutex<Option<String>>>,
+ /// The client-side listener bus of the streaming queries of this session
(shared across
+ /// clones), which each of its `StreamingQueryManager`s adds to, removes
from and posts to,
+ /// as the one `StreamingQueryListenerBus` of PySpark Connect's
`spark.streams`.
+ listener_bus: Arc<crate::streaming::ListenerBus>,
}
impl SparkSession {
@@ -111,6 +115,7 @@ impl SparkSession {
python_accumulators: PythonAccumulators::default(),
stopped: Arc::new(AtomicBool::new(false)),
server_version: Arc::new(Mutex::new(None)),
+ listener_bus: Arc::default(),
}
}
@@ -276,6 +281,7 @@ impl SparkSession {
python_accumulators: PythonAccumulators::default(),
stopped: Arc::new(AtomicBool::new(false)),
server_version: Arc::new(Mutex::new(None)),
+ listener_bus: Arc::default(),
}
}
@@ -656,6 +662,11 @@ impl SparkSession {
crate::streaming::StreamingQueryManager::new(self.clone())
}
+ /// The client-side listener bus of the streaming queries of this session.
+ pub(crate) fn listener_bus(&self) -> Arc<crate::streaming::ListenerBus> {
+ Arc::clone(&self.listener_bus)
+ }
+
/// Get the catalog for this session.
///
/// Mirrors `pyspark.sql.SparkSession.catalog`.
@@ -1106,6 +1117,7 @@ impl Clone for SparkSession {
python_accumulators: self.python_accumulators.clone(),
stopped: Arc::clone(&self.stopped),
server_version: Arc::clone(&self.server_version),
+ listener_bus: Arc::clone(&self.listener_bus),
}
}
}
diff --git a/crates/spark-connect/src/streaming.rs
b/crates/spark-connect/src/streaming.rs
index 9e17be0e..f8a72ec0 100644
--- a/crates/spark-connect/src/streaming.rs
+++ b/crates/spark-connect/src/streaming.rs
@@ -465,6 +465,7 @@ impl DataStreamWriter {
let mut query_id = String::new();
let mut run_id = String::new();
let mut name = self.query_name.clone();
+ let mut started_event = None;
for resp in responses {
if let Some(
proto::execute_plan_response::ResponseType::WriteStreamOperationStartResult(res),
@@ -477,6 +478,7 @@ impl DataStreamWriter {
if !res.name.is_empty() {
name = Some(res.name);
}
+ started_event = res.query_started_event_json;
}
}
if query_id.is_empty() {
@@ -484,6 +486,17 @@ impl DataStreamWriter {
"writeStream.start: server returned no
WriteStreamOperationStartResult",
));
}
+ // As PySpark Connect's DataStreamWriter: the server sends the started
event of the query
+ // with the result of its start, as the session has client-side
listeners, and the client
+ // posts it to them before it gives back the query.
+ if let Some(event_json) = started_event {
+ self.session
+ .streams()
+ .post_to_all(&StreamingQueryListenerEvent {
+ event_type: QUERY_STARTED_EVENT,
+ event_json,
+ });
+ }
Ok(StreamingQuery {
session: self.session,
@@ -744,6 +757,46 @@ pub struct ListenerEventStream {
done: bool,
}
+impl ListenerEventStream {
+ /// Read the responses of the server until the one that tells that it
added the listener of
+ /// the session (`listener_bus_listener_added`), keeping the events of
them, which the
+ /// iterator gives first, as PySpark Connect's
`_register_server_side_listener` waits for it.
+ fn wait_until_listener_added(&mut self) -> Result<()> {
+ loop {
+ match block_on(self.stream.message()) {
+ Ok(Some(resp)) => {
+ if let Some(
+
proto::execute_plan_response::ResponseType::StreamingQueryListenerEventsResult(res),
+ ) = resp.response_type
+ {
+ if !res.events.is_empty() {
+ let mut events: Vec<(i32, String)> =
+ std::mem::take(&mut
self.buffered_events).collect();
+ events.extend(
+ res.events
+ .into_iter()
+ .map(|event| (event.event_type,
event.event_json)),
+ );
+ self.buffered_events = events.into_iter();
+ }
+ if res.listener_bus_listener_added.is_some() {
+ return Ok(());
+ }
+ }
+ }
+ Ok(None) => {
+ self.done = true;
+ return Ok(());
+ }
+ Err(e) => {
+ self.done = true;
+ return Err(e);
+ }
+ }
+ }
+ }
+}
+
impl Iterator for ListenerEventStream {
type Item = Result<(i32, String)>;
@@ -796,10 +849,18 @@ pub const QUERY_PROGRESS_EVENT: i32 = 1;
pub const QUERY_TERMINATED_EVENT: i32 = 2;
pub const QUERY_IDLE_EVENT: i32 = 3;
+/// The started event of a query. The server sends it with the result of the
start of the query
+/// (`query_started_event_json`), when the session has a client-side listener,
rather than on the
+/// listener bus, and the client posts it to its listeners as the query
starts, as PySpark
+/// Connect's `DataStreamWriter.start` does: so it is not of the server's
+/// `StreamingQueryEventType`.
+pub const QUERY_STARTED_EVENT: i32 = -1;
+
/// A streaming-query listener event delivered by the client-side listener bus.
///
/// `event_json` is the server-provided JSON for the event; `event_type` is
one of
-/// [`QUERY_PROGRESS_EVENT`], [`QUERY_TERMINATED_EVENT`], [`QUERY_IDLE_EVENT`].
+/// [`QUERY_STARTED_EVENT`], [`QUERY_PROGRESS_EVENT`],
[`QUERY_TERMINATED_EVENT`],
+/// [`QUERY_IDLE_EVENT`].
#[derive(Debug, Clone)]
pub struct StreamingQueryListenerEvent {
pub event_type: i32,
@@ -811,43 +872,64 @@ pub struct StreamingQueryListenerEvent {
/// runs a background bus that streams events from the server and dispatches
them here.
pub trait StreamingQueryListener: Send + Sync {
fn on_event(&self, event: &StreamingQueryListenerEvent);
+
+ /// This listener as `Any`, for a client that wraps its own listeners to
find them among
+ /// [`StreamingQueryManager::listeners`]. `None` by default.
+ fn as_any(&self) -> Option<&dyn std::any::Any> {
+ None
+ }
+}
+
+/// The client-side listener bus of the streaming queries of a session, which
all of its
+/// `StreamingQueryManager`s share, as PySpark Connect's
`StreamingQueryListenerBus` of
+/// `spark.streams`: the registered listeners, in their order, and the thread
that posts the
+/// events of the server to them, from the first listener to the removal of
the last. `ops`
+/// serializes the additions and removals, which talk to the server, while the
thread takes
+/// `state` only to read the listeners, and calls them without any lock.
+#[derive(Default)]
+pub(crate) struct ListenerBus {
+ ops: Mutex<()>,
+ state: Mutex<ListenerBusState>,
}
-/// Shared state of the client-side listener bus: the registered listeners and
the
-/// background dispatch thread (started with the first listener, stopped with
the last).
#[derive(Default)]
struct ListenerBusState {
listeners: Vec<(String, Arc<dyn StreamingQueryListener>)>,
thread: Option<std::thread::JoinHandle<()>>,
}
-/// The background dispatch loop: opens the server event stream and forwards
each event
-/// to every currently-registered listener. Ends when the stream closes (the
server was
-/// asked to stop via `RemoveListenerBusListener`) or all listeners are
removed.
-fn run_listener_event_loop(session: SparkSession, bus:
Arc<Mutex<ListenerBusState>>) {
- let stream = match
StreamingQueryManager::new(session).listener_event_stream() {
- Ok(s) => s,
- Err(_) => return,
- };
+impl ListenerBus {
+ /// Post `event` to every listener, as PySpark Connect's `post_to_all`.
+ fn post_to_all(&self, event: &StreamingQueryListenerEvent) {
+ let listeners: Vec<Arc<dyn StreamingQueryListener>> = self
+ .state
+ .lock()
+ .unwrap()
+ .listeners
+ .iter()
+ .map(|(_, listener)| Arc::clone(listener))
+ .collect();
+ for listener in &listeners {
+ listener.on_event(event);
+ }
+ }
+}
+
+/// The background dispatch loop: posts each event of the stream of the
server, on which the
+/// server has added the listener of the session, to the listeners. It ends
when the stream does,
+/// as when the server was asked to remove its listener
(`RemoveListenerBusListener`). An error
+/// of the stream removes all the listeners, as PySpark Connect's handler
thread does.
+fn run_listener_event_loop(stream: ListenerEventStream, bus: Arc<ListenerBus>)
{
for item in stream {
- let listeners: Vec<Arc<dyn StreamingQueryListener>> = {
- let st = bus.lock().unwrap();
- if st.listeners.is_empty() {
- break;
- }
- st.listeners.iter().map(|(_, l)| l.clone()).collect()
- };
match item {
- Ok((event_type, event_json)) => {
- let ev = StreamingQueryListenerEvent {
- event_type,
- event_json,
- };
- for l in &listeners {
- l.on_event(&ev);
- }
+ Ok((event_type, event_json)) =>
bus.post_to_all(&StreamingQueryListenerEvent {
+ event_type,
+ event_json,
+ }),
+ Err(_) => {
+ bus.state.lock().unwrap().listeners.clear();
+ return;
}
- Err(_) => break,
}
}
}
@@ -855,81 +937,136 @@ fn run_listener_event_loop(session: SparkSession, bus:
Arc<Mutex<ListenerBusStat
/// Manager for active streaming queries.
///
/// Mirrors `pyspark.sql.connect.streaming.StreamingQueryManager`, including a
native
-/// client-side listener bus (so Rust clients get the listener feature too).
+/// client-side listener bus (so Rust clients get the listener feature too),
which is that of
+/// its session.
pub struct StreamingQueryManager {
session: SparkSession,
- bus: Arc<Mutex<ListenerBusState>>,
+ bus: Arc<ListenerBus>,
}
impl StreamingQueryManager {
- /// Create a new StreamingQueryManager.
+ /// Create a new StreamingQueryManager, of the listener bus of `session`.
pub(crate) fn new(session: SparkSession) -> Self {
- StreamingQueryManager {
- session,
- bus: Arc::new(Mutex::new(ListenerBusState::default())),
- }
+ let bus = session.listener_bus();
+ StreamingQueryManager { session, bus }
}
/// Register a client-side listener. Returns an id that can be passed to
- /// [`StreamingQueryManager::remove_listener`]. Starts the background
dispatch
- /// thread when it is the first listener. Mirrors
`StreamingQueryManager.addListener`.
+ /// [`StreamingQueryManager::remove_listener`]. Mirrors
`StreamingQueryManager.addListener`:
+ /// for the first listener of the session, the server adds its listener
before this returns,
+ /// so that the queries started then send their started events, and a
thread posts the events
+ /// of the server to the listeners. Should the server fail to add it, the
listener is not
+ /// added.
pub fn add_listener(&self, listener: Arc<dyn StreamingQueryListener>) ->
Result<String> {
let id = uuid::Uuid::new_v4().to_string();
- let mut st = self.bus.lock().unwrap();
- st.listeners.push((id.clone(), listener));
- if st.listeners.len() == 1 {
- let session = self.session.clone();
- let bus = self.bus.clone();
- st.thread = Some(std::thread::spawn(move || {
- run_listener_event_loop(session, bus);
- }));
+ let _ops = self.bus.ops.lock().unwrap();
+ let first = {
+ let mut state = self.bus.state.lock().unwrap();
+ state.listeners.push((id.clone(), listener));
+ state.listeners.len() == 1
+ };
+ if first {
+ let registered = self.listener_event_stream().and_then(|mut
stream| {
+ stream.wait_until_listener_added()?;
+ Ok(stream)
+ });
+ let stream = match registered {
+ Ok(stream) => stream,
+ Err(e) => {
+ self.bus.state.lock().unwrap().listeners.clear();
+ return Err(e);
+ }
+ };
+ let bus = Arc::clone(&self.bus);
+ let thread = std::thread::spawn(move ||
run_listener_event_loop(stream, bus));
+ self.bus.state.lock().unwrap().thread = Some(thread);
}
Ok(id)
}
- /// Remove a client-side listener by id. Stops the background dispatch
thread when
- /// the last listener is removed. Mirrors
`StreamingQueryManager.removeListener`.
+ /// Remove a client-side listener by id, which does nothing for an id of
no listener. Mirrors
+ /// `StreamingQueryManager.removeListener`: for the last listener, the
server is asked to
+ /// remove its listener, which ends its stream, and this returns once the
thread has posted
+ /// the events left. Should that request fail, the listener stays, and the
error is returned.
pub fn remove_listener(&self, id: &str) -> Result<()> {
- let now_empty = {
- let mut st = self.bus.lock().unwrap();
- st.listeners.retain(|(lid, _)| lid != id);
- st.listeners.is_empty()
- };
- if now_empty {
- self.stop_listener_bus();
+ let _ops = self.bus.ops.lock().unwrap();
+ {
+ let mut state = self.bus.state.lock().unwrap();
+ let Some(index) = state.listeners.iter().position(|(lid, _)| lid
== id) else {
+ return Ok(());
+ };
+ if state.listeners.len() > 1 {
+ state.listeners.remove(index);
+ return Ok(());
+ }
}
+ self.remove_server_listener()?;
+ self.join_listener_thread();
+ self.bus
+ .state
+ .lock()
+ .unwrap()
+ .listeners
+ .retain(|(lid, _)| lid != id);
Ok(())
}
/// Remove all client-side listeners and stop the dispatch thread. Mirrors
- /// `StreamingQueryManager.close`.
+ /// `StreamingQueryManager.close`. Should the server fail to remove its
listener, the thread
+ /// is left to end with its stream.
pub fn close(&self) -> Result<()> {
- let had_listeners = {
- let mut st = self.bus.lock().unwrap();
- let had = !st.listeners.is_empty();
- st.listeners.clear();
- had
- };
- if had_listeners {
- self.stop_listener_bus();
+ let _ops = self.bus.ops.lock().unwrap();
+ if self.bus.state.lock().unwrap().listeners.is_empty() {
+ return Ok(());
+ }
+ if self.remove_server_listener().is_ok() {
+ self.join_listener_thread();
}
+ let mut state = self.bus.state.lock().unwrap();
+ state.listeners.clear();
+ state.thread = None;
Ok(())
}
- /// Ask the server to stop streaming listener events (which ends the
background
- /// thread's stream), then join the thread.
- fn stop_listener_bus(&self) {
+ /// The client-side listeners of the session, in the order that they were
added, as the
+ /// `_listener_bus` of PySpark Connect's `StreamingQueryListenerBus`.
+ pub fn listeners(&self) -> Vec<Arc<dyn StreamingQueryListener>> {
+ self.bus
+ .state
+ .lock()
+ .unwrap()
+ .listeners
+ .iter()
+ .map(|(_, listener)| Arc::clone(listener))
+ .collect()
+ }
+
+ /// Post `event` to the client-side listeners of the session.
+ pub(crate) fn post_to_all(&self, event: &StreamingQueryListenerEvent) {
+ self.bus.post_to_all(event);
+ }
+
+ /// Ask the server to remove the listener of the session, which ends its
stream of events.
+ fn remove_server_listener(&self) -> Result<()> {
let mut lb = proto::StreamingQueryListenerBusCommand::default();
lb.command = Some(
proto::streaming_query_listener_bus_command::Command::RemoveListenerBusListener(true),
);
- let _ = crate::dataframe::execute_command_collect(
+ crate::dataframe::execute_command_collect(
&self.session,
proto::command::CommandType::StreamingQueryListenerBusCommand(lb),
- );
- let handle = self.bus.lock().unwrap().thread.take();
- if let Some(h) = handle {
- let _ = h.join();
+ )
+ .map(|_| ())
+ }
+
+ /// Wait for the dispatch thread to post the events left of its ended
stream, unless this is
+ /// that thread, as when a listener removes the last listener.
+ fn join_listener_thread(&self) {
+ let thread = self.bus.state.lock().unwrap().thread.take();
+ if let Some(thread) = thread {
+ if thread.thread().id() != std::thread::current().id() {
+ let _ = thread.join();
+ }
}
}
diff --git a/crates/spark-connect/tests/e2e_streaming.rs
b/crates/spark-connect/tests/e2e_streaming.rs
index b2283a7a..8092457b 100644
--- a/crates/spark-connect/tests/e2e_streaming.rs
+++ b/crates/spark-connect/tests/e2e_streaming.rs
@@ -168,3 +168,73 @@ fn streaming_native_listener_bus() {
"expected at least one native listener event, got {got}"
);
}
+
+/// As PySpark Connect's `spark.streams`: the managers of a session share its
listener bus, the
+/// started event of a query reaches the listeners before `start` returns, and
a listener removed
+/// through another manager of the session gets no more events.
+#[test]
+fn streaming_listeners_are_those_of_the_session() {
+ if !should_run() {
+ return;
+ }
+ use spark_connect::streaming::{
+ StreamingQueryListener, StreamingQueryListenerEvent,
QUERY_PROGRESS_EVENT,
+ QUERY_STARTED_EVENT,
+ };
+ use std::sync::{Arc, Mutex};
+ use std::time::Duration;
+
+ struct Recording {
+ events: Mutex<Vec<i32>>,
+ }
+ impl StreamingQueryListener for Recording {
+ fn on_event(&self, event: &StreamingQueryListenerEvent) {
+ self.events.lock().unwrap().push(event.event_type);
+ }
+ }
+
+ let spark = session();
+ let listener = Arc::new(Recording {
+ events: Mutex::new(Vec::new()),
+ });
+ let id = spark
+ .streams()
+ .add_listener(listener.clone())
+ .expect("add_listener");
+ assert_eq!(spark.streams().listeners().len(), 1);
+ let query = spark
+ .read_stream()
+ .format("rate")
+ .load(None)
+ .write_stream()
+ .format("noop")
+ .query_name("e2e_session_listener_bus")
+ .start("")
+ .expect("start streaming query");
+ assert_eq!(
+ listener.events.lock().unwrap().first(),
+ Some(&QUERY_STARTED_EVENT)
+ );
+ for _ in 0..60 {
+ if listener
+ .events
+ .lock()
+ .unwrap()
+ .contains(&QUERY_PROGRESS_EVENT)
+ {
+ break;
+ }
+ std::thread::sleep(Duration::from_millis(500));
+ }
+ spark
+ .streams()
+ .remove_listener(&id)
+ .expect("remove_listener");
+ assert!(spark.streams().listeners().is_empty());
+ let seen = listener.events.lock().unwrap().len();
+ std::thread::sleep(Duration::from_secs(2));
+ query.stop().expect("stop");
+ let events = listener.events.lock().unwrap();
+ assert!(events.contains(&QUERY_PROGRESS_EVENT), "{events:?}");
+ assert_eq!(events.len(), seen, "{events:?}");
+}
diff --git a/python/pyspark/sql/streaming/query.py
b/python/pyspark/sql/streaming/query.py
index 919fcb83..c0ae0ae9 100644
--- a/python/pyspark/sql/streaming/query.py
+++ b/python/pyspark/sql/streaming/query.py
@@ -48,7 +48,10 @@ if TYPE_CHECKING:
from pyspark.sql.session import SparkSession
# StreamingQueryEventType values (spark.connect.StreamingQueryEventType); the
Rust
-# listener stream yields these as the event_type int alongside each event's
JSON.
+# listener stream yields these as the event_type int alongside each event's
JSON. The started
+# event of a query comes with the result of its start, which the Rust core
posts as
+# spark_connect::streaming::QUERY_STARTED_EVENT, of no StreamingQueryEventType.
+_QUERY_STARTED_EVENT = -1
_QUERY_PROGRESS_EVENT = 1
_QUERY_TERMINATED_EVENT = 2
_QUERY_IDLE_EVENT = 3
@@ -61,20 +64,22 @@ def _dispatch_listener_event(listener, event_type,
event_json):
"""Deserialize a listener event and invoke the matching callback on
``listener``.
Called from the native (Rust) listener bus for each event. Mirrors the
reference
- ``StreamingQueryListenerBus`` dispatch: build the typed event and route it
to
- onQueryProgress / onQueryIdle / onQueryTerminated. Callback exceptions are
surfaced
- as warnings so one listener cannot break the dispatch thread.
+ ``StreamingQueryListenerBus.post_to_all``: build the typed event and route
it to
+ onQueryStarted / onQueryProgress / onQueryIdle / onQueryTerminated.
Callback exceptions
+ are surfaced as warnings, as PySpark's, so one listener cannot break the
dispatch thread.
"""
try:
j = json.loads(event_json)
- if event_type == _QUERY_PROGRESS_EVENT:
+ if event_type == _QUERY_STARTED_EVENT:
+ listener.onQueryStarted(QueryStartedEvent.fromJson(j))
+ elif event_type == _QUERY_PROGRESS_EVENT:
listener.onQueryProgress(QueryProgressEvent.fromJson(j))
elif event_type == _QUERY_IDLE_EVENT:
listener.onQueryIdle(QueryIdleEvent.fromJson(j))
elif event_type == _QUERY_TERMINATED_EVENT:
listener.onQueryTerminated(QueryTerminatedEvent.fromJson(j))
except Exception as e: # noqa: BLE001
- warnings.warn(f"Listener callback raised exception: {e}")
+ warnings.warn(f"Listener {str(listener)} threw an exception\n{e}")
# ``spark.streams`` returns the native (Rust) StreamingQueryManager, which
owns the
diff --git a/python/tests/test_dropin_e2e.py b/python/tests/test_dropin_e2e.py
index 61d746fc..90ee3602 100644
--- a/python/tests/test_dropin_e2e.py
+++ b/python/tests/test_dropin_e2e.py
@@ -3071,6 +3071,57 @@ def
test_dataframes_of_a_stream_are_streaming_as_pyspark(session):
assert session.range(1).isStreaming is False
+def test_streaming_listeners_are_those_of_the_session_as_pyspark(session):
+ # As PySpark Connect's spark.streams: one manager, of the listener bus of
the session, whose
+ # listeners get the session, are listed by _sqlb, get the started event of
a query before
+ # start returns, and get no events once removed.
+ from pyspark.sql.streaming import StreamingQueryListener
+
+ class Listener(StreamingQueryListener):
+ def __init__(self):
+ self.events = []
+
+ def onQueryStarted(self, event):
+ self.events.append(("started", str(event.id)))
+
+ def onQueryProgress(self, event):
+ self.events.append(("progress", str(event.progress.id)))
+
+ def onQueryIdle(self, event):
+ pass
+
+ def onQueryTerminated(self, event):
+ self.events.append(("terminated", str(event.id)))
+
+ assert session.streams is session.streams
+ first, second = Listener(), Listener()
+ session.streams.addListener(first)
+ session.streams.addListener(second)
+ query = None
+ try:
+ assert first.spark.range(3).count() == 3
+ assert session.streams._sqlb._listener_bus == [first, second]
+ query =
session.readStream.format("rate").load().writeStream.format("noop").start()
+ assert first.events[:1] == second.events[:1] == [("started", query.id)]
+ deadline = time.time() + 60
+ while ("progress", query.id) not in first.events and time.time() <
deadline:
+ time.sleep(0.5)
+ session.streams.removeListener(first)
+ assert session.streams._sqlb._listener_bus == [second]
+ seen = list(first.events)
+ query.stop()
+ deadline = time.time() + 30
+ while ("terminated", query.id) not in second.events and time.time() <
deadline:
+ time.sleep(0.5)
+ assert ("progress", query.id) in seen and first.events == seen
+ assert ("terminated", query.id) in second.events
+ finally:
+ for listener in session.streams._sqlb._listener_bus:
+ session.streams.removeListener(listener)
+ if query is not None:
+ query.stop()
+
+
def
test_streaming_queries_return_the_status_and_the_exception_of_pyspark(session):
# As PySpark's: the status of a query is a dict, and the exception of a
failed query is
# the StreamingQueryException of pyspark.errors, of the message of the
server and its JVM
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]