This is an automated email from the ASF dual-hosted git repository.
HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git
The following commit(s) were added to refs/heads/master by this push:
new 646b9a66 [SPARK-61187][CONNECT] Raise NO_ACTIVE_SESSION when a stream
of a stopped session reattaches, as PySpark does
646b9a66 is described below
commit 646b9a66a7b1ec8212edc01aad3da0d7bb34f46f
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Sun Nov 8 19:27:43 2026 +0900
[SPARK-61187][CONNECT] Raise NO_ACTIVE_SESSION when a stream of a stopped
session reattaches, as PySpark does
### What changes were proposed in this pull request?
[SPARK-61187](https://issues.apache.org/jira/browse/SPARK-61187): raise
`NO_ACTIVE_SESSION` when a result stream of a stopped session needs to reattach.
- `ReattachableResponseStream::reattach` checks the closed state of the
client (`ensure_open`) before each attempt, that is, before the ReattachExecute
and before an ExecutePlan sent again. A closed client ends the stream with the
`NO_ACTIVE_SESSION` error, through `fail`, which releases the stream as the
other failures do.
- The background ReleaseExecute of the stream is not changed.
Official PySpark 4.3.0: the iterator reattaches through `client._stub`,
which raises `NO_ACTIVE_SESSION` once the client is closed, so a read that
needs a reattach after `stop()` fails with that error and sends nothing.
### Why are the changes needed?
A result stream is reattached when the server ends it, for example after
`senderMaxStreamDuration` for a query that runs for minutes. If the session is
stopped while the stream is still being read, only the RPCs of the closed
client were refused: the reattach of the open stream went to the server, so the
stopped session kept reading. Official PySpark raises `NO_ACTIVE_SESSION`
there. The server-side split of a stream cannot be triggered from a Python
client, so this is covered by a Ru [...]
### Does this PR introduce _any_ user-facing change?
Yes. A result stream that is read after its session was stopped and that
needs a reattach raises `SparkConnectException` with `NO_ACTIVE_SESSION`,
instead of returning more rows.
### How was this patch tested?
- Rust
`reattach_of_a_stream_whose_client_was_closed_raises_no_active_session` in
`tests/deadlines.rs`, with the mock server: the first response is read, the
client is closed, and the next read fails with `NO_ACTIVE_SESSION`; no
ReattachExecute is sent, the stream is ended, and no ExecutePlan is sent again.
It fails without the change, where the reattach returns the rest of the result.
- The live `stop` and `close` drop-in tests, and the offline Python suites.
- The lib tests of both crates (599 passed), the build of all tests, also
with `wasm-udf`, and the offline Python suites (2227 passed, 8 skipped), and
the deadline and reattach tests of the core against mock servers (14 passed);
rustfmt passes, and clippy reports no diagnostics in the changed lines.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
This pull request and its description were written by Isaac.
Closes #367 from HyukjinKwon/SPARK-61187-reattach-after-stop.
Authored-by: Hyukjin Kwon <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
---
crates/spark-connect-core/src/client.rs | 5 +++++
crates/spark-connect-core/tests/deadlines.rs | 31 ++++++++++++++++++++++++++++
2 files changed, 36 insertions(+)
diff --git a/crates/spark-connect-core/src/client.rs
b/crates/spark-connect-core/src/client.rs
index 0545b225..a01eeb7b 100644
--- a/crates/spark-connect-core/src/client.rs
+++ b/crates/spark-connect-core/src/client.rs
@@ -1725,6 +1725,11 @@ impl ReattachableResponseStream {
retry_exception_since: &mut Option<std::time::Instant>,
) -> Result<()> {
loop {
+ // As the reference's `_stub`: once the client is closed (its
session stopped), the
+ // reattach and the ExecutePlan sent again raise
NO_ACTIVE_SESSION, and the stream ends.
+ if let Err(error) = self.client.ensure_open() {
+ return Err(self.fail(error));
+ }
let opened = if self.execute_again {
self.open_execute_plan_again().await
} else {
diff --git a/crates/spark-connect-core/tests/deadlines.rs
b/crates/spark-connect-core/tests/deadlines.rs
index a8fee2e3..9ec72340 100644
--- a/crates/spark-connect-core/tests/deadlines.rs
+++ b/crates/spark-connect-core/tests/deadlines.rs
@@ -791,3 +791,34 @@ impl UnaryService<ConfigRequest> for Config {
})
}
}
+
+#[tokio::test]
+async fn
reattach_of_a_stream_whose_client_was_closed_raises_no_active_session() {
+ // As the reference: an execution that is still read when its session is
stopped, and that
+ // needs a reattach, raises NO_ACTIVE_SESSION from the stub of the closed
client, instead of
+ // reattaching. The stream ends with that error, and the server gets no
reattach.
+ let fixture = Fixture::start(
+ vec![streamed(vec![response("r1", false)])],
+ vec![streamed(vec![response("r2", true)])],
+ quick_retries(),
+ )
+ .await;
+ let mut stream =
bounded(fixture.client.execute_plan_reattachable(execute_request()))
+ .await
+ .unwrap();
+ let first = bounded(stream.message()).await.unwrap().unwrap();
+ assert_eq!(first.response_id, "r1");
+
+ // The stream ends after r1, so the next read reattaches, which the closed
client refuses.
+ fixture.client.close();
+ let error = bounded(stream.message()).await.unwrap_err();
+ assert_eq!(
+ error.get_condition().as_deref(),
+ Some("NO_ACTIVE_SESSION"),
+ "{error:?}"
+ );
+ assert!(bounded(stream.message()).await.unwrap().is_none());
+
+ assert!(fixture.state.reattaches.lock().unwrap().is_empty());
+ assert_eq!(*fixture.state.executes.lock().unwrap(), 1);
+}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]