This is an automated email from the ASF dual-hosted git repository.

HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git


The following commit(s) were added to refs/heads/master by this push:
     new 646b9a66 [SPARK-61187][CONNECT] Raise NO_ACTIVE_SESSION when a stream 
of a stopped session reattaches, as PySpark does
646b9a66 is described below

commit 646b9a66a7b1ec8212edc01aad3da0d7bb34f46f
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Sun Nov 8 19:27:43 2026 +0900

    [SPARK-61187][CONNECT] Raise NO_ACTIVE_SESSION when a stream of a stopped 
session reattaches, as PySpark does
    
    ### What changes were proposed in this pull request?
    
    [SPARK-61187](https://issues.apache.org/jira/browse/SPARK-61187): raise 
`NO_ACTIVE_SESSION` when a result stream of a stopped session needs to reattach.
    
    - `ReattachableResponseStream::reattach` checks the closed state of the 
client (`ensure_open`) before each attempt, that is, before the ReattachExecute 
and before an ExecutePlan sent again. A closed client ends the stream with the 
`NO_ACTIVE_SESSION` error, through `fail`, which releases the stream as the 
other failures do.
    - The background ReleaseExecute of the stream is not changed.
    
    Official PySpark 4.3.0: the iterator reattaches through `client._stub`, 
which raises `NO_ACTIVE_SESSION` once the client is closed, so a read that 
needs a reattach after `stop()` fails with that error and sends nothing.
    
    ### Why are the changes needed?
    
    A result stream is reattached when the server ends it, for example after 
`senderMaxStreamDuration` for a query that runs for minutes. If the session is 
stopped while the stream is still being read, only the RPCs of the closed 
client were refused: the reattach of the open stream went to the server, so the 
stopped session kept reading. Official PySpark raises `NO_ACTIVE_SESSION` 
there. The server-side split of a stream cannot be triggered from a Python 
client, so this is covered by a Ru [...]
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes. A result stream that is read after its session was stopped and that 
needs a reattach raises `SparkConnectException` with `NO_ACTIVE_SESSION`, 
instead of returning more rows.
    
    ### How was this patch tested?
    
    - Rust 
`reattach_of_a_stream_whose_client_was_closed_raises_no_active_session` in 
`tests/deadlines.rs`, with the mock server: the first response is read, the 
client is closed, and the next read fails with `NO_ACTIVE_SESSION`; no 
ReattachExecute is sent, the stream is ended, and no ExecutePlan is sent again. 
It fails without the change, where the reattach returns the rest of the result.
    - The live `stop` and `close` drop-in tests, and the offline Python suites.
    - The lib tests of both crates (599 passed), the build of all tests, also 
with `wasm-udf`, and the offline Python suites (2227 passed, 8 skipped), and 
the deadline and reattach tests of the core against mock servers (14 passed); 
rustfmt passes, and clippy reports no diagnostics in the changed lines.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Opus 5.5
    
    This pull request and its description were written by Isaac.
    
    Closes #367 from HyukjinKwon/SPARK-61187-reattach-after-stop.
    
    Authored-by: Hyukjin Kwon <[email protected]>
    Signed-off-by: Hyukjin Kwon <[email protected]>
---
 crates/spark-connect-core/src/client.rs      |  5 +++++
 crates/spark-connect-core/tests/deadlines.rs | 31 ++++++++++++++++++++++++++++
 2 files changed, 36 insertions(+)

diff --git a/crates/spark-connect-core/src/client.rs 
b/crates/spark-connect-core/src/client.rs
index 0545b225..a01eeb7b 100644
--- a/crates/spark-connect-core/src/client.rs
+++ b/crates/spark-connect-core/src/client.rs
@@ -1725,6 +1725,11 @@ impl ReattachableResponseStream {
         retry_exception_since: &mut Option<std::time::Instant>,
     ) -> Result<()> {
         loop {
+            // As the reference's `_stub`: once the client is closed (its 
session stopped), the
+            // reattach and the ExecutePlan sent again raise 
NO_ACTIVE_SESSION, and the stream ends.
+            if let Err(error) = self.client.ensure_open() {
+                return Err(self.fail(error));
+            }
             let opened = if self.execute_again {
                 self.open_execute_plan_again().await
             } else {
diff --git a/crates/spark-connect-core/tests/deadlines.rs 
b/crates/spark-connect-core/tests/deadlines.rs
index a8fee2e3..9ec72340 100644
--- a/crates/spark-connect-core/tests/deadlines.rs
+++ b/crates/spark-connect-core/tests/deadlines.rs
@@ -791,3 +791,34 @@ impl UnaryService<ConfigRequest> for Config {
         })
     }
 }
+
+#[tokio::test]
+async fn 
reattach_of_a_stream_whose_client_was_closed_raises_no_active_session() {
+    // As the reference: an execution that is still read when its session is 
stopped, and that
+    // needs a reattach, raises NO_ACTIVE_SESSION from the stub of the closed 
client, instead of
+    // reattaching. The stream ends with that error, and the server gets no 
reattach.
+    let fixture = Fixture::start(
+        vec![streamed(vec![response("r1", false)])],
+        vec![streamed(vec![response("r2", true)])],
+        quick_retries(),
+    )
+    .await;
+    let mut stream = 
bounded(fixture.client.execute_plan_reattachable(execute_request()))
+        .await
+        .unwrap();
+    let first = bounded(stream.message()).await.unwrap().unwrap();
+    assert_eq!(first.response_id, "r1");
+
+    // The stream ends after r1, so the next read reattaches, which the closed 
client refuses.
+    fixture.client.close();
+    let error = bounded(stream.message()).await.unwrap_err();
+    assert_eq!(
+        error.get_condition().as_deref(),
+        Some("NO_ACTIVE_SESSION"),
+        "{error:?}"
+    );
+    assert!(bounded(stream.message()).await.unwrap().is_none());
+
+    assert!(fixture.state.reattaches.lock().unwrap().is_empty());
+    assert_eq!(*fixture.state.executes.lock().unwrap(), 1);
+}


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to