This is an automated email from the ASF dual-hosted git repository.

HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git


The following commit(s) were added to refs/heads/master by this push:
     new 3bd11e10 [SPARK-61198][CONNECT] Raise NO_ACTIVE_SESSION in the retries 
of an RPC whose session was stopped, in the Rust client as PySpark does
3bd11e10 is described below

commit 3bd11e10841a554f9063b1e547b35bee5f832536
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Mon Nov 9 09:07:23 2026 +0900

    [SPARK-61198][CONNECT] Raise NO_ACTIVE_SESSION in the retries of an RPC 
whose session was stopped, in the Rust client as PySpark does
    
    ### What changes were proposed in this pull request?
    
    [SPARK-61198](https://issues.apache.org/jira/browse/SPARK-61198): PySpark 
retries an RPC in `for attempt in self._retrying()`, and each attempt goes 
through `SparkConnectClient._stub`, which raises `NO_ACTIVE_SESSION` once the 
client is closed. So when a session is stopped while one of its calls waits to 
retry, the next attempt raises and nothing more is sent.
    
    The retry loops of the Rust client now check that the client is open after 
each backoff, before the next attempt, and raise `NO_ACTIVE_SESSION` if it is 
closed:
    
    - `with_retry`, which retries the initial ExecutePlan of an execution and 
the unary RPCs: AnalyzePlan, Config, Interrupt, CloneSession and the artifact 
status calls.
    - the retries of an artifact upload.
    
    This is the check that the reattach of a result stream already does before 
each attempt (SPARK-61187). The background ReleaseExecute is not changed.
    
    ### Why are the changes needed?
    
    The client was checked only when a call started. A call that retried, for 
example an ExecutePlan that got `UNAVAILABLE` from a server that went away, 
kept calling the server after `spark.stop()`, until the retry policy gave up. 
With the default policy (15 retries, backoffs up to 60 seconds) that takes 
minutes, and the thread of the call stays busy:
    
    ```
    spark = SparkSession.builder.remote("sc://localhost:1").getOrCreate()   # 
no server there
    t = threading.Thread(target=lambda: spark.range(2).count(), daemon=True); 
t.start()
    time.sleep(3); spark.stop(); t.join(15)
    # before: the thread is still retrying 15 seconds after stop()
    # after:  the count raised SparkConnectException [NO_ACTIVE_SESSION] at its 
next attempt
    ```
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes. A call that waits to retry when its session is stopped raises 
`SparkConnectException` with `NO_ACTIVE_SESSION` at its next attempt, instead 
of retrying until the policy gives up.
    
    In PySpark, the ExecutePlan of a reattachable execution (the default) is 
retried by its iterator, which holds the gRPC stub of the closed channel. Its 
next attempt fails at once with `ValueError: Cannot invoke RPC on closed 
channel!` rather than `NO_ACTIVE_SESSION`. Either way, nothing more is sent.
    
    ### How was this patch tested?
    
    - New Rust test 
`retry_of_an_execution_whose_client_was_closed_raises_no_active_session` in 
`crates/spark-connect-core/tests/deadlines.rs`, with the mock server. The 
server fails every ExecutePlan with `UNAVAILABLE` under a policy of 1000 
retries. Once the execution has been retried, the client is closed, the call 
raises `NO_ACTIVE_SESSION`, and the server gets no more ExecutePlans. Without 
the change, the call keeps retrying, and the test fails at its 5-second bound.
    - The other tests of `deadlines.rs` (15 passed).
    - The lib tests of both crates (618 passed), the build of all tests, also 
with `wasm-udf`, the offline Python suites (2456 passed, 8 skipped), and the 
live drop-in and Arrow suites (217 passed) and live cases (11 passed) against a 
local Spark 4.3.0 server. Clippy reports no diagnostics in the changed lines.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Opus 5.5
    
    This pull request and its description were written by Isaac.
    
    Closes #386 from HyukjinKwon/SPARK-61198-retry-after-stop.
    
    Authored-by: Hyukjin Kwon <[email protected]>
    Signed-off-by: Hyukjin Kwon <[email protected]>
---
 crates/spark-connect-core/src/client.rs      |  6 ++++
 crates/spark-connect-core/tests/deadlines.rs | 46 ++++++++++++++++++++++++++++
 2 files changed, 52 insertions(+)

diff --git a/crates/spark-connect-core/src/client.rs 
b/crates/spark-connect-core/src/client.rs
index 50603bc1..9cb0ba5c 100644
--- a/crates/spark-connect-core/src/client.rs
+++ b/crates/spark-connect-core/src/client.rs
@@ -843,6 +843,10 @@ impl SparkConnectClient {
                     if self.retry_policy.can_retry(&status) {
                         if let Some(wait_ms) = state.next_attempt(None) {
                             
tokio::time::sleep(std::time::Duration::from_millis(wait_ms)).await;
+                            // As the reference's `_stub`, through which each 
attempt goes: once
+                            // the client is closed (its session stopped while 
the call waited),
+                            // the retry raises NO_ACTIVE_SESSION and sends 
nothing.
+                            self.ensure_open()?;
                             continue;
                         }
                     }
@@ -1574,6 +1578,8 @@ impl SparkConnectClient {
                     if self.retry_policy.can_retry(&status) {
                         if let Some(wait_ms) = state.next_attempt(None) {
                             
tokio::time::sleep(std::time::Duration::from_millis(wait_ms)).await;
+                            // As in `with_retry`: a closed client sends no 
more attempts.
+                            self.ensure_open()?;
                             continue;
                         }
                     }
diff --git a/crates/spark-connect-core/tests/deadlines.rs 
b/crates/spark-connect-core/tests/deadlines.rs
index f5631371..878cc4e4 100644
--- a/crates/spark-connect-core/tests/deadlines.rs
+++ b/crates/spark-connect-core/tests/deadlines.rs
@@ -824,3 +824,49 @@ async fn 
reattach_of_a_stream_whose_client_was_closed_raises_no_active_session()
     assert!(fixture.state.reattaches.lock().unwrap().is_empty());
     assert_eq!(*fixture.state.executes.lock().unwrap(), 1);
 }
+
+#[tokio::test]
+async fn 
retry_of_an_execution_whose_client_was_closed_raises_no_active_session() {
+    // As the reference: each attempt of a call goes through the stub of the 
client, which
+    // raises NO_ACTIVE_SESSION once the client is closed. An execution that 
waits to retry when
+    // its session is stopped raises that error at its next attempt, and sends 
nothing more.
+    let fixture = Fixture::start(
+        (0..1000)
+            .map(|_| Step::Fail(Status::unavailable("server unavailable")))
+            .collect(),
+        vec![],
+        RetryPolicy {
+            max_retries: Some(1000),
+            initial_backoff_ms: 20,
+            max_backoff_ms: Some(20),
+            ..RetryPolicy::default()
+        },
+    )
+    .await;
+    let client = fixture.client.clone();
+    let call = tokio::spawn(async move {
+        client
+            .execute_plan_reattachable(execute_request())
+            .await
+            .map(|_| ())
+    });
+    // The execution is retried before its session stops.
+    let state = fixture.state.clone();
+    bounded(async {
+        while *state.executes.lock().unwrap() < 2 {
+            tokio::time::sleep(Duration::from_millis(5)).await;
+        }
+    })
+    .await;
+
+    fixture.client.close();
+    let error = bounded(call).await.unwrap().unwrap_err();
+    assert_eq!(
+        error.get_condition().as_deref(),
+        Some("NO_ACTIVE_SESSION"),
+        "{error:?}"
+    );
+    let sent = *fixture.state.executes.lock().unwrap();
+    tokio::time::sleep(Duration::from_millis(100)).await;
+    assert_eq!(*fixture.state.executes.lock().unwrap(), sent);
+}


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to