This is an automated email from the ASF dual-hosted git repository.
HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git
The following commit(s) were added to refs/heads/master by this push:
new 3bd11e10 [SPARK-61198][CONNECT] Raise NO_ACTIVE_SESSION in the retries
of an RPC whose session was stopped, in the Rust client as PySpark does
3bd11e10 is described below
commit 3bd11e10841a554f9063b1e547b35bee5f832536
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Mon Nov 9 09:07:23 2026 +0900
[SPARK-61198][CONNECT] Raise NO_ACTIVE_SESSION in the retries of an RPC
whose session was stopped, in the Rust client as PySpark does
### What changes were proposed in this pull request?
[SPARK-61198](https://issues.apache.org/jira/browse/SPARK-61198): PySpark
retries an RPC in `for attempt in self._retrying()`, and each attempt goes
through `SparkConnectClient._stub`, which raises `NO_ACTIVE_SESSION` once the
client is closed. So when a session is stopped while one of its calls waits to
retry, the next attempt raises and nothing more is sent.
The retry loops of the Rust client now check that the client is open after
each backoff, before the next attempt, and raise `NO_ACTIVE_SESSION` if it is
closed:
- `with_retry`, which retries the initial ExecutePlan of an execution and
the unary RPCs: AnalyzePlan, Config, Interrupt, CloneSession and the artifact
status calls.
- the retries of an artifact upload.
This is the check that the reattach of a result stream already does before
each attempt (SPARK-61187). The background ReleaseExecute is not changed.
### Why are the changes needed?
The client was checked only when a call started. A call that retried, for
example an ExecutePlan that got `UNAVAILABLE` from a server that went away,
kept calling the server after `spark.stop()`, until the retry policy gave up.
With the default policy (15 retries, backoffs up to 60 seconds) that takes
minutes, and the thread of the call stays busy:
```
spark = SparkSession.builder.remote("sc://localhost:1").getOrCreate() #
no server there
t = threading.Thread(target=lambda: spark.range(2).count(), daemon=True);
t.start()
time.sleep(3); spark.stop(); t.join(15)
# before: the thread is still retrying 15 seconds after stop()
# after: the count raised SparkConnectException [NO_ACTIVE_SESSION] at its
next attempt
```
### Does this PR introduce _any_ user-facing change?
Yes. A call that waits to retry when its session is stopped raises
`SparkConnectException` with `NO_ACTIVE_SESSION` at its next attempt, instead
of retrying until the policy gives up.
In PySpark, the ExecutePlan of a reattachable execution (the default) is
retried by its iterator, which holds the gRPC stub of the closed channel. Its
next attempt fails at once with `ValueError: Cannot invoke RPC on closed
channel!` rather than `NO_ACTIVE_SESSION`. Either way, nothing more is sent.
### How was this patch tested?
- New Rust test
`retry_of_an_execution_whose_client_was_closed_raises_no_active_session` in
`crates/spark-connect-core/tests/deadlines.rs`, with the mock server. The
server fails every ExecutePlan with `UNAVAILABLE` under a policy of 1000
retries. Once the execution has been retried, the client is closed, the call
raises `NO_ACTIVE_SESSION`, and the server gets no more ExecutePlans. Without
the change, the call keeps retrying, and the test fails at its 5-second bound.
- The other tests of `deadlines.rs` (15 passed).
- The lib tests of both crates (618 passed), the build of all tests, also
with `wasm-udf`, the offline Python suites (2456 passed, 8 skipped), and the
live drop-in and Arrow suites (217 passed) and live cases (11 passed) against a
local Spark 4.3.0 server. Clippy reports no diagnostics in the changed lines.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
This pull request and its description were written by Isaac.
Closes #386 from HyukjinKwon/SPARK-61198-retry-after-stop.
Authored-by: Hyukjin Kwon <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
---
crates/spark-connect-core/src/client.rs | 6 ++++
crates/spark-connect-core/tests/deadlines.rs | 46 ++++++++++++++++++++++++++++
2 files changed, 52 insertions(+)
diff --git a/crates/spark-connect-core/src/client.rs
b/crates/spark-connect-core/src/client.rs
index 50603bc1..9cb0ba5c 100644
--- a/crates/spark-connect-core/src/client.rs
+++ b/crates/spark-connect-core/src/client.rs
@@ -843,6 +843,10 @@ impl SparkConnectClient {
if self.retry_policy.can_retry(&status) {
if let Some(wait_ms) = state.next_attempt(None) {
tokio::time::sleep(std::time::Duration::from_millis(wait_ms)).await;
+ // As the reference's `_stub`, through which each
attempt goes: once
+ // the client is closed (its session stopped while
the call waited),
+ // the retry raises NO_ACTIVE_SESSION and sends
nothing.
+ self.ensure_open()?;
continue;
}
}
@@ -1574,6 +1578,8 @@ impl SparkConnectClient {
if self.retry_policy.can_retry(&status) {
if let Some(wait_ms) = state.next_attempt(None) {
tokio::time::sleep(std::time::Duration::from_millis(wait_ms)).await;
+ // As in `with_retry`: a closed client sends no
more attempts.
+ self.ensure_open()?;
continue;
}
}
diff --git a/crates/spark-connect-core/tests/deadlines.rs
b/crates/spark-connect-core/tests/deadlines.rs
index f5631371..878cc4e4 100644
--- a/crates/spark-connect-core/tests/deadlines.rs
+++ b/crates/spark-connect-core/tests/deadlines.rs
@@ -824,3 +824,49 @@ async fn
reattach_of_a_stream_whose_client_was_closed_raises_no_active_session()
assert!(fixture.state.reattaches.lock().unwrap().is_empty());
assert_eq!(*fixture.state.executes.lock().unwrap(), 1);
}
+
+#[tokio::test]
+async fn
retry_of_an_execution_whose_client_was_closed_raises_no_active_session() {
+ // As the reference: each attempt of a call goes through the stub of the
client, which
+ // raises NO_ACTIVE_SESSION once the client is closed. An execution that
waits to retry when
+ // its session is stopped raises that error at its next attempt, and sends
nothing more.
+ let fixture = Fixture::start(
+ (0..1000)
+ .map(|_| Step::Fail(Status::unavailable("server unavailable")))
+ .collect(),
+ vec![],
+ RetryPolicy {
+ max_retries: Some(1000),
+ initial_backoff_ms: 20,
+ max_backoff_ms: Some(20),
+ ..RetryPolicy::default()
+ },
+ )
+ .await;
+ let client = fixture.client.clone();
+ let call = tokio::spawn(async move {
+ client
+ .execute_plan_reattachable(execute_request())
+ .await
+ .map(|_| ())
+ });
+ // The execution is retried before its session stops.
+ let state = fixture.state.clone();
+ bounded(async {
+ while *state.executes.lock().unwrap() < 2 {
+ tokio::time::sleep(Duration::from_millis(5)).await;
+ }
+ })
+ .await;
+
+ fixture.client.close();
+ let error = bounded(call).await.unwrap().unwrap_err();
+ assert_eq!(
+ error.get_condition().as_deref(),
+ Some("NO_ACTIVE_SESSION"),
+ "{error:?}"
+ );
+ let sent = *fixture.state.executes.lock().unwrap();
+ tokio::time::sleep(Duration::from_millis(100)).await;
+ assert_eq!(*fixture.state.executes.lock().unwrap(), sent);
+}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]