This is an automated email from the ASF dual-hosted git repository.
HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git
The following commit(s) were added to refs/heads/master by this push:
new bf7bd6a [SPARK-60964][CONNECT] Retry each reattach afresh, and
release an execution that fails at once, as PySpark does
bf7bd6a is described below
commit bf7bd6a4e384998556cf0916b1f545cb95d4352c
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Fri Nov 6 18:26:03 2026 +0900
[SPARK-60964][CONNECT] Retry each reattach afresh, and release an execution
that fails at once, as PySpark does
### What changes were proposed in this pull request?
[SPARK-60964](https://issues.apache.org/jira/browse/SPARK-60964): two
changes to the reattachable execution of the Rust transport, as PySpark's
`ExecutePlanResponseReattachableIterator` does.
- **Each response read has the retries of the policy afresh.**
`ReattachableResponseStream::message` makes a retry state of its own, for the
reattaches that it needs, as `_has_next` makes a `Retrying` of its own. Before,
one retry state served the whole stream.
- **An execution whose first `ExecutePlan` call fails is released.** An
analysis or parse error is an example. `execute_plan_reattachable` sends
`ReleaseExecute` (`ReleaseAll`) before it returns the error, as the iterator
releases an execution on any error. The client now sends releases with the
metadata of the execution, for a stream and for such a failure alike.
### Why are the changes needed?
- The attempts of the shared retry state were never reset, and its
elapsed-time limit counted from the start of the stream. So a query, a
statement or the streaming listener that ran past the limit, or that reattached
more often than the maximum number of retries, could not retry a transient
`UNAVAILABLE` of a later reattach. The call failed and the execution was
released.
- The error of a failing first call returned before the reattachable stream
existed, so no `ReleaseExecute` was sent, and the server kept the execution
until its detached timeout.
### Does this PR introduce _any_ user-facing change?
Yes. Long executions survive transient failures of their reattaches, and
executions that fail at once are released, as with PySpark 4.3.0.
### How was this patch tested?
- New tests against the mock Connect server of the transport's reattach
tests. The server can now fail the `ExecutePlan` call itself, and script the
results of `ReattachExecute`.
- An `ExecutePlan` call that fails at once is released once, with the
identity of the request.
- Three reattaches that each fail once with `UNAVAILABLE` read the whole
result, under a policy of two retries.
Both fail without this change.
- The lib tests and every integration test, e2e included, against a local
Spark 4.3.0 server (660 passed, and 125 of the transport crate), the build of
all tests with `wasm-udf`, the offline Python suites (886 passed, 9 skipped),
and the live drop-in and Arrow suites (81 passed) and live cases (10 passed).
Clippy reports no diagnostics in the changed lines.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
This pull request and its description were written by Isaac.
Closes #201 from HyukjinKwon/SPARK-60964-reattach-retries.
Authored-by: Hyukjin Kwon <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
---
crates/spark-connect-core/src/client.rs | 127 ++++++++++++---------
.../spark-connect-core/tests/reattach_cleanup.rs | 61 +++++++++-
2 files changed, 135 insertions(+), 53 deletions(-)
diff --git a/crates/spark-connect-core/src/client.rs
b/crates/spark-connect-core/src/client.rs
index 347c7a3..18df94b 100644
--- a/crates/spark-connect-core/src/client.rs
+++ b/crates/spark-connect-core/src/client.rs
@@ -407,12 +407,20 @@ impl SparkConnectClient {
let mut stub = self.stub.clone();
async move {
stub.execute_plan(req).await.map(Response::into_inner) }
})
- .await
- .map_err(|e| e.with_operation_id(iter.operation_id()))?;
+ .await;
+ let stream = match stream {
+ Ok(stream) => stream,
+ Err(error) => {
+ // As the reference iterator releases an execution that fails
at once, such
+ // as for an analysis error, so the server does not keep it
until its
+ // detached timeout.
+ self.spawn_release(self.execution_metadata(),
iter.release_all_request());
+ return Err(error.with_operation_id(iter.operation_id()));
+ }
+ };
let retry_policy = self.retry_policy.clone();
Ok(ReattachableResponseStream {
client: self.clone(),
- retry_state: RetryPolicyState::new(retry_policy.clone()),
retry_policy,
iter,
stream,
@@ -1013,7 +1021,6 @@ pub struct ReattachableResponseStream {
/// The client that started the execution, whose stub, headers and session
it uses.
client: SparkConnectClient,
retry_policy: RetryPolicy,
- retry_state: RetryPolicyState,
iter: ExecutePlanResponseReattachableIterator,
stream: Streaming<ExecutePlanResponse>,
done: bool,
@@ -1021,19 +1028,19 @@ pub struct ReattachableResponseStream {
released: bool,
}
-impl ReattachableResponseStream {
- /// The gRPC metadata of every request about this execution.
- fn metadata_map(&self) -> MetadataMap {
+impl SparkConnectClient {
+ /// The gRPC metadata of every request about an execution.
+ fn execution_metadata(&self) -> MetadataMap {
let mut metadata = MetadataMap::new();
- if let Ok(v) = MetadataValue::from_str(&self.client.user_agent) {
+ if let Ok(v) = MetadataValue::from_str(&self.user_agent) {
let _ = metadata.insert("user-agent", v);
}
- if let Some(user_id) = &self.client.user_id {
+ if let Some(user_id) = &self.user_id {
if let Ok(v) = MetadataValue::from_str(user_id) {
let _ = metadata.insert("user_id", v);
}
}
- for (k, v) in &self.client.metadata {
+ for (k, v) in &self.metadata {
if let Ok(v) = MetadataValue::from_str(v) {
if let Ok(key) =
tonic::metadata::MetadataKey::from_bytes(k.as_bytes()) {
let _ = metadata.insert(key, v);
@@ -1043,6 +1050,52 @@ impl ReattachableResponseStream {
metadata
}
+ /// Sends `request` in the background, so that reading is not delayed,
retrying under the
+ /// client's retry policy. Like the reference client, a failure is only
logged: the server
+ /// removes an execution that is never released once it has been detached
for
+ /// `spark.connect.execute.manager.detachedTimeout`.
+ fn spawn_release(&self, metadata: MetadataMap, request:
ReleaseExecuteRequest) {
+ let mut stub = self.stub.clone();
+ let policy = self.retry_policy.clone();
+ let release = async move {
+ let mut state = RetryPolicyState::new(policy.clone());
+ loop {
+ let mut req = Request::new(request.clone());
+ *req.metadata_mut() = metadata.clone();
+ match stub.release_execute(req).await {
+ Ok(_) => return,
+ Err(status) => {
+ if policy.can_retry(&status) {
+ if let Some(w) = state.next_attempt(None) {
+
tokio::time::sleep(Duration::from_millis(w)).await;
+ continue;
+ }
+ }
+ tracing::warn!(
+ "ReleaseExecute of operation {} failed: {}",
+ request.operation_id,
+ status
+ );
+ return;
+ }
+ }
+ }
+ };
+ // A stream is read inside a runtime, but it can be dropped outside of
any, for
+ // example after the `block_on` that read it.
+ match tokio::runtime::Handle::try_current() {
+ Ok(handle) => drop(handle.spawn(release)),
+ Err(_) => drop(crate::runtime::get_runtime().spawn(release)),
+ }
+ }
+}
+
+impl ReattachableResponseStream {
+ /// The gRPC metadata of every request about this execution.
+ fn metadata_map(&self) -> MetadataMap {
+ self.client.execution_metadata()
+ }
+
fn attach_metadata<T>(&self, req: &mut Request<T>) {
*req.metadata_mut() = self.metadata_map();
}
@@ -1053,6 +1106,9 @@ impl ReattachableResponseStream {
if self.done {
return Ok(None);
}
+ // As the reference iterator's `_has_next` retries with a `Retrying`
of its own, the
+ // reattaches of each response have the attempts and the elapsed time
of the policy.
+ let mut retry_state = RetryPolicyState::new(self.retry_policy.clone());
loop {
match self.stream.message().await {
Ok(Some(resp)) => {
@@ -1068,21 +1124,22 @@ impl ReattachableResponseStream {
// The server completes a reattachable stream before
`ResultComplete` on
// purpose, after `senderMaxStreamDuration` or
`senderMaxStreamSize`: the rest
// of the result is read by reattaching.
- Ok(None) => self.reattach().await?,
+ Ok(None) => self.reattach(&mut retry_state).await?,
Err(status) => {
// Only reattach a still-running execution on a retriable
drop.
if self.iter.is_completed().await ||
!self.retry_policy.can_retry(&status) {
let error =
self.client.error_from_status(status).await;
return Err(self.fail(error));
}
- self.reattach().await?;
+ self.reattach(&mut retry_state).await?;
}
}
}
}
- /// Resumes the execution from the last response read, retrying transient
failures.
- async fn reattach(&mut self) -> Result<()> {
+ /// Resumes the execution from the last response read, retrying transient
failures under
+ /// `retry_state`.
+ async fn reattach(&mut self, retry_state: &mut RetryPolicyState) ->
Result<()> {
loop {
let mut req =
Request::new(self.iter.create_reattach_request().await);
self.attach_metadata(&mut req);
@@ -1093,7 +1150,7 @@ impl ReattachableResponseStream {
}
Err(e) => {
if self.retry_policy.can_retry(&e) {
- if let Some(w) = self.retry_state.next_attempt(None) {
+ if let Some(w) = retry_state.next_attempt(None) {
tokio::time::sleep(Duration::from_millis(w)).await;
continue;
}
@@ -1129,44 +1186,10 @@ impl ReattachableResponseStream {
}
}
- /// Sends `request` in the background, so that reading is not delayed,
retrying under the
- /// client's retry policy. Like the reference client, a failure is only
logged: the server
- /// removes an execution that is never released once it has been detached
for
- /// `spark.connect.execute.manager.detachedTimeout`.
+ /// Sends `request` in the background, as
[`SparkConnectClient::spawn_release`] does.
fn spawn_release(&self, request: ReleaseExecuteRequest) {
- let mut stub = self.client.stub.clone();
- let metadata = self.metadata_map();
- let policy = self.retry_policy.clone();
- let release = async move {
- let mut state = RetryPolicyState::new(policy.clone());
- loop {
- let mut req = Request::new(request.clone());
- *req.metadata_mut() = metadata.clone();
- match stub.release_execute(req).await {
- Ok(_) => return,
- Err(status) => {
- if policy.can_retry(&status) {
- if let Some(w) = state.next_attempt(None) {
-
tokio::time::sleep(Duration::from_millis(w)).await;
- continue;
- }
- }
- tracing::warn!(
- "ReleaseExecute of operation {} failed: {}",
- request.operation_id,
- status
- );
- return;
- }
- }
- }
- };
- // A stream is read inside a runtime, but it can be dropped outside of
any, for
- // example after the `block_on` that read it.
- match tokio::runtime::Handle::try_current() {
- Ok(handle) => drop(handle.spawn(release)),
- Err(_) => drop(crate::runtime::get_runtime().spawn(release)),
- }
+ self.client
+ .spawn_release(self.client.execution_metadata(), request);
}
}
diff --git a/crates/spark-connect-core/tests/reattach_cleanup.rs
b/crates/spark-connect-core/tests/reattach_cleanup.rs
index 7655d73..9d15c0f 100644
--- a/crates/spark-connect-core/tests/reattach_cleanup.rs
+++ b/crates/spark-connect-core/tests/reattach_cleanup.rs
@@ -59,6 +59,9 @@ enum Scenario {
Error,
Eof,
Disconnect,
+ /// The ExecutePlan call fails itself, with a trailers-only response, as
for an
+ /// analysis error.
+ Reject,
}
#[derive(Clone)]
@@ -71,6 +74,9 @@ struct State {
scenario: Scenario,
releases: Mutex<Vec<CapturedRelease>>,
reattaches: Mutex<Vec<ReattachExecuteRequest>>,
+ /// What each ReattachExecute call does, in order: fail with a code, or
stream responses.
+ /// When none is left, a call completes the result.
+ reattach_script: Mutex<VecDeque<Result<Vec<ExecutePlanResponse>, Code>>>,
release_errors: Mutex<VecDeque<Code>>,
release_finished: AtomicUsize,
block_release: AtomicBool,
@@ -104,6 +110,7 @@ impl Fixture {
scenario,
releases: Mutex::new(Vec::new()),
reattaches: Mutex::new(Vec::new()),
+ reattach_script: Mutex::new(VecDeque::new()),
release_errors: Mutex::new(VecDeque::new()),
release_finished: AtomicUsize::new(0),
block_release: AtomicBool::new(false),
@@ -269,6 +276,9 @@ impl ServerStreamingService<ExecutePlanRequest> for Execute
{
type Future = BoxFuture<Response<Responses>, Status>;
fn call(&mut self, _: Request<ExecutePlanRequest>) -> Self::Future {
+ if let Scenario::Reject = self.0.scenario {
+ return Box::pin(async { Err(Status::invalid_argument("plan
sentinel")) });
+ }
let responses: Responses = match self.0.scenario {
Scenario::Complete =>
Box::pin(stream::iter([Ok(response("complete", true))])),
Scenario::Error =>
Box::pin(stream::iter([Err(Status::invalid_argument(
@@ -282,6 +292,7 @@ impl ServerStreamingService<ExecutePlanRequest> for Execute
{
Ok(response("first", false)),
Err(Status::unavailable("stream disconnected")),
])),
+ Scenario::Reject => unreachable!("rejected before streaming"),
};
Box::pin(async { Ok(Response::new(responses)) })
}
@@ -302,7 +313,13 @@ impl ServerStreamingService<ReattachExecuteRequest> for
Reattach {
if state.block_reattach.load(Ordering::SeqCst) {
state.reattach_gate.notified().await;
}
- let responses: Responses =
Box::pin(stream::iter([Ok(response("complete", true))]));
+ let scripted = state.reattach_script.lock().unwrap().pop_front();
+ let responses = match scripted {
+ Some(Err(code)) => return Err(Status::new(code, "reattach
sentinel")),
+ Some(Ok(responses)) => responses,
+ None => vec![response("complete", true)],
+ };
+ let responses: Responses =
Box::pin(stream::iter(responses.into_iter().map(Ok)));
Ok(Response::new(responses))
})
}
@@ -537,3 +554,45 @@ async fn
detached_stream_reattaches_before_releasing_the_execution() {
fixture.assert_release_all_count(1).await;
}
}
+
+#[tokio::test]
+async fn initial_failure_releases_the_execution() {
+ // As the reference iterator releases an execution whose first ExecutePlan
call fails,
+ // such as for an analysis error, rather than leave it until its detached
timeout.
+ let fixture = Fixture::start(Scenario::Reject).await;
+ let result =
bounded(fixture.client.execute_plan_reattachable(execute_request())).await;
+ let Err(error) = result else {
+ panic!("the rejected ExecutePlan returned a stream");
+ };
+ let error = error.to_string();
+ assert!(error.contains("plan sentinel"), "unexpected error: {error}");
+ fixture.assert_release_all_count(1).await;
+ assert_identity(&fixture.state.all_releases()[0]);
+}
+
+#[tokio::test]
+async fn each_response_retries_its_reattaches_afresh() {
+ // Each response read has the retries of the policy, as the reference
iterator's
+ // `_has_next` makes a `Retrying` of its own: three reattaches that fail
once each read
+ // the whole result, though the policy allows two retries.
+ let fixture = Fixture::start(Scenario::Eof).await;
+ {
+ let mut script = fixture.state.reattach_script.lock().unwrap();
+ for id in ["second", "third"] {
+ script.push_back(Err(Code::Unavailable));
+ script.push_back(Ok(vec![response(id, false)]));
+ }
+ script.push_back(Err(Code::Unavailable));
+ script.push_back(Ok(vec![response("complete", true)]));
+ }
+ let mut stream =
bounded(fixture.client.execute_plan_reattachable(execute_request()))
+ .await
+ .unwrap();
+ let mut ids = Vec::new();
+ while let Some(response) = bounded(stream.message()).await.unwrap() {
+ ids.push(response.response_id);
+ }
+ assert_eq!(ids, ["first", "second", "third", "complete"]);
+ assert_eq!(fixture.state.reattaches.lock().unwrap().len(), 6);
+ fixture.assert_release_all_count(1).await;
+}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]