This is an automated email from the ASF dual-hosted git repository.

HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git


The following commit(s) were added to refs/heads/master by this push:
     new bf7bd6a  [SPARK-60964][CONNECT] Retry each reattach afresh, and 
release an execution that fails at once, as PySpark does
bf7bd6a is described below

commit bf7bd6a4e384998556cf0916b1f545cb95d4352c
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Fri Nov 6 18:26:03 2026 +0900

    [SPARK-60964][CONNECT] Retry each reattach afresh, and release an execution 
that fails at once, as PySpark does
    
    ### What changes were proposed in this pull request?
    
    [SPARK-60964](https://issues.apache.org/jira/browse/SPARK-60964): two 
changes to the reattachable execution of the Rust transport, as PySpark's 
`ExecutePlanResponseReattachableIterator` does.
    
    - **Each response read has the retries of the policy afresh.** 
`ReattachableResponseStream::message` makes a retry state of its own, for the 
reattaches that it needs, as `_has_next` makes a `Retrying` of its own. Before, 
one retry state served the whole stream.
    - **An execution whose first `ExecutePlan` call fails is released.** An 
analysis or parse error is an example. `execute_plan_reattachable` sends 
`ReleaseExecute` (`ReleaseAll`) before it returns the error, as the iterator 
releases an execution on any error. The client now sends releases with the 
metadata of the execution, for a stream and for such a failure alike.
    
    ### Why are the changes needed?
    
    - The attempts of the shared retry state were never reset, and its 
elapsed-time limit counted from the start of the stream. So a query, a 
statement or the streaming listener that ran past the limit, or that reattached 
more often than the maximum number of retries, could not retry a transient 
`UNAVAILABLE` of a later reattach. The call failed and the execution was 
released.
    - The error of a failing first call returned before the reattachable stream 
existed, so no `ReleaseExecute` was sent, and the server kept the execution 
until its detached timeout.
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes. Long executions survive transient failures of their reattaches, and 
executions that fail at once are released, as with PySpark 4.3.0.
    
    ### How was this patch tested?
    
    - New tests against the mock Connect server of the transport's reattach 
tests. The server can now fail the `ExecutePlan` call itself, and script the 
results of `ReattachExecute`.
      - An `ExecutePlan` call that fails at once is released once, with the 
identity of the request.
      - Three reattaches that each fail once with `UNAVAILABLE` read the whole 
result, under a policy of two retries.
    
      Both fail without this change.
    - The lib tests and every integration test, e2e included, against a local 
Spark 4.3.0 server (660 passed, and 125 of the transport crate), the build of 
all tests with `wasm-udf`, the offline Python suites (886 passed, 9 skipped), 
and the live drop-in and Arrow suites (81 passed) and live cases (10 passed). 
Clippy reports no diagnostics in the changed lines.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Opus 5.5
    
    This pull request and its description were written by Isaac.
    
    Closes #201 from HyukjinKwon/SPARK-60964-reattach-retries.
    
    Authored-by: Hyukjin Kwon <[email protected]>
    Signed-off-by: Hyukjin Kwon <[email protected]>
---
 crates/spark-connect-core/src/client.rs            | 127 ++++++++++++---------
 .../spark-connect-core/tests/reattach_cleanup.rs   |  61 +++++++++-
 2 files changed, 135 insertions(+), 53 deletions(-)

diff --git a/crates/spark-connect-core/src/client.rs 
b/crates/spark-connect-core/src/client.rs
index 347c7a3..18df94b 100644
--- a/crates/spark-connect-core/src/client.rs
+++ b/crates/spark-connect-core/src/client.rs
@@ -407,12 +407,20 @@ impl SparkConnectClient {
                 let mut stub = self.stub.clone();
                 async move { 
stub.execute_plan(req).await.map(Response::into_inner) }
             })
-            .await
-            .map_err(|e| e.with_operation_id(iter.operation_id()))?;
+            .await;
+        let stream = match stream {
+            Ok(stream) => stream,
+            Err(error) => {
+                // As the reference iterator releases an execution that fails 
at once, such
+                // as for an analysis error, so the server does not keep it 
until its
+                // detached timeout.
+                self.spawn_release(self.execution_metadata(), 
iter.release_all_request());
+                return Err(error.with_operation_id(iter.operation_id()));
+            }
+        };
         let retry_policy = self.retry_policy.clone();
         Ok(ReattachableResponseStream {
             client: self.clone(),
-            retry_state: RetryPolicyState::new(retry_policy.clone()),
             retry_policy,
             iter,
             stream,
@@ -1013,7 +1021,6 @@ pub struct ReattachableResponseStream {
     /// The client that started the execution, whose stub, headers and session 
it uses.
     client: SparkConnectClient,
     retry_policy: RetryPolicy,
-    retry_state: RetryPolicyState,
     iter: ExecutePlanResponseReattachableIterator,
     stream: Streaming<ExecutePlanResponse>,
     done: bool,
@@ -1021,19 +1028,19 @@ pub struct ReattachableResponseStream {
     released: bool,
 }
 
-impl ReattachableResponseStream {
-    /// The gRPC metadata of every request about this execution.
-    fn metadata_map(&self) -> MetadataMap {
+impl SparkConnectClient {
+    /// The gRPC metadata of every request about an execution.
+    fn execution_metadata(&self) -> MetadataMap {
         let mut metadata = MetadataMap::new();
-        if let Ok(v) = MetadataValue::from_str(&self.client.user_agent) {
+        if let Ok(v) = MetadataValue::from_str(&self.user_agent) {
             let _ = metadata.insert("user-agent", v);
         }
-        if let Some(user_id) = &self.client.user_id {
+        if let Some(user_id) = &self.user_id {
             if let Ok(v) = MetadataValue::from_str(user_id) {
                 let _ = metadata.insert("user_id", v);
             }
         }
-        for (k, v) in &self.client.metadata {
+        for (k, v) in &self.metadata {
             if let Ok(v) = MetadataValue::from_str(v) {
                 if let Ok(key) = 
tonic::metadata::MetadataKey::from_bytes(k.as_bytes()) {
                     let _ = metadata.insert(key, v);
@@ -1043,6 +1050,52 @@ impl ReattachableResponseStream {
         metadata
     }
 
+    /// Sends `request` in the background, so that reading is not delayed, 
retrying under the
+    /// client's retry policy. Like the reference client, a failure is only 
logged: the server
+    /// removes an execution that is never released once it has been detached 
for
+    /// `spark.connect.execute.manager.detachedTimeout`.
+    fn spawn_release(&self, metadata: MetadataMap, request: 
ReleaseExecuteRequest) {
+        let mut stub = self.stub.clone();
+        let policy = self.retry_policy.clone();
+        let release = async move {
+            let mut state = RetryPolicyState::new(policy.clone());
+            loop {
+                let mut req = Request::new(request.clone());
+                *req.metadata_mut() = metadata.clone();
+                match stub.release_execute(req).await {
+                    Ok(_) => return,
+                    Err(status) => {
+                        if policy.can_retry(&status) {
+                            if let Some(w) = state.next_attempt(None) {
+                                
tokio::time::sleep(Duration::from_millis(w)).await;
+                                continue;
+                            }
+                        }
+                        tracing::warn!(
+                            "ReleaseExecute of operation {} failed: {}",
+                            request.operation_id,
+                            status
+                        );
+                        return;
+                    }
+                }
+            }
+        };
+        // A stream is read inside a runtime, but it can be dropped outside of 
any, for
+        // example after the `block_on` that read it.
+        match tokio::runtime::Handle::try_current() {
+            Ok(handle) => drop(handle.spawn(release)),
+            Err(_) => drop(crate::runtime::get_runtime().spawn(release)),
+        }
+    }
+}
+
+impl ReattachableResponseStream {
+    /// The gRPC metadata of every request about this execution.
+    fn metadata_map(&self) -> MetadataMap {
+        self.client.execution_metadata()
+    }
+
     fn attach_metadata<T>(&self, req: &mut Request<T>) {
         *req.metadata_mut() = self.metadata_map();
     }
@@ -1053,6 +1106,9 @@ impl ReattachableResponseStream {
         if self.done {
             return Ok(None);
         }
+        // As the reference iterator's `_has_next` retries with a `Retrying` 
of its own, the
+        // reattaches of each response have the attempts and the elapsed time 
of the policy.
+        let mut retry_state = RetryPolicyState::new(self.retry_policy.clone());
         loop {
             match self.stream.message().await {
                 Ok(Some(resp)) => {
@@ -1068,21 +1124,22 @@ impl ReattachableResponseStream {
                 // The server completes a reattachable stream before 
`ResultComplete` on
                 // purpose, after `senderMaxStreamDuration` or 
`senderMaxStreamSize`: the rest
                 // of the result is read by reattaching.
-                Ok(None) => self.reattach().await?,
+                Ok(None) => self.reattach(&mut retry_state).await?,
                 Err(status) => {
                     // Only reattach a still-running execution on a retriable 
drop.
                     if self.iter.is_completed().await || 
!self.retry_policy.can_retry(&status) {
                         let error = 
self.client.error_from_status(status).await;
                         return Err(self.fail(error));
                     }
-                    self.reattach().await?;
+                    self.reattach(&mut retry_state).await?;
                 }
             }
         }
     }
 
-    /// Resumes the execution from the last response read, retrying transient 
failures.
-    async fn reattach(&mut self) -> Result<()> {
+    /// Resumes the execution from the last response read, retrying transient 
failures under
+    /// `retry_state`.
+    async fn reattach(&mut self, retry_state: &mut RetryPolicyState) -> 
Result<()> {
         loop {
             let mut req = 
Request::new(self.iter.create_reattach_request().await);
             self.attach_metadata(&mut req);
@@ -1093,7 +1150,7 @@ impl ReattachableResponseStream {
                 }
                 Err(e) => {
                     if self.retry_policy.can_retry(&e) {
-                        if let Some(w) = self.retry_state.next_attempt(None) {
+                        if let Some(w) = retry_state.next_attempt(None) {
                             tokio::time::sleep(Duration::from_millis(w)).await;
                             continue;
                         }
@@ -1129,44 +1186,10 @@ impl ReattachableResponseStream {
         }
     }
 
-    /// Sends `request` in the background, so that reading is not delayed, 
retrying under the
-    /// client's retry policy. Like the reference client, a failure is only 
logged: the server
-    /// removes an execution that is never released once it has been detached 
for
-    /// `spark.connect.execute.manager.detachedTimeout`.
+    /// Sends `request` in the background, as 
[`SparkConnectClient::spawn_release`] does.
     fn spawn_release(&self, request: ReleaseExecuteRequest) {
-        let mut stub = self.client.stub.clone();
-        let metadata = self.metadata_map();
-        let policy = self.retry_policy.clone();
-        let release = async move {
-            let mut state = RetryPolicyState::new(policy.clone());
-            loop {
-                let mut req = Request::new(request.clone());
-                *req.metadata_mut() = metadata.clone();
-                match stub.release_execute(req).await {
-                    Ok(_) => return,
-                    Err(status) => {
-                        if policy.can_retry(&status) {
-                            if let Some(w) = state.next_attempt(None) {
-                                
tokio::time::sleep(Duration::from_millis(w)).await;
-                                continue;
-                            }
-                        }
-                        tracing::warn!(
-                            "ReleaseExecute of operation {} failed: {}",
-                            request.operation_id,
-                            status
-                        );
-                        return;
-                    }
-                }
-            }
-        };
-        // A stream is read inside a runtime, but it can be dropped outside of 
any, for
-        // example after the `block_on` that read it.
-        match tokio::runtime::Handle::try_current() {
-            Ok(handle) => drop(handle.spawn(release)),
-            Err(_) => drop(crate::runtime::get_runtime().spawn(release)),
-        }
+        self.client
+            .spawn_release(self.client.execution_metadata(), request);
     }
 }
 
diff --git a/crates/spark-connect-core/tests/reattach_cleanup.rs 
b/crates/spark-connect-core/tests/reattach_cleanup.rs
index 7655d73..9d15c0f 100644
--- a/crates/spark-connect-core/tests/reattach_cleanup.rs
+++ b/crates/spark-connect-core/tests/reattach_cleanup.rs
@@ -59,6 +59,9 @@ enum Scenario {
     Error,
     Eof,
     Disconnect,
+    /// The ExecutePlan call fails itself, with a trailers-only response, as 
for an
+    /// analysis error.
+    Reject,
 }
 
 #[derive(Clone)]
@@ -71,6 +74,9 @@ struct State {
     scenario: Scenario,
     releases: Mutex<Vec<CapturedRelease>>,
     reattaches: Mutex<Vec<ReattachExecuteRequest>>,
+    /// What each ReattachExecute call does, in order: fail with a code, or 
stream responses.
+    /// When none is left, a call completes the result.
+    reattach_script: Mutex<VecDeque<Result<Vec<ExecutePlanResponse>, Code>>>,
     release_errors: Mutex<VecDeque<Code>>,
     release_finished: AtomicUsize,
     block_release: AtomicBool,
@@ -104,6 +110,7 @@ impl Fixture {
             scenario,
             releases: Mutex::new(Vec::new()),
             reattaches: Mutex::new(Vec::new()),
+            reattach_script: Mutex::new(VecDeque::new()),
             release_errors: Mutex::new(VecDeque::new()),
             release_finished: AtomicUsize::new(0),
             block_release: AtomicBool::new(false),
@@ -269,6 +276,9 @@ impl ServerStreamingService<ExecutePlanRequest> for Execute 
{
     type Future = BoxFuture<Response<Responses>, Status>;
 
     fn call(&mut self, _: Request<ExecutePlanRequest>) -> Self::Future {
+        if let Scenario::Reject = self.0.scenario {
+            return Box::pin(async { Err(Status::invalid_argument("plan 
sentinel")) });
+        }
         let responses: Responses = match self.0.scenario {
             Scenario::Complete => 
Box::pin(stream::iter([Ok(response("complete", true))])),
             Scenario::Error => 
Box::pin(stream::iter([Err(Status::invalid_argument(
@@ -282,6 +292,7 @@ impl ServerStreamingService<ExecutePlanRequest> for Execute 
{
                 Ok(response("first", false)),
                 Err(Status::unavailable("stream disconnected")),
             ])),
+            Scenario::Reject => unreachable!("rejected before streaming"),
         };
         Box::pin(async { Ok(Response::new(responses)) })
     }
@@ -302,7 +313,13 @@ impl ServerStreamingService<ReattachExecuteRequest> for 
Reattach {
             if state.block_reattach.load(Ordering::SeqCst) {
                 state.reattach_gate.notified().await;
             }
-            let responses: Responses = 
Box::pin(stream::iter([Ok(response("complete", true))]));
+            let scripted = state.reattach_script.lock().unwrap().pop_front();
+            let responses = match scripted {
+                Some(Err(code)) => return Err(Status::new(code, "reattach 
sentinel")),
+                Some(Ok(responses)) => responses,
+                None => vec![response("complete", true)],
+            };
+            let responses: Responses = 
Box::pin(stream::iter(responses.into_iter().map(Ok)));
             Ok(Response::new(responses))
         })
     }
@@ -537,3 +554,45 @@ async fn 
detached_stream_reattaches_before_releasing_the_execution() {
         fixture.assert_release_all_count(1).await;
     }
 }
+
+#[tokio::test]
+async fn initial_failure_releases_the_execution() {
+    // As the reference iterator releases an execution whose first ExecutePlan 
call fails,
+    // such as for an analysis error, rather than leave it until its detached 
timeout.
+    let fixture = Fixture::start(Scenario::Reject).await;
+    let result = 
bounded(fixture.client.execute_plan_reattachable(execute_request())).await;
+    let Err(error) = result else {
+        panic!("the rejected ExecutePlan returned a stream");
+    };
+    let error = error.to_string();
+    assert!(error.contains("plan sentinel"), "unexpected error: {error}");
+    fixture.assert_release_all_count(1).await;
+    assert_identity(&fixture.state.all_releases()[0]);
+}
+
+#[tokio::test]
+async fn each_response_retries_its_reattaches_afresh() {
+    // Each response read has the retries of the policy, as the reference 
iterator's
+    // `_has_next` makes a `Retrying` of its own: three reattaches that fail 
once each read
+    // the whole result, though the policy allows two retries.
+    let fixture = Fixture::start(Scenario::Eof).await;
+    {
+        let mut script = fixture.state.reattach_script.lock().unwrap();
+        for id in ["second", "third"] {
+            script.push_back(Err(Code::Unavailable));
+            script.push_back(Ok(vec![response(id, false)]));
+        }
+        script.push_back(Err(Code::Unavailable));
+        script.push_back(Ok(vec![response("complete", true)]));
+    }
+    let mut stream = 
bounded(fixture.client.execute_plan_reattachable(execute_request()))
+        .await
+        .unwrap();
+    let mut ids = Vec::new();
+    while let Some(response) = bounded(stream.message()).await.unwrap() {
+        ids.push(response.response_id);
+    }
+    assert_eq!(ids, ["first", "second", "third", "complete"]);
+    assert_eq!(fixture.state.reattaches.lock().unwrap().len(), 6);
+    fixture.assert_release_all_count(1).await;
+}


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to