This is an automated email from the ASF dual-hosted git repository.

HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git


The following commit(s) were added to refs/heads/master by this push:
     new 0c1cde94 [SPARK-61149][CONNECT] Retry by the policy past the bound of 
RetryExceptions, take SESSION_NOT_FOUND and a response already received in 
reattach, and bound the config read of an error, as PySpark does
0c1cde94 is described below

commit 0c1cde946af73fa1c227f759e70ec73c799c035e
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Sun Nov 8 12:40:56 2026 +0900

    [SPARK-61149][CONNECT] Retry by the policy past the bound of 
RetryExceptions, take SESSION_NOT_FOUND and a response already received in 
reattach, and bound the config read of an error, as PySpark does
    
    ### What changes were proposed in this pull request?
    
    [SPARK-61149](https://issues.apache.org/jira/browse/SPARK-61149): the 
retries of the client follow PySpark's `RetryPolicyState`, `_call_iter` and the 
config read of an error.
    
    - `RetryPolicyState::next_attempt` checks only `max_retries`, as PySpark's 
`RetryPolicyState.next_attempt` does. The elapsed-time bound (one hour by 
default) stays on the retries that PySpark makes as `RetryException`s: the 
deadlines, and the reattaches to an unknown execution, which the client checks 
with `retry_exception_bound_passed`.
    - A reattach that finds the session or the operation unknown 
(`INVALID_HANDLE.SESSION_NOT_FOUND` as well as 
`INVALID_HANDLE.OPERATION_NOT_FOUND`) sends the ExecutePlan again when no 
response was read. Once a response was read, it raises 
`RESPONSE_ALREADY_RECEIVED` (`PySparkRuntimeError`, with `error_type` set to 
the code), as PySpark's `_call_iter` does. Before, only the operation code was 
handled, and after a response it surfaced the server error.
    - The config read that shows the stack trace of a server error is bounded 
by `rpc_deadlines.config`, as PySpark's `RuntimeConf.get` is. Before, a 
connection that never answered hung the call that should raise the error.
    
    ### Why are the changes needed?
    
    - A transient `UNAVAILABLE` on a reattach more than one hour into a single 
`message()` call failed at once, with retries left in the policy, because 
`next_attempt` checked the RetryException bound too.
    - A reattach after a response, to an execution whose session the server no 
longer knows, raised the server's `INVALID_HANDLE.SESSION_NOT_FOUND` error, 
where PySpark raises `RESPONSE_ALREADY_RECEIVED`. Before any response, the 
execution was not sent again.
    - Raising a server error with stack-trace details sent an unbounded 
`Config` call.
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes, in error cases only: a policy retry is no longer cut by the one-hour 
bound, an unknown session after a response raises `PySparkRuntimeError` with 
`RESPONSE_ALREADY_RECEIVED` (as PySpark), and a server error no longer waits 
forever on the config read.
    
    ### How was this patch tested?
    
    - Rust `test_elapsed_time_ceiling_does_not_cut_policy_retries` and 
`test_policy_retries_after_the_elapsed_bound_until_max_retries` in `retries.rs` 
(the policy keeps its `max_retries` budget past the elapsed bound).
    - Rust `policy_retry_of_a_reattach_is_not_cut_by_the_retry_exception_bound` 
in `deadlines.rs`: a reattach that fails with `UNAVAILABLE` is retried under a 
zero RetryException bound and returns the results.
    - Rust `session_not_found_before_any_response_resends_the_execute_plan` in 
`deadlines.rs`: the ExecutePlan is sent again, and its results are returned.
    - Rust `session_not_found_after_a_response_is_response_already_received` 
and 
`session_not_found_in_the_reattach_call_after_a_response_is_response_already_received`,
 and the updated 
`operation_not_found_after_a_response_is_the_error_and_is_not_resent`: 
`RESPONSE_ALREADY_RECEIVED` with the matched `error_type`, and no second 
ExecutePlan.
    - Rust `error_of_a_server_with_details_bounds_the_config_read` in 
`deadlines.rs`: the error is raised with a `Config` call that carries its 
`grpc-timeout`, while the config does not answer.
    - The tests of both crates, e2e included, against a local Spark 4.3.0 
server (881 passed), the build of all tests, also with `wasm-udf`, the offline 
Python suites (2201 passed, 8 skipped), and the live drop-in, Arrow and ML 
suites (184 passed) and live cases (10 passed). rustfmt passes, and clippy 
reports no diagnostics in the changed lines.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Opus 5.5
    
    This pull request and its description were written by Isaac.
    
    Closes #339 from HyukjinKwon/SPARK-61149-retries-reattach-config-deadline.
    
    Authored-by: Hyukjin Kwon <[email protected]>
    Signed-off-by: Hyukjin Kwon <[email protected]>
---
 crates/spark-connect-core/src/client.rs      |  55 +++++---
 crates/spark-connect-core/src/retries.rs     |  49 ++++---
 crates/spark-connect-core/tests/deadlines.rs | 183 ++++++++++++++++++++++++++-
 3 files changed, 253 insertions(+), 34 deletions(-)

diff --git a/crates/spark-connect-core/src/client.rs 
b/crates/spark-connect-core/src/client.rs
index a4d0a6a7..84b4d2eb 100644
--- a/crates/spark-connect-core/src/client.rs
+++ b/crates/spark-connect-core/src/client.rs
@@ -24,7 +24,7 @@ use crate::artifact::{
     FileArtifact, NamedArtifact,
 };
 use crate::channel::{ChannelBuilder, GRPC_MAX_MESSAGE_LENGTH_DEFAULT};
-use crate::error::{Result, SparkError};
+use crate::error::{Result, SparkError, SparkErrorKind};
 use crate::reattach::ExecutePlanResponseReattachableIterator;
 use crate::retries::{RetryPolicy, RetryPolicyState};
 
@@ -186,12 +186,26 @@ async fn within_deadline<T>(
     }
 }
 
-/// Whether `status` says that the server does not know the operation of a 
reattach, which
-/// the reference reads as the initial ExecutePlan never having reached the 
server.
-fn is_operation_not_found(status: &tonic::Status) -> bool {
-    status
-        .message()
-        .contains("INVALID_HANDLE.OPERATION_NOT_FOUND")
+/// The error class of `status` when the server does not know the operation of 
a reattach, or
+/// its session, which the reference reads alike: the initial ExecutePlan 
never reached the
+/// server, unless a response was read already.
+fn unknown_operation(status: &tonic::Status) -> Option<&'static str> {
+    [
+        "INVALID_HANDLE.OPERATION_NOT_FOUND",
+        "INVALID_HANDLE.SESSION_NOT_FOUND",
+    ]
+    .into_iter()
+    .find(|error| status.message().contains(error))
+}
+
+/// The error of a reattach to an operation that the server does not know, 
once a response was
+/// read: `RESPONSE_ALREADY_RECEIVED`, with the error class of the server as 
`error_type`.
+fn response_already_received(error_type: &str) -> SparkError {
+    SparkError::classed(
+        SparkErrorKind::RuntimeError,
+        "RESPONSE_ALREADY_RECEIVED",
+        &[("error_type", error_type)],
+    )
 }
 
 /// Whether the retries that the reference makes as `RetryException`s have run 
for the policy's
@@ -871,9 +885,13 @@ impl SparkConnectClient {
             }),
             ..Default::default()
         };
-        let mut req = Request::new(request);
+        let deadline = self.rpc_deadlines.config;
+        let mut req = with_deadline(Request::new(request), deadline);
         self._attach_metadata(&mut req);
-        let response = self.stub.clone().config(req).await.ok()?.into_inner();
+        let response = within_deadline(deadline, self.stub.clone().config(req))
+            .await
+            .ok()?
+            .into_inner();
         response.pairs.into_iter().next()?.value
     }
 
@@ -1655,14 +1673,16 @@ impl ReattachableResponseStream {
                     }
                     self.stream = None;
                 }
-                // As the reference iterator: the server does not know the 
execution, so the
-                // initial ExecutePlan never reached it, and it is sent again, 
if no response was
-                // read yet.
+                // As the reference iterator: the server does not know the 
execution (or its
+                // session), so the initial ExecutePlan never reached it, and 
it is sent again,
+                // if no response was read yet. Once one was, the reference 
raises an error.
                 Err(status)
-                    if is_operation_not_found(&status)
-                        && self.iter.last_response_id().await.is_none()
-                        && !self.iter.is_completed().await =>
+                    if unknown_operation(&status).is_some() && 
!self.iter.is_completed().await =>
                 {
+                    if self.iter.last_response_id().await.is_some() {
+                        let error_type = 
unknown_operation(&status).unwrap_or_default();
+                        return 
Err(self.fail(response_already_received(error_type)));
+                    }
                     if retry_exception_bound_passed(&self.retry_policy, &mut 
retry_exception_since)
                     {
                         return Err(self.fail_with_status(status).await);
@@ -1705,7 +1725,10 @@ impl ReattachableResponseStream {
                 }
                 Err(status) => status,
             };
-            if is_operation_not_found(&status) && 
self.iter.last_response_id().await.is_none() {
+            if let Some(error_type) = unknown_operation(&status) {
+                if self.iter.last_response_id().await.is_some() {
+                    return 
Err(self.fail(response_already_received(error_type)));
+                }
                 if retry_exception_bound_passed(&self.retry_policy, 
retry_exception_since) {
                     return Err(self.fail_with_status(status).await);
                 }
diff --git a/crates/spark-connect-core/src/retries.rs 
b/crates/spark-connect-core/src/retries.rs
index b545201c..75eaaaac 100644
--- a/crates/spark-connect-core/src/retries.rs
+++ b/crates/spark-connect-core/src/retries.rs
@@ -27,8 +27,9 @@ pub struct RetryPolicy {
     pub recognize_server_retry_delay: bool,
     /// Maximum server-provided retry delay (in milliseconds).
     pub max_server_retry_delay_ms: Option<u64>,
-    /// Maximum cumulative elapsed time for retries (in milliseconds).
-    /// Defaults to 1 hour. Set to None to disable the ceiling.
+    /// Maximum cumulative elapsed time for the retries of a `RetryException`, 
in milliseconds.
+    /// Defaults to 1 hour. Set to None to disable the ceiling. Not applied to 
the retries of a
+    /// policy, as the reference's `RetryPolicyState` does not.
     pub max_retry_exception_elapsed_time_ms: Option<u64>,
 }
 
@@ -85,7 +86,6 @@ pub struct RetryPolicyState {
     policy: RetryPolicy,
     attempt: u32,
     next_wait_ms: f64,
-    started: std::time::Instant,
 }
 
 impl RetryPolicyState {
@@ -94,22 +94,16 @@ impl RetryPolicyState {
             next_wait_ms: policy.initial_backoff_ms as f64,
             policy,
             attempt: 0,
-            started: std::time::Instant::now(),
         }
     }
 
     /// Compute the wait time before the next retry.
     ///
     /// Returns the number of milliseconds to wait, or None if no more retries 
are allowed.
-    /// This is deterministic and doesn't perform any actual sleeping.
+    /// This is deterministic and doesn't perform any actual sleeping. As the 
reference's
+    /// `RetryPolicyState.next_attempt`, only `max_retries` bounds the retries 
here; the elapsed
+    /// time bounds the retries of a `RetryException`, which the client checks 
itself.
     pub fn next_attempt(&mut self, server_retry_delay_ms: Option<u64>) -> 
Option<u64> {
-        // Check if we've exceeded the elapsed-time ceiling
-        if let Some(max_ms) = self.policy.max_retry_exception_elapsed_time_ms {
-            if self.started.elapsed().as_millis() as u64 >= max_ms {
-                return None;
-            }
-        }
-
         // Check if we've exhausted the retry budget
         if let Some(max) = self.policy.max_retries {
             if self.attempt >= max {
@@ -388,9 +382,10 @@ mod tests {
     }
 
     #[test]
-    fn test_elapsed_time_ceiling_exceeded() {
-        // With max_retry_exception_elapsed_time_ms: Some(0), next_attempt 
should
-        // immediately return None, enforcing the elapsed-time ceiling 
deterministically.
+    fn test_elapsed_time_ceiling_does_not_cut_policy_retries() {
+        // With max_retry_exception_elapsed_time_ms: Some(0), the elapsed 
bound of the
+        // RetryExceptions has passed at once, but a policy retry is bounded 
by max_retries
+        // alone, as the reference's RetryPolicyState: the retry is allowed.
         let policy = RetryPolicy {
             max_retries: Some(10),
             initial_backoff_ms: 50,
@@ -404,7 +399,29 @@ mod tests {
         };
 
         let mut state = RetryPolicyState::new(policy);
-        // Even with max_retries remaining, elapsed time ceiling exceeded so 
None.
+        assert_eq!(state.next_attempt(None), Some(50));
+        assert_eq!(state.attempt(), 1);
+    }
+
+    #[test]
+    fn test_policy_retries_after_the_elapsed_bound_until_max_retries() {
+        // A wait past the elapsed bound: the retries of the policy are still 
those of max_retries.
+        let policy = RetryPolicy {
+            max_retries: Some(2),
+            initial_backoff_ms: 50,
+            max_backoff_ms: None,
+            backoff_multiplier: 1.0,
+            jitter_ms: 0,
+            min_jitter_threshold_ms: 0,
+            recognize_server_retry_delay: false,
+            max_server_retry_delay_ms: None,
+            max_retry_exception_elapsed_time_ms: Some(1),
+        };
+
+        let mut state = RetryPolicyState::new(policy);
+        std::thread::sleep(std::time::Duration::from_millis(5));
+        assert_eq!(state.next_attempt(None), Some(50));
+        assert_eq!(state.next_attempt(None), Some(50));
         assert_eq!(state.next_attempt(None), None);
     }
 
diff --git a/crates/spark-connect-core/tests/deadlines.rs 
b/crates/spark-connect-core/tests/deadlines.rs
index 0a4282fa..a8fee2e3 100644
--- a/crates/spark-connect-core/tests/deadlines.rs
+++ b/crates/spark-connect-core/tests/deadlines.rs
@@ -72,6 +72,13 @@ fn operation_not_found() -> Status {
     ))
 }
 
+fn session_not_found() -> Status {
+    // As the server reports a session it does not know, with the same shape.
+    Status::internal(
+        "[INVALID_HANDLE.SESSION_NOT_FOUND] Session not found: 
00000000-0000-4000-8000-000000000001",
+    )
+}
+
 #[derive(Clone)]
 struct CapturedRelease {
     request: ReleaseExecuteRequest,
@@ -452,14 +459,186 @@ async fn 
operation_not_found_after_a_response_is_the_error_and_is_not_resent() {
         "r1"
     );
     let error = bounded(stream.message()).await.unwrap_err();
-    assert!(
-        error.message.contains("INVALID_HANDLE.OPERATION_NOT_FOUND"),
+    assert_response_already_received(&error, 
"INVALID_HANDLE.OPERATION_NOT_FOUND");
+    assert_eq!(*fixture.state.executes.lock().unwrap(), 1);
+    fixture.assert_release_all_count(1).await;
+}
+
+/// The error of a reattach to an execution that the server does not know 
after a response was
+/// read, as the reference: `RESPONSE_ALREADY_RECEIVED` of a 
`PySparkRuntimeError`.
+fn assert_response_already_received(error: &spark_connect_core::SparkError, 
error_type: &str) {
+    assert_eq!(
+        error.kind,
+        spark_connect_core::error::SparkErrorKind::RuntimeError,
         "{error:?}"
     );
+    assert_eq!(error.error_class, "RESPONSE_ALREADY_RECEIVED", "{error:?}");
+    assert_eq!(
+        error.params.get("error_type").map(String::as_str),
+        Some(error_type),
+        "{error:?}"
+    );
+}
+
+#[tokio::test]
+async fn session_not_found_before_any_response_resends_the_execute_plan() {
+    // As the reference, which reads SESSION_NOT_FOUND as OPERATION_NOT_FOUND: 
no response was
+    // read, so the ExecutePlan is sent again, and its results are returned.
+    let fixture = Fixture::start(
+        vec![
+            streamed(vec![]),
+            streamed(vec![response("r1", false), response("r2", true)]),
+        ],
+        vec![Step::Stream(vec![Err(session_not_found())])],
+        quick_retries(),
+    )
+    .await;
+    let mut stream = 
bounded(fixture.client.execute_plan_reattachable(execute_request()))
+        .await
+        .unwrap();
+    let ids = drain(&mut stream).await.unwrap();
+    assert_eq!(ids, ["r1", "r2"]);
+
+    assert_eq!(*fixture.state.executes.lock().unwrap(), 2);
+    assert_eq!(fixture.state.reattaches.lock().unwrap().len(), 1);
+    fixture.assert_release_all_count(1).await;
+}
+
+#[tokio::test]
+async fn session_not_found_after_a_response_is_response_already_received() {
+    // As the reference: once a response was read, SESSION_NOT_FOUND is the 
error of the reattach,
+    // not a reason to send the execution again.
+    let fixture = Fixture::start(
+        vec![streamed(vec![response("r1", false)])],
+        vec![Step::Stream(vec![Err(session_not_found())])],
+        quick_retries(),
+    )
+    .await;
+    let mut stream = 
bounded(fixture.client.execute_plan_reattachable(execute_request()))
+        .await
+        .unwrap();
+    assert_eq!(
+        bounded(stream.message())
+            .await
+            .unwrap()
+            .unwrap()
+            .response_id,
+        "r1"
+    );
+    let error = bounded(stream.message()).await.unwrap_err();
+    assert_response_already_received(&error, 
"INVALID_HANDLE.SESSION_NOT_FOUND");
     assert_eq!(*fixture.state.executes.lock().unwrap(), 1);
     fixture.assert_release_all_count(1).await;
 }
 
+#[tokio::test]
+async fn 
session_not_found_in_the_reattach_call_after_a_response_is_response_already_received()
 {
+    // As the reference, for the reattach call itself failing with 
SESSION_NOT_FOUND.
+    let fixture = Fixture::start(
+        vec![streamed(vec![response("r1", false)])],
+        vec![Step::Fail(session_not_found())],
+        quick_retries(),
+    )
+    .await;
+    let mut stream = 
bounded(fixture.client.execute_plan_reattachable(execute_request()))
+        .await
+        .unwrap();
+    assert_eq!(
+        bounded(stream.message())
+            .await
+            .unwrap()
+            .unwrap()
+            .response_id,
+        "r1"
+    );
+    let error = bounded(stream.message()).await.unwrap_err();
+    assert_response_already_received(&error, 
"INVALID_HANDLE.SESSION_NOT_FOUND");
+    assert_eq!(*fixture.state.executes.lock().unwrap(), 1);
+    fixture.assert_release_all_count(1).await;
+}
+
+#[tokio::test]
+async fn policy_retry_of_a_reattach_is_not_cut_by_the_retry_exception_bound() {
+    // As the reference's RetryPolicyState: the elapsed bound of the 
RetryExceptions (here 0, as
+    // if an hour had passed) does not cut a retry of the policy. The reattach 
call fails with
+    // UNAVAILABLE, and it is retried with the policy's attempts, here 
reattaching with success.
+    let fixture = Fixture::start(
+        vec![streamed(vec![response("r1", false)])],
+        vec![Step::Fail(Status::unavailable("server restarting"))],
+        RetryPolicy {
+            max_retry_exception_elapsed_time_ms: Some(0),
+            ..quick_retries()
+        },
+    )
+    .await;
+    let mut stream = 
bounded(fixture.client.execute_plan_reattachable(execute_request()))
+        .await
+        .unwrap();
+    let ids = drain(&mut stream).await.unwrap();
+    assert_eq!(ids, ["r1", "complete"]);
+
+    let reattaches = fixture.state.reattaches.lock().unwrap().clone();
+    assert_eq!(reattaches.len(), 2);
+    assert!(reattaches
+        .iter()
+        .all(|r| r.last_response_id.as_deref() == Some("r1")));
+    assert_eq!(*fixture.state.executes.lock().unwrap(), 1);
+    fixture.assert_release_all_count(1).await;
+}
+
+#[tokio::test]
+async fn error_of_a_server_with_details_bounds_the_config_read() {
+    // As the reference: the stack trace of an error is shown as its config 
says, read with the
+    // config deadline. A config that never answers is not waited for, and the 
error is raised.
+    let fixture = Fixture::start(vec![], vec![], quick_retries()).await;
+    let status = Status::with_details(Code::Internal, "boom", 
error_info_details().into());
+    let error = bounded(fixture.client.error_from_status(status)).await;
+    assert!(error.message.contains("boom"), "{error:?}");
+
+    let timeouts = fixture.state.timeouts.lock().unwrap().clone();
+    assert!(
+        timeouts
+            .iter()
+            .any(|(call, timeout)| *call == "Config" && timeout.is_some()),
+        "{timeouts:?}"
+    );
+}
+
+/// A `google.rpc.Status` with an `ErrorInfo` detail, as the server sends the 
errors it raises.
+fn error_info_details() -> Vec<u8> {
+    use prost::Message;
+
+    #[derive(Clone, PartialEq, prost::Message)]
+    struct ErrorInfo {
+        #[prost(string, tag = "1")]
+        reason: String,
+        #[prost(string, tag = "2")]
+        domain: String,
+    }
+    #[derive(Clone, PartialEq, prost::Message)]
+    struct RpcStatus {
+        #[prost(int32, tag = "1")]
+        code: i32,
+        #[prost(string, tag = "2")]
+        message: String,
+        #[prost(message, repeated, tag = "3")]
+        details: Vec<prost_types::Any>,
+    }
+    RpcStatus {
+        code: Code::Internal as i32,
+        message: "boom".into(),
+        details: vec![prost_types::Any {
+            type_url: "type.googleapis.com/google.rpc.ErrorInfo".into(),
+            value: ErrorInfo {
+                reason: "BOOM".into(),
+                domain: "org.apache.spark".into(),
+            }
+            .encode_to_vec(),
+        }],
+    }
+    .encode_to_vec()
+}
+
 /// The loopback service: ExecutePlan, ReattachExecute, ReleaseExecute and 
Config.
 #[derive(Clone)]
 struct TestService(Arc<State>);


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to