This is an automated email from the ASF dual-hosted git repository.
HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git
The following commit(s) were added to refs/heads/master by this push:
new 0c1cde94 [SPARK-61149][CONNECT] Retry by the policy past the bound of
RetryExceptions, take SESSION_NOT_FOUND and a response already received in
reattach, and bound the config read of an error, as PySpark does
0c1cde94 is described below
commit 0c1cde946af73fa1c227f759e70ec73c799c035e
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Sun Nov 8 12:40:56 2026 +0900
[SPARK-61149][CONNECT] Retry by the policy past the bound of
RetryExceptions, take SESSION_NOT_FOUND and a response already received in
reattach, and bound the config read of an error, as PySpark does
### What changes were proposed in this pull request?
[SPARK-61149](https://issues.apache.org/jira/browse/SPARK-61149): the
retries of the client follow PySpark's `RetryPolicyState`, `_call_iter` and the
config read of an error.
- `RetryPolicyState::next_attempt` checks only `max_retries`, as PySpark's
`RetryPolicyState.next_attempt` does. The elapsed-time bound (one hour by
default) stays on the retries that PySpark makes as `RetryException`s: the
deadlines, and the reattaches to an unknown execution, which the client checks
with `retry_exception_bound_passed`.
- A reattach that finds the session or the operation unknown
(`INVALID_HANDLE.SESSION_NOT_FOUND` as well as
`INVALID_HANDLE.OPERATION_NOT_FOUND`) sends the ExecutePlan again when no
response was read. Once a response was read, it raises
`RESPONSE_ALREADY_RECEIVED` (`PySparkRuntimeError`, with `error_type` set to
the code), as PySpark's `_call_iter` does. Before, only the operation code was
handled, and after a response it surfaced the server error.
- The config read that shows the stack trace of a server error is bounded
by `rpc_deadlines.config`, as PySpark's `RuntimeConf.get` is. Before, a
connection that never answered hung the call that should raise the error.
### Why are the changes needed?
- A transient `UNAVAILABLE` on a reattach more than one hour into a single
`message()` call failed at once, with retries left in the policy, because
`next_attempt` checked the RetryException bound too.
- A reattach after a response, to an execution whose session the server no
longer knows, raised the server's `INVALID_HANDLE.SESSION_NOT_FOUND` error,
where PySpark raises `RESPONSE_ALREADY_RECEIVED`. Before any response, the
execution was not sent again.
- Raising a server error with stack-trace details sent an unbounded
`Config` call.
### Does this PR introduce _any_ user-facing change?
Yes, in error cases only: a policy retry is no longer cut by the one-hour
bound, an unknown session after a response raises `PySparkRuntimeError` with
`RESPONSE_ALREADY_RECEIVED` (as PySpark), and a server error no longer waits
forever on the config read.
### How was this patch tested?
- Rust `test_elapsed_time_ceiling_does_not_cut_policy_retries` and
`test_policy_retries_after_the_elapsed_bound_until_max_retries` in `retries.rs`
(the policy keeps its `max_retries` budget past the elapsed bound).
- Rust `policy_retry_of_a_reattach_is_not_cut_by_the_retry_exception_bound`
in `deadlines.rs`: a reattach that fails with `UNAVAILABLE` is retried under a
zero RetryException bound and returns the results.
- Rust `session_not_found_before_any_response_resends_the_execute_plan` in
`deadlines.rs`: the ExecutePlan is sent again, and its results are returned.
- Rust `session_not_found_after_a_response_is_response_already_received`
and
`session_not_found_in_the_reattach_call_after_a_response_is_response_already_received`,
and the updated
`operation_not_found_after_a_response_is_the_error_and_is_not_resent`:
`RESPONSE_ALREADY_RECEIVED` with the matched `error_type`, and no second
ExecutePlan.
- Rust `error_of_a_server_with_details_bounds_the_config_read` in
`deadlines.rs`: the error is raised with a `Config` call that carries its
`grpc-timeout`, while the config does not answer.
- The tests of both crates, e2e included, against a local Spark 4.3.0
server (881 passed), the build of all tests, also with `wasm-udf`, the offline
Python suites (2201 passed, 8 skipped), and the live drop-in, Arrow and ML
suites (184 passed) and live cases (10 passed). rustfmt passes, and clippy
reports no diagnostics in the changed lines.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
This pull request and its description were written by Isaac.
Closes #339 from HyukjinKwon/SPARK-61149-retries-reattach-config-deadline.
Authored-by: Hyukjin Kwon <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
---
crates/spark-connect-core/src/client.rs | 55 +++++---
crates/spark-connect-core/src/retries.rs | 49 ++++---
crates/spark-connect-core/tests/deadlines.rs | 183 ++++++++++++++++++++++++++-
3 files changed, 253 insertions(+), 34 deletions(-)
diff --git a/crates/spark-connect-core/src/client.rs
b/crates/spark-connect-core/src/client.rs
index a4d0a6a7..84b4d2eb 100644
--- a/crates/spark-connect-core/src/client.rs
+++ b/crates/spark-connect-core/src/client.rs
@@ -24,7 +24,7 @@ use crate::artifact::{
FileArtifact, NamedArtifact,
};
use crate::channel::{ChannelBuilder, GRPC_MAX_MESSAGE_LENGTH_DEFAULT};
-use crate::error::{Result, SparkError};
+use crate::error::{Result, SparkError, SparkErrorKind};
use crate::reattach::ExecutePlanResponseReattachableIterator;
use crate::retries::{RetryPolicy, RetryPolicyState};
@@ -186,12 +186,26 @@ async fn within_deadline<T>(
}
}
-/// Whether `status` says that the server does not know the operation of a
reattach, which
-/// the reference reads as the initial ExecutePlan never having reached the
server.
-fn is_operation_not_found(status: &tonic::Status) -> bool {
- status
- .message()
- .contains("INVALID_HANDLE.OPERATION_NOT_FOUND")
+/// The error class of `status` when the server does not know the operation of
a reattach, or
+/// its session, which the reference reads alike: the initial ExecutePlan
never reached the
+/// server, unless a response was read already.
+fn unknown_operation(status: &tonic::Status) -> Option<&'static str> {
+ [
+ "INVALID_HANDLE.OPERATION_NOT_FOUND",
+ "INVALID_HANDLE.SESSION_NOT_FOUND",
+ ]
+ .into_iter()
+ .find(|error| status.message().contains(error))
+}
+
+/// The error of a reattach to an operation that the server does not know,
once a response was
+/// read: `RESPONSE_ALREADY_RECEIVED`, with the error class of the server as
`error_type`.
+fn response_already_received(error_type: &str) -> SparkError {
+ SparkError::classed(
+ SparkErrorKind::RuntimeError,
+ "RESPONSE_ALREADY_RECEIVED",
+ &[("error_type", error_type)],
+ )
}
/// Whether the retries that the reference makes as `RetryException`s have run
for the policy's
@@ -871,9 +885,13 @@ impl SparkConnectClient {
}),
..Default::default()
};
- let mut req = Request::new(request);
+ let deadline = self.rpc_deadlines.config;
+ let mut req = with_deadline(Request::new(request), deadline);
self._attach_metadata(&mut req);
- let response = self.stub.clone().config(req).await.ok()?.into_inner();
+ let response = within_deadline(deadline, self.stub.clone().config(req))
+ .await
+ .ok()?
+ .into_inner();
response.pairs.into_iter().next()?.value
}
@@ -1655,14 +1673,16 @@ impl ReattachableResponseStream {
}
self.stream = None;
}
- // As the reference iterator: the server does not know the
execution, so the
- // initial ExecutePlan never reached it, and it is sent again,
if no response was
- // read yet.
+ // As the reference iterator: the server does not know the
execution (or its
+ // session), so the initial ExecutePlan never reached it, and
it is sent again,
+ // if no response was read yet. Once one was, the reference
raises an error.
Err(status)
- if is_operation_not_found(&status)
- && self.iter.last_response_id().await.is_none()
- && !self.iter.is_completed().await =>
+ if unknown_operation(&status).is_some() &&
!self.iter.is_completed().await =>
{
+ if self.iter.last_response_id().await.is_some() {
+ let error_type =
unknown_operation(&status).unwrap_or_default();
+ return
Err(self.fail(response_already_received(error_type)));
+ }
if retry_exception_bound_passed(&self.retry_policy, &mut
retry_exception_since)
{
return Err(self.fail_with_status(status).await);
@@ -1705,7 +1725,10 @@ impl ReattachableResponseStream {
}
Err(status) => status,
};
- if is_operation_not_found(&status) &&
self.iter.last_response_id().await.is_none() {
+ if let Some(error_type) = unknown_operation(&status) {
+ if self.iter.last_response_id().await.is_some() {
+ return
Err(self.fail(response_already_received(error_type)));
+ }
if retry_exception_bound_passed(&self.retry_policy,
retry_exception_since) {
return Err(self.fail_with_status(status).await);
}
diff --git a/crates/spark-connect-core/src/retries.rs
b/crates/spark-connect-core/src/retries.rs
index b545201c..75eaaaac 100644
--- a/crates/spark-connect-core/src/retries.rs
+++ b/crates/spark-connect-core/src/retries.rs
@@ -27,8 +27,9 @@ pub struct RetryPolicy {
pub recognize_server_retry_delay: bool,
/// Maximum server-provided retry delay (in milliseconds).
pub max_server_retry_delay_ms: Option<u64>,
- /// Maximum cumulative elapsed time for retries (in milliseconds).
- /// Defaults to 1 hour. Set to None to disable the ceiling.
+ /// Maximum cumulative elapsed time for the retries of a `RetryException`,
in milliseconds.
+ /// Defaults to 1 hour. Set to None to disable the ceiling. Not applied to
the retries of a
+ /// policy, as the reference's `RetryPolicyState` does not.
pub max_retry_exception_elapsed_time_ms: Option<u64>,
}
@@ -85,7 +86,6 @@ pub struct RetryPolicyState {
policy: RetryPolicy,
attempt: u32,
next_wait_ms: f64,
- started: std::time::Instant,
}
impl RetryPolicyState {
@@ -94,22 +94,16 @@ impl RetryPolicyState {
next_wait_ms: policy.initial_backoff_ms as f64,
policy,
attempt: 0,
- started: std::time::Instant::now(),
}
}
/// Compute the wait time before the next retry.
///
/// Returns the number of milliseconds to wait, or None if no more retries
are allowed.
- /// This is deterministic and doesn't perform any actual sleeping.
+ /// This is deterministic and doesn't perform any actual sleeping. As the
reference's
+ /// `RetryPolicyState.next_attempt`, only `max_retries` bounds the retries
here; the elapsed
+ /// time bounds the retries of a `RetryException`, which the client checks
itself.
pub fn next_attempt(&mut self, server_retry_delay_ms: Option<u64>) ->
Option<u64> {
- // Check if we've exceeded the elapsed-time ceiling
- if let Some(max_ms) = self.policy.max_retry_exception_elapsed_time_ms {
- if self.started.elapsed().as_millis() as u64 >= max_ms {
- return None;
- }
- }
-
// Check if we've exhausted the retry budget
if let Some(max) = self.policy.max_retries {
if self.attempt >= max {
@@ -388,9 +382,10 @@ mod tests {
}
#[test]
- fn test_elapsed_time_ceiling_exceeded() {
- // With max_retry_exception_elapsed_time_ms: Some(0), next_attempt
should
- // immediately return None, enforcing the elapsed-time ceiling
deterministically.
+ fn test_elapsed_time_ceiling_does_not_cut_policy_retries() {
+ // With max_retry_exception_elapsed_time_ms: Some(0), the elapsed
bound of the
+ // RetryExceptions has passed at once, but a policy retry is bounded
by max_retries
+ // alone, as the reference's RetryPolicyState: the retry is allowed.
let policy = RetryPolicy {
max_retries: Some(10),
initial_backoff_ms: 50,
@@ -404,7 +399,29 @@ mod tests {
};
let mut state = RetryPolicyState::new(policy);
- // Even with max_retries remaining, elapsed time ceiling exceeded so
None.
+ assert_eq!(state.next_attempt(None), Some(50));
+ assert_eq!(state.attempt(), 1);
+ }
+
+ #[test]
+ fn test_policy_retries_after_the_elapsed_bound_until_max_retries() {
+ // A wait past the elapsed bound: the retries of the policy are still
those of max_retries.
+ let policy = RetryPolicy {
+ max_retries: Some(2),
+ initial_backoff_ms: 50,
+ max_backoff_ms: None,
+ backoff_multiplier: 1.0,
+ jitter_ms: 0,
+ min_jitter_threshold_ms: 0,
+ recognize_server_retry_delay: false,
+ max_server_retry_delay_ms: None,
+ max_retry_exception_elapsed_time_ms: Some(1),
+ };
+
+ let mut state = RetryPolicyState::new(policy);
+ std::thread::sleep(std::time::Duration::from_millis(5));
+ assert_eq!(state.next_attempt(None), Some(50));
+ assert_eq!(state.next_attempt(None), Some(50));
assert_eq!(state.next_attempt(None), None);
}
diff --git a/crates/spark-connect-core/tests/deadlines.rs
b/crates/spark-connect-core/tests/deadlines.rs
index 0a4282fa..a8fee2e3 100644
--- a/crates/spark-connect-core/tests/deadlines.rs
+++ b/crates/spark-connect-core/tests/deadlines.rs
@@ -72,6 +72,13 @@ fn operation_not_found() -> Status {
))
}
+fn session_not_found() -> Status {
+ // As the server reports a session it does not know, with the same shape.
+ Status::internal(
+ "[INVALID_HANDLE.SESSION_NOT_FOUND] Session not found:
00000000-0000-4000-8000-000000000001",
+ )
+}
+
#[derive(Clone)]
struct CapturedRelease {
request: ReleaseExecuteRequest,
@@ -452,14 +459,186 @@ async fn
operation_not_found_after_a_response_is_the_error_and_is_not_resent() {
"r1"
);
let error = bounded(stream.message()).await.unwrap_err();
- assert!(
- error.message.contains("INVALID_HANDLE.OPERATION_NOT_FOUND"),
+ assert_response_already_received(&error,
"INVALID_HANDLE.OPERATION_NOT_FOUND");
+ assert_eq!(*fixture.state.executes.lock().unwrap(), 1);
+ fixture.assert_release_all_count(1).await;
+}
+
+/// The error of a reattach to an execution that the server does not know
after a response was
+/// read, as the reference: `RESPONSE_ALREADY_RECEIVED` of a
`PySparkRuntimeError`.
+fn assert_response_already_received(error: &spark_connect_core::SparkError,
error_type: &str) {
+ assert_eq!(
+ error.kind,
+ spark_connect_core::error::SparkErrorKind::RuntimeError,
"{error:?}"
);
+ assert_eq!(error.error_class, "RESPONSE_ALREADY_RECEIVED", "{error:?}");
+ assert_eq!(
+ error.params.get("error_type").map(String::as_str),
+ Some(error_type),
+ "{error:?}"
+ );
+}
+
+#[tokio::test]
+async fn session_not_found_before_any_response_resends_the_execute_plan() {
+ // As the reference, which reads SESSION_NOT_FOUND as OPERATION_NOT_FOUND:
no response was
+ // read, so the ExecutePlan is sent again, and its results are returned.
+ let fixture = Fixture::start(
+ vec![
+ streamed(vec![]),
+ streamed(vec![response("r1", false), response("r2", true)]),
+ ],
+ vec![Step::Stream(vec![Err(session_not_found())])],
+ quick_retries(),
+ )
+ .await;
+ let mut stream =
bounded(fixture.client.execute_plan_reattachable(execute_request()))
+ .await
+ .unwrap();
+ let ids = drain(&mut stream).await.unwrap();
+ assert_eq!(ids, ["r1", "r2"]);
+
+ assert_eq!(*fixture.state.executes.lock().unwrap(), 2);
+ assert_eq!(fixture.state.reattaches.lock().unwrap().len(), 1);
+ fixture.assert_release_all_count(1).await;
+}
+
+#[tokio::test]
+async fn session_not_found_after_a_response_is_response_already_received() {
+ // As the reference: once a response was read, SESSION_NOT_FOUND is the
error of the reattach,
+ // not a reason to send the execution again.
+ let fixture = Fixture::start(
+ vec![streamed(vec![response("r1", false)])],
+ vec![Step::Stream(vec![Err(session_not_found())])],
+ quick_retries(),
+ )
+ .await;
+ let mut stream =
bounded(fixture.client.execute_plan_reattachable(execute_request()))
+ .await
+ .unwrap();
+ assert_eq!(
+ bounded(stream.message())
+ .await
+ .unwrap()
+ .unwrap()
+ .response_id,
+ "r1"
+ );
+ let error = bounded(stream.message()).await.unwrap_err();
+ assert_response_already_received(&error,
"INVALID_HANDLE.SESSION_NOT_FOUND");
assert_eq!(*fixture.state.executes.lock().unwrap(), 1);
fixture.assert_release_all_count(1).await;
}
+#[tokio::test]
+async fn
session_not_found_in_the_reattach_call_after_a_response_is_response_already_received()
{
+ // As the reference, for the reattach call itself failing with
SESSION_NOT_FOUND.
+ let fixture = Fixture::start(
+ vec![streamed(vec![response("r1", false)])],
+ vec![Step::Fail(session_not_found())],
+ quick_retries(),
+ )
+ .await;
+ let mut stream =
bounded(fixture.client.execute_plan_reattachable(execute_request()))
+ .await
+ .unwrap();
+ assert_eq!(
+ bounded(stream.message())
+ .await
+ .unwrap()
+ .unwrap()
+ .response_id,
+ "r1"
+ );
+ let error = bounded(stream.message()).await.unwrap_err();
+ assert_response_already_received(&error,
"INVALID_HANDLE.SESSION_NOT_FOUND");
+ assert_eq!(*fixture.state.executes.lock().unwrap(), 1);
+ fixture.assert_release_all_count(1).await;
+}
+
+#[tokio::test]
+async fn policy_retry_of_a_reattach_is_not_cut_by_the_retry_exception_bound() {
+ // As the reference's RetryPolicyState: the elapsed bound of the
RetryExceptions (here 0, as
+ // if an hour had passed) does not cut a retry of the policy. The reattach
call fails with
+ // UNAVAILABLE, and it is retried with the policy's attempts, here
reattaching with success.
+ let fixture = Fixture::start(
+ vec![streamed(vec![response("r1", false)])],
+ vec![Step::Fail(Status::unavailable("server restarting"))],
+ RetryPolicy {
+ max_retry_exception_elapsed_time_ms: Some(0),
+ ..quick_retries()
+ },
+ )
+ .await;
+ let mut stream =
bounded(fixture.client.execute_plan_reattachable(execute_request()))
+ .await
+ .unwrap();
+ let ids = drain(&mut stream).await.unwrap();
+ assert_eq!(ids, ["r1", "complete"]);
+
+ let reattaches = fixture.state.reattaches.lock().unwrap().clone();
+ assert_eq!(reattaches.len(), 2);
+ assert!(reattaches
+ .iter()
+ .all(|r| r.last_response_id.as_deref() == Some("r1")));
+ assert_eq!(*fixture.state.executes.lock().unwrap(), 1);
+ fixture.assert_release_all_count(1).await;
+}
+
+#[tokio::test]
+async fn error_of_a_server_with_details_bounds_the_config_read() {
+ // As the reference: the stack trace of an error is shown as its config
says, read with the
+ // config deadline. A config that never answers is not waited for, and the
error is raised.
+ let fixture = Fixture::start(vec![], vec![], quick_retries()).await;
+ let status = Status::with_details(Code::Internal, "boom",
error_info_details().into());
+ let error = bounded(fixture.client.error_from_status(status)).await;
+ assert!(error.message.contains("boom"), "{error:?}");
+
+ let timeouts = fixture.state.timeouts.lock().unwrap().clone();
+ assert!(
+ timeouts
+ .iter()
+ .any(|(call, timeout)| *call == "Config" && timeout.is_some()),
+ "{timeouts:?}"
+ );
+}
+
+/// A `google.rpc.Status` with an `ErrorInfo` detail, as the server sends the
errors it raises.
+fn error_info_details() -> Vec<u8> {
+ use prost::Message;
+
+ #[derive(Clone, PartialEq, prost::Message)]
+ struct ErrorInfo {
+ #[prost(string, tag = "1")]
+ reason: String,
+ #[prost(string, tag = "2")]
+ domain: String,
+ }
+ #[derive(Clone, PartialEq, prost::Message)]
+ struct RpcStatus {
+ #[prost(int32, tag = "1")]
+ code: i32,
+ #[prost(string, tag = "2")]
+ message: String,
+ #[prost(message, repeated, tag = "3")]
+ details: Vec<prost_types::Any>,
+ }
+ RpcStatus {
+ code: Code::Internal as i32,
+ message: "boom".into(),
+ details: vec![prost_types::Any {
+ type_url: "type.googleapis.com/google.rpc.ErrorInfo".into(),
+ value: ErrorInfo {
+ reason: "BOOM".into(),
+ domain: "org.apache.spark".into(),
+ }
+ .encode_to_vec(),
+ }],
+ }
+ .encode_to_vec()
+}
+
/// The loopback service: ExecutePlan, ReattachExecute, ReleaseExecute and
Config.
#[derive(Clone)]
struct TestService(Arc<State>);
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]