This is an automated email from the ASF dual-hosted git repository.
HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git
The following commit(s) were added to refs/heads/master by this push:
new 91c1a37 [SPARK-61064][CONNECT] Raise DEADLINE_EXCEEDED and reattach
when a per-RPC deadline passes before the response headers, and execute again
on OPERATION_NOT_FOUND, as PySpark does
91c1a37 is described below
commit 91c1a3795d89d78b9c11d2449446ff7cd8563c64
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Sat Nov 7 11:41:53 2026 +0900
[SPARK-61064][CONNECT] Raise DEADLINE_EXCEEDED and reattach when a per-RPC
deadline passes before the response headers, and execute again on
OPERATION_NOT_FOUND, as PySpark does
### What changes were proposed in this pull request?
The per-RPC deadlines of PySpark Connect, which SPARK-60997 applied with
tonic's `grpc-timeout`, now behave as PySpark's when they pass before the
response headers (`crates/spark-connect-core/src/client.rs`):
- `within_deadline` bounds each RPC as a whole by its deadline, and reads
tonic's `CANCELLED` "Timeout expired", which tonic gives when its
`grpc-timeout` passes before the headers, as the `DEADLINE_EXCEEDED` it is. A
unary call stalled after its headers is now bounded as well.
- A reattachable execution whose ExecutePlan or ReattachExecute passes its
deadline reattaches, for the policy's bound of such retries from the first one,
as the reference's `RetryException` does.
- A reattach that finds `INVALID_HANDLE.OPERATION_NOT_FOUND` before any
response, from the call or from its stream, sends the ExecutePlan again, as the
reference iterator does.
### Why are the changes needed?
[SPARK-61064](https://issues.apache.org/jira/browse/SPARK-61064): with a
server that stalls before sending headers (an in-process mock that compares the
two clients):
| Scenario | Current master | This change and PySpark 4.3.0 |
| --- | --- | --- |
| First ExecutePlan of an execution stalls past its deadline | fails with
`[The operation was cancelled] Timeout expired` after 10 minutes, no reattach |
reattaches, executes again on OPERATION_NOT_FOUND, returns the rows |
| ReattachExecute stalls past its deadline | fails the same way |
reattaches again and returns the rows |
| Config stalls past its deadline | `CANCELLED` | `DEADLINE_EXCEEDED`, with
PySpark's message about the per-RPC timeouts |
The existing tests stall response bodies, never headers, which the client
already handled.
### Does this PR introduce _any_ user-facing change?
Yes, in these failure cases, as in PySpark.
### How was this patch tested?
- `crates/spark-connect-core/tests/deadlines.rs`, 8 tests against a
loopback service that stalls before its headers: the three scenarios above, the
bound of the retries past deadlines for the first call and for reattaches, and
OPERATION_NOT_FOUND from the call, from the stream, and after a response (which
still fails). Seven of them fail without the change.
- The core tests (lib 119, reattach cleanup 11, user context 4, all passed)
and the build of every integration test of `apache-spark-connect`.
- The stalled-ExecutePlan scenario end to end through the Python drop-in:
the rows are returned after the reattach and the second ExecutePlan.
Unchanged and different from PySpark: PySpark also executes again on
`INVALID_HANDLE.SESSION_NOT_FOUND`, and raises `RESPONSE_ALREADY_RECEIVED` for
OPERATION_NOT_FOUND after a response.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
This pull request and its description were written by Isaac.
Closes #255 from HyukjinKwon/SPARK-61064-deadlines.
Authored-by: Hyukjin Kwon <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
---
crates/spark-connect-core/src/client.rs | 300 +++++++++----
crates/spark-connect-core/tests/deadlines.rs | 614 +++++++++++++++++++++++++++
2 files changed, 838 insertions(+), 76 deletions(-)
diff --git a/crates/spark-connect-core/src/client.rs
b/crates/spark-connect-core/src/client.rs
index 786192a..7d68f87 100644
--- a/crates/spark-connect-core/src/client.rs
+++ b/crates/spark-connect-core/src/client.rs
@@ -160,6 +160,51 @@ fn with_deadline<T>(mut request: Request<T>, deadline:
Option<Duration>) -> Requ
request
}
+/// `call` bounded as a whole by `deadline`, as gRPC bounds an RPC: past it,
the call fails with
+/// `DEADLINE_EXCEEDED`, as PySpark's does, however far it got.
+///
+/// The `grpc-timeout` of [`with_deadline`] is also enforced by tonic, but
only while the
+/// response headers are awaited, and an expiry there is a `CANCELLED`
"Timeout expired". So
+/// the call is bounded here too, and that status is read as the deadline it
is. A call stalled
+/// after its headers, as for a unary response, is then bounded as well.
+async fn within_deadline<T>(
+ deadline: Option<Duration>,
+ call: impl std::future::Future<Output = std::result::Result<T,
tonic::Status>>,
+) -> std::result::Result<T, tonic::Status> {
+ let Some(deadline) = deadline else {
+ return call.await;
+ };
+ match tokio::time::timeout(deadline, call).await {
+ Ok(Err(status))
+ if status.code() == tonic::Code::Cancelled && status.message() ==
"Timeout expired" =>
+ {
+ Err(tonic::Status::deadline_exceeded("Deadline Exceeded"))
+ }
+ Ok(result) => result,
+ Err(_) => Err(tonic::Status::deadline_exceeded("Deadline Exceeded")),
+ }
+}
+
+/// Whether `status` says that the server does not know the operation of a
reattach, which
+/// the reference reads as the initial ExecutePlan never having reached the
server.
+fn is_operation_not_found(status: &tonic::Status) -> bool {
+ status
+ .message()
+ .contains("INVALID_HANDLE.OPERATION_NOT_FOUND")
+}
+
+/// Whether the retries that the reference makes as `RetryException`s have run
for the policy's
+/// elapsed-time bound, measured from the first of them, which starts `since`
when it is unset.
+fn retry_exception_bound_passed(
+ policy: &RetryPolicy,
+ since: &mut Option<std::time::Instant>,
+) -> bool {
+ let since = *since.get_or_insert_with(std::time::Instant::now);
+ policy
+ .max_retry_exception_elapsed_time_ms
+ .is_some_and(|ms| since.elapsed() >= Duration::from_millis(ms))
+}
+
/// A Spark Connect client for communicating with a remote Spark server.
///
/// Mirrors `pyspark.sql.connect.client.core.SparkConnectClient`.
@@ -420,15 +465,18 @@ impl SparkConnectClient {
.max_decoding_message_size(GRPC_MAX_MESSAGE_LENGTH_DEFAULT)
.max_encoding_message_size(GRPC_MAX_MESSAGE_LENGTH_DEFAULT);
wait_ready(&mut grpc).await?;
- let mut req = with_deadline(Request::new(request),
self.rpc_deadlines.analyze_plan);
+ let deadline = self.rpc_deadlines.analyze_plan;
+ let mut req = with_deadline(Request::new(request), deadline);
self._attach_metadata(&mut req);
let path = tonic::codegen::http::uri::PathAndQuery::from_static(
"/spark.connect.SparkConnectService/AnalyzePlan",
);
- let resp = grpc
- .unary(req, path, crate::bytes_codec::BytesCodec)
- .await
- .map_err(SparkError::from_grpc_status)?;
+ let resp = within_deadline(
+ deadline,
+ grpc.unary(req, path, crate::bytes_codec::BytesCodec),
+ )
+ .await
+ .map_err(SparkError::from_grpc_status)?;
Ok(resp.into_inner())
}
@@ -441,13 +489,15 @@ impl SparkConnectClient {
) -> Result<AnalyzePlanResponse> {
self.fill_user_context(&mut request.user_context);
self.with_retry(|| {
- let mut req = with_deadline(
- Request::new(request.clone()),
- self.rpc_deadlines.analyze_plan,
- );
+ let deadline = self.rpc_deadlines.analyze_plan;
+ let mut req = with_deadline(Request::new(request.clone()),
deadline);
self._attach_metadata(&mut req);
let mut stub = self.stub.clone();
- async move {
stub.analyze_plan(req).await.map(Response::into_inner) }
+ async move {
+ within_deadline(deadline, stub.analyze_plan(req))
+ .await
+ .map(Response::into_inner)
+ }
})
.await
}
@@ -502,7 +552,11 @@ impl SparkConnectClient {
self._attach_metadata(&mut req);
stream_deadline = deadline.map(|d| tokio::time::Instant::now()
+ d);
let mut stub = self.stub.clone();
- async move {
stub.execute_plan(req).await.map(Response::into_inner) }
+ async move {
+ within_deadline(deadline, stub.execute_plan(req))
+ .await
+ .map(Response::into_inner)
+ }
})
.await;
let stream = match stream {
@@ -525,6 +579,7 @@ impl SparkConnectClient {
iter,
stream,
stream_deadline,
+ execute_again: false,
done: false,
released: false,
})
@@ -539,12 +594,10 @@ impl SparkConnectClient {
mut request: FetchErrorDetailsRequest,
) -> Result<FetchErrorDetailsResponse> {
self.fill_user_context(&mut request.user_context);
- let mut req = with_deadline(
- Request::new(request),
- self.rpc_deadlines.fetch_error_details,
- );
+ let deadline = self.rpc_deadlines.fetch_error_details;
+ let mut req = with_deadline(Request::new(request), deadline);
self._attach_metadata(&mut req);
- let resp = self.stub.clone().fetch_error_details(req).await;
+ let resp = within_deadline(deadline,
self.stub.clone().fetch_error_details(req)).await;
resp.map(Response::into_inner)
.map_err(SparkError::from_grpc_status)
}
@@ -580,12 +633,12 @@ impl SparkConnectClient {
}),
..Default::default()
};
- let mut req = with_deadline(
- Request::new(request),
- self.rpc_deadlines.fetch_error_details,
- );
+ let deadline = self.rpc_deadlines.fetch_error_details;
+ let mut req = with_deadline(Request::new(request), deadline);
self._attach_metadata(&mut req);
- let response = self.stub.clone().fetch_error_details(req).await.ok()?;
+ let response = within_deadline(deadline,
self.stub.clone().fetch_error_details(req))
+ .await
+ .ok()?;
Some(response.into_inner())
}
@@ -635,10 +688,15 @@ impl SparkConnectClient {
pub async fn config(&self, mut request: ConfigRequest) ->
Result<ConfigResponse> {
self.fill_user_context(&mut request.user_context);
self.with_retry(|| {
- let mut req = with_deadline(Request::new(request.clone()),
self.rpc_deadlines.config);
+ let deadline = self.rpc_deadlines.config;
+ let mut req = with_deadline(Request::new(request.clone()),
deadline);
self._attach_metadata(&mut req);
let mut stub = self.stub.clone();
- async move { stub.config(req).await.map(Response::into_inner) }
+ async move {
+ within_deadline(deadline, stub.config(req))
+ .await
+ .map(Response::into_inner)
+ }
})
.await
}
@@ -649,11 +707,15 @@ impl SparkConnectClient {
pub async fn interrupt(&self, mut request: InterruptRequest) ->
Result<InterruptResponse> {
self.fill_user_context(&mut request.user_context);
self.with_retry(|| {
- let mut req =
- with_deadline(Request::new(request.clone()),
self.rpc_deadlines.interrupt);
+ let deadline = self.rpc_deadlines.interrupt;
+ let mut req = with_deadline(Request::new(request.clone()),
deadline);
self._attach_metadata(&mut req);
let mut stub = self.stub.clone();
- async move { stub.interrupt(req).await.map(Response::into_inner) }
+ async move {
+ within_deadline(deadline, stub.interrupt(req))
+ .await
+ .map(Response::into_inner)
+ }
})
.await
}
@@ -957,13 +1019,15 @@ impl SparkConnectClient {
};
let response = self
.with_retry(|| {
- let mut req = with_deadline(
- Request::new(request.clone()),
- self.rpc_deadlines.clone_session,
- );
+ let deadline = self.rpc_deadlines.clone_session;
+ let mut req = with_deadline(Request::new(request.clone()),
deadline);
self._attach_metadata(&mut req);
let mut stub = self.stub.clone();
- async move {
stub.clone_session(req).await.map(Response::into_inner) }
+ async move {
+ within_deadline(deadline, stub.clone_session(req))
+ .await
+ .map(Response::into_inner)
+ }
})
.await?;
if let Some(requested) = new_session_id {
@@ -987,9 +1051,10 @@ impl SparkConnectClient {
..Default::default()
});
- let mut req = with_deadline(Request::new(request),
self.rpc_deadlines.release_session);
+ let deadline = self.rpc_deadlines.release_session;
+ let mut req = with_deadline(Request::new(request), deadline);
self._attach_metadata(&mut req);
- match self.stub.clone().release_session(req).await {
+ match within_deadline(deadline,
self.stub.clone().release_session(req)).await {
Ok(_) => Ok(()),
Err(status) => Err(self.error_from_status(status).await),
}
@@ -1138,14 +1203,10 @@ impl SparkConnectClient {
request
})
.collect();
- let mut req = with_deadline(
- Request::new(futures::stream::iter(requests)),
- self.rpc_deadlines.add_artifacts,
- );
+ let deadline = self.rpc_deadlines.add_artifacts;
+ let mut req =
with_deadline(Request::new(futures::stream::iter(requests)), deadline);
self._attach_metadata(&mut req);
- self.stub
- .clone()
- .add_artifacts(req)
+ within_deadline(deadline, self.stub.clone().add_artifacts(req))
.await
.map(Response::into_inner)
}
@@ -1175,9 +1236,12 @@ impl SparkConnectClient {
request.client_type = Some(self.user_agent.clone());
request.names = names.to_vec();
- let mut req = with_deadline(Request::new(request),
self.rpc_deadlines.artifact_status);
+ let deadline = self.rpc_deadlines.artifact_status;
+ let mut req = with_deadline(Request::new(request), deadline);
self._attach_metadata(&mut req);
- let response =
self.stub.clone().artifact_status(req).await?.into_inner();
+ let response = within_deadline(deadline,
self.stub.clone().artifact_status(req))
+ .await?
+ .into_inner();
Ok(response
.statuses
.into_iter()
@@ -1235,6 +1299,9 @@ pub struct ReattachableResponseStream {
stream: Option<Streaming<ExecutePlanResponse>>,
/// When the deadline of the stream passes, as gRPC's of the call that
opened it.
stream_deadline: Option<tokio::time::Instant>,
+ /// Whether the next stream is opened by sending the ExecutePlan again,
rather than by
+ /// reattaching: the server does not know the execution, which never
reached it.
+ execute_again: bool,
done: bool,
/// Whether `ReleaseAll` was sent, after which nothing else is released.
released: bool,
@@ -1321,12 +1388,23 @@ impl ReattachableResponseStream {
// As the reference iterator's `_has_next` retries with a `Retrying`
of its own, the
// reattaches of each response have the attempts and the elapsed time
of the policy.
let mut retry_state = RetryPolicyState::new(self.retry_policy.clone());
- // When the first stream of this response passed its deadline, for the
bound of the
- // reattaches after deadlines, as of the reference `RetryException`s.
- let mut past_deadlines_since: Option<std::time::Instant> = None;
+ // The clock of the retries that the reference makes as
`RetryException`s, bounded by the
+ // policy's elapsed time: it starts at the first deadline that passes,
the deadline of
+ // the initial ExecutePlan included, and the reattaches after a
deadline or an unknown
+ // execution are its retries.
+ let mut retry_exception_since: Option<std::time::Instant> = None;
+ if self.stream.is_none()
+ && retry_exception_bound_passed(&self.retry_policy, &mut
retry_exception_since)
+ {
+ // The initial ExecutePlan passed its deadline, a RetryException
that fails at once
+ // when the bound of the retries has passed, as it has when the
bound is 0.
+ let status = tonic::Status::deadline_exceeded("Deadline Exceeded");
+ return Err(self.fail_with_status(status).await);
+ }
loop {
let Some(stream) = self.stream.as_mut() else {
- self.reattach(&mut retry_state).await?;
+ self.reattach(&mut retry_state, &mut retry_exception_since)
+ .await?;
continue;
};
let next = match self.stream_deadline {
@@ -1349,7 +1427,10 @@ impl ReattachableResponseStream {
// The server completes a reattachable stream before
`ResultComplete` on
// purpose, after `senderMaxStreamDuration` or
`senderMaxStreamSize`: the rest
// of the result is read by reattaching.
- Ok(None) => self.reattach(&mut retry_state).await?,
+ Ok(None) => {
+ self.reattach(&mut retry_state, &mut retry_exception_since)
+ .await?
+ }
// As the reference iterator: the per-RPC deadline of the
stream passed, and the
// execution goes on on the server, so the iteration
reattaches at once, for up
// to the policy's bound of such retries.
@@ -1357,55 +1438,122 @@ impl ReattachableResponseStream {
if status.code() == tonic::Code::DeadlineExceeded
&& !self.iter.is_completed().await =>
{
- let since =
*past_deadlines_since.get_or_insert_with(std::time::Instant::now);
- let bound =
self.retry_policy.max_retry_exception_elapsed_time_ms;
- if bound.is_some_and(|ms| since.elapsed() >=
Duration::from_millis(ms)) {
- let error =
self.client.error_from_status(status).await;
- return Err(self.fail(error));
+ if retry_exception_bound_passed(&self.retry_policy, &mut
retry_exception_since)
+ {
+ return Err(self.fail_with_status(status).await);
}
self.stream = None;
}
+ // As the reference iterator: the server does not know the
execution, so the
+ // initial ExecutePlan never reached it, and it is sent again,
if no response was
+ // read yet.
+ Err(status)
+ if is_operation_not_found(&status)
+ && self.iter.last_response_id().await.is_none()
+ && !self.iter.is_completed().await =>
+ {
+ if retry_exception_bound_passed(&self.retry_policy, &mut
retry_exception_since)
+ {
+ return Err(self.fail_with_status(status).await);
+ }
+ self.execute_again = true;
+ self.stream = None;
+ }
Err(status) => {
// Only reattach a still-running execution on a retriable
drop.
if self.iter.is_completed().await ||
!self.retry_policy.can_retry(&status) {
- let error =
self.client.error_from_status(status).await;
- return Err(self.fail(error));
+ return Err(self.fail_with_status(status).await);
}
- self.reattach(&mut retry_state).await?;
+ self.reattach(&mut retry_state, &mut retry_exception_since)
+ .await?;
}
}
}
}
- /// Resumes the execution from the last response read, retrying transient
failures under
- /// `retry_state`.
- async fn reattach(&mut self, retry_state: &mut RetryPolicyState) ->
Result<()> {
- let deadline = self.client.rpc_deadlines.reattach_execute;
+ /// Opens the stream of the execution again: by reattaching to it from the
last response
+ /// read, or, when the server does not know it, by sending the ExecutePlan
again. Transient
+ /// failures are retried under `retry_state`, and a deadline or an unknown
execution as the
+ /// reference's `RetryException`s, bounded by `retry_exception_since`.
+ async fn reattach(
+ &mut self,
+ retry_state: &mut RetryPolicyState,
+ retry_exception_since: &mut Option<std::time::Instant>,
+ ) -> Result<()> {
loop {
- let mut req =
Request::new(self.iter.create_reattach_request().await);
- self.attach_metadata(&mut req);
- // After the metadata, which the deadline is a header of.
- let req = with_deadline(req, deadline);
- self.stream_deadline = deadline.map(|d|
tokio::time::Instant::now() + d);
- match self.client.stub.clone().reattach_execute(req).await {
- Ok(r) => {
- self.stream = Some(r.into_inner());
+ let opened = if self.execute_again {
+ self.open_execute_plan_again().await
+ } else {
+ self.open_reattach().await
+ };
+ let status = match opened {
+ Ok(stream) => {
+ self.stream = Some(stream);
+ self.execute_again = false;
return Ok(());
}
- Err(e) => {
- if self.retry_policy.can_retry(&e) {
- if let Some(w) = retry_state.next_attempt(None) {
- tokio::time::sleep(Duration::from_millis(w)).await;
- continue;
- }
- }
- let error = self.client.error_from_status(e).await;
- return Err(self.fail(error));
+ Err(status) => status,
+ };
+ if is_operation_not_found(&status) &&
self.iter.last_response_id().await.is_none() {
+ if retry_exception_bound_passed(&self.retry_policy,
retry_exception_since) {
+ return Err(self.fail_with_status(status).await);
}
+ self.execute_again = true;
+ continue;
}
+ if status.code() == tonic::Code::DeadlineExceeded {
+ // As the reference: a call past its deadline, whether it is
sent again or
+ // reattached, is retried at once, the execution going on on
the server.
+ if retry_exception_bound_passed(&self.retry_policy,
retry_exception_since) {
+ return Err(self.fail_with_status(status).await);
+ }
+ self.execute_again = false;
+ continue;
+ }
+ if self.retry_policy.can_retry(&status) {
+ if let Some(w) = retry_state.next_attempt(None) {
+ tokio::time::sleep(Duration::from_millis(w)).await;
+ continue;
+ }
+ }
+ return Err(self.fail_with_status(status).await);
}
}
+ /// ReattachExecute from the last response read, bounded by the deadline
of the call.
+ async fn open_reattach(
+ &mut self,
+ ) -> std::result::Result<Streaming<ExecutePlanResponse>, tonic::Status> {
+ let deadline = self.client.rpc_deadlines.reattach_execute;
+ let mut req = Request::new(self.iter.create_reattach_request().await);
+ self.attach_metadata(&mut req);
+ // After the metadata, which the deadline is a header of.
+ let req = with_deadline(req, deadline);
+ self.stream_deadline = deadline.map(|d| tokio::time::Instant::now() +
d);
+ let response =
+ within_deadline(deadline,
self.client.stub.clone().reattach_execute(req)).await?;
+ Ok(response.into_inner())
+ }
+
+ /// ExecutePlan of the request again, as the initial one, bounded by its
deadline.
+ async fn open_execute_plan_again(
+ &mut self,
+ ) -> std::result::Result<Streaming<ExecutePlanResponse>, tonic::Status> {
+ let deadline = self.client.rpc_deadlines.reattachable_execute_plan;
+ let mut req = with_deadline(Request::new(self.iter.request().clone()),
deadline);
+ self.client._attach_metadata(&mut req);
+ self.stream_deadline = deadline.map(|d| tokio::time::Instant::now() +
d);
+ let response =
+ within_deadline(deadline,
self.client.stub.clone().execute_plan(req)).await?;
+ Ok(response.into_inner())
+ }
+
+ /// Ends the stream with the error of `status`, as [`Self::fail`].
+ async fn fail_with_status(&mut self, status: tonic::Status) -> SparkError {
+ let error = self.client.error_from_status(status).await;
+ self.fail(error)
+ }
+
/// Ends the stream with `error`, releasing the execution as the reference
client does
/// when an execution fails.
/// The error carries the ExecutePlan operation id
(`SparkConnectException.operation_id`).
diff --git a/crates/spark-connect-core/tests/deadlines.rs
b/crates/spark-connect-core/tests/deadlines.rs
new file mode 100644
index 0000000..0a4282f
--- /dev/null
+++ b/crates/spark-connect-core/tests/deadlines.rs
@@ -0,0 +1,614 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Per-RPC deadlines that expire before the response headers, and the
reattach to an
+//! execution that the server does not know. No Spark server required: a
loopback tonic
+//! service stalls its calls before they send headers, as a stalled connection
does.
+
+use std::collections::{HashSet, VecDeque};
+use std::convert::Infallible;
+use std::future::Future;
+use std::pin::Pin;
+use std::sync::{Arc, Mutex};
+use std::task::{Context, Poll};
+use std::time::Duration;
+
+use futures::{stream, Stream};
+use spark_connect_core::{ChannelBuilder, RetryPolicy, RpcDeadlines,
SparkConnectClient};
+use spark_connect_proto::execute_plan_response::{ResponseType, ResultComplete};
+use spark_connect_proto::release_execute_request::Release;
+use spark_connect_proto::{
+ ConfigRequest, ConfigResponse, ExecutePlanRequest, ExecutePlanResponse,
ReattachExecuteRequest,
+ ReleaseExecuteRequest, ReleaseExecuteResponse, UserContext,
+};
+use tokio::net::TcpListener;
+use tonic::body::Body;
+use tonic::codegen::{http, BoxFuture, Service};
+use tonic::metadata::MetadataMap;
+use tonic::server::{NamedService, ServerStreamingService, UnaryService};
+use tonic::transport::Server;
+use tonic::{Code, Request, Response, Status};
+
+/// The per-RPC deadline of the tests: short, so that the stalls end quickly.
+const DEADLINE: Duration = Duration::from_millis(200);
+/// Bound of every RPC of a test, so that a call that waits forever fails the
test.
+const WAIT: Duration = Duration::from_secs(5);
+const OPERATION: &str = "00000000-0000-4000-8000-000000000002";
+type Responses = Pin<Box<dyn Stream<Item = Result<ExecutePlanResponse,
Status>> + Send>>;
+
+async fn bounded<T>(future: impl Future<Output = T>) -> T {
+ tokio::time::timeout(WAIT, future)
+ .await
+ .expect("test RPC waited past the bound")
+}
+
+/// What the server does with one call: stall before its headers, fail the
call, or stream
+/// these items (an error item ends the stream with that status).
+enum Step {
+ Stall,
+ Fail(Status),
+ Stream(Vec<Result<ExecutePlanResponse, Status>>),
+}
+
+fn operation_not_found() -> Status {
+ // As the server reports an execution it never started: an INTERNAL status
whose message
+ // carries the error class.
+ Status::internal(format!(
+ "[INVALID_HANDLE.OPERATION_NOT_FOUND] Operation not found: {OPERATION}"
+ ))
+}
+
+#[derive(Clone)]
+struct CapturedRelease {
+ request: ReleaseExecuteRequest,
+}
+
+struct State {
+ /// What each ExecutePlan call does, in order. When none is left, it
completes the result.
+ execute_script: Mutex<VecDeque<Step>>,
+ /// What each ReattachExecute call does, in order. When none is left, it
completes the
+ /// result of an execution that the server knows, and fails with
OPERATION_NOT_FOUND for
+ /// one it does not.
+ reattach_script: Mutex<VecDeque<Step>>,
+ /// The operations that the server started, that is, whose ExecutePlan
streamed.
+ known: Mutex<HashSet<String>>,
+ /// The `grpc-timeout` of each call, in order.
+ timeouts: Mutex<Vec<(&'static str, Option<String>)>>,
+ executes: Mutex<usize>,
+ reattaches: Mutex<Vec<ReattachExecuteRequest>>,
+ releases: Mutex<Vec<CapturedRelease>>,
+}
+
+impl State {
+ fn new(execute: Vec<Step>, reattach: Vec<Step>) -> Self {
+ State {
+ execute_script: Mutex::new(execute.into()),
+ reattach_script: Mutex::new(reattach.into()),
+ known: Mutex::new(HashSet::new()),
+ timeouts: Mutex::new(Vec::new()),
+ executes: Mutex::new(0),
+ reattaches: Mutex::new(Vec::new()),
+ releases: Mutex::new(Vec::new()),
+ }
+ }
+
+ fn record_timeout(&self, call: &'static str, metadata: &MetadataMap) {
+ let timeout = metadata
+ .get("grpc-timeout")
+ .map(|value| value.to_str().unwrap().to_string());
+ self.timeouts.lock().unwrap().push((call, timeout));
+ }
+
+ fn release_all_count(&self) -> usize {
+ self.releases
+ .lock()
+ .unwrap()
+ .iter()
+ .filter(|r| matches!(r.request.release,
Some(Release::ReleaseAll(_))))
+ .count()
+ }
+}
+
+struct Fixture {
+ state: Arc<State>,
+ server: tokio::task::JoinHandle<()>,
+ client: SparkConnectClient,
+}
+
+impl Fixture {
+ /// A client with `deadline` for the calls of a reattachable execution and
for Config, and
+ /// the given retry policy.
+ async fn start(execute: Vec<Step>, reattach: Vec<Step>, retry_policy:
RetryPolicy) -> Self {
+ let state = Arc::new(State::new(execute, reattach));
+ let listener = TcpListener::bind("127.0.0.1:0").await.unwrap();
+ let address = listener.local_addr().unwrap();
+ let incoming = stream::unfold(listener, |listener| async {
+ Some((listener.accept().await.map(|(socket, _)| socket), listener))
+ });
+ let service = TestService(state.clone());
+ let server = tokio::spawn(async move {
+ Server::builder()
+ .add_service(service)
+ .serve_with_incoming(incoming)
+ .await
+ .unwrap();
+ });
+ let builder = ChannelBuilder::parse(&format!(
+ "sc://{address}/;user_id=transport-user;token=fixture-token"
+ ))
+ .unwrap();
+ let mut client = bounded(SparkConnectClient::connect(&builder))
+ .await
+ .unwrap();
+ client.set_retry_policy(retry_policy);
+ client.set_rpc_deadlines(RpcDeadlines {
+ reattachable_execute_plan: Some(DEADLINE),
+ reattach_execute: Some(DEADLINE),
+ config: Some(DEADLINE),
+ ..RpcDeadlines::default()
+ });
+ Self {
+ state,
+ server,
+ client,
+ }
+ }
+
+ /// Waits until the release of the whole execution is sent once, and
checks that it is not
+ /// sent again.
+ async fn assert_release_all_count(&self, expected: usize) {
+ let state = self.state.clone();
+ bounded(async {
+ while state.release_all_count() < expected {
+ tokio::time::sleep(Duration::from_millis(5)).await;
+ }
+ })
+ .await;
+ tokio::time::sleep(Duration::from_millis(100)).await;
+ assert_eq!(self.state.release_all_count(), expected);
+ }
+}
+
+impl Drop for Fixture {
+ fn drop(&mut self) {
+ self.server.abort();
+ }
+}
+
+fn execute_request() -> ExecutePlanRequest {
+ ExecutePlanRequest {
+ session_id: "00000000-0000-4000-8000-000000000001".into(),
+ user_context: Some(UserContext {
+ user_id: "request-user".into(),
+ ..Default::default()
+ }),
+ operation_id: Some(OPERATION.into()),
+ client_type: Some("deadline-regression".into()),
+ ..Default::default()
+ }
+}
+
+fn response(id: &str, complete: bool) -> ExecutePlanResponse {
+ ExecutePlanResponse {
+ response_id: id.into(),
+ response_type:
complete.then_some(ResponseType::ResultComplete(ResultComplete {})),
+ ..Default::default()
+ }
+}
+
+/// A stream of `items`, as the server sends them.
+fn streamed(items: Vec<ExecutePlanResponse>) -> Step {
+ Step::Stream(items.into_iter().map(Ok).collect())
+}
+
+/// The retry policy of the tests: as the cleanup tests, with short backoffs.
+fn quick_retries() -> RetryPolicy {
+ RetryPolicy {
+ max_retries: Some(2),
+ initial_backoff_ms: 1,
+ max_backoff_ms: Some(2),
+ ..RetryPolicy::default()
+ }
+}
+
+/// Drains `stream`, returning the ids of its responses, or the error that
ends it.
+async fn drain(
+ stream: &mut spark_connect_core::client::ReattachableResponseStream,
+) -> Result<Vec<String>, spark_connect_core::SparkError> {
+ let mut ids = Vec::new();
+ while let Some(response) = bounded(stream.message()).await? {
+ ids.push(response.response_id);
+ }
+ Ok(ids)
+}
+
+#[tokio::test]
+async fn stalled_first_execute_plan_reattaches_then_resends_the_execution() {
+ // As the reference: the first ExecutePlan sends no headers, so its
deadline passes. The
+ // execution is reattached, and as the server does not know it, it is sent
again, and its
+ // results are returned.
+ let fixture = Fixture::start(
+ vec![
+ Step::Stall,
+ streamed(vec![response("r1", false), response("r2", true)]),
+ ],
+ vec![],
+ quick_retries(),
+ )
+ .await;
+ let mut stream =
bounded(fixture.client.execute_plan_reattachable(execute_request()))
+ .await
+ .unwrap();
+ let ids = drain(&mut stream).await.unwrap();
+ assert_eq!(ids, ["r1", "r2"]);
+
+ assert_eq!(*fixture.state.executes.lock().unwrap(), 2);
+ let reattaches = fixture.state.reattaches.lock().unwrap().clone();
+ assert_eq!(reattaches.len(), 1);
+ assert_eq!(reattaches[0].last_response_id, None);
+ let timeouts = fixture.state.timeouts.lock().unwrap().clone();
+ let calls: Vec<_> = timeouts.iter().map(|(call, _)| *call).collect();
+ assert_eq!(calls, ["ExecutePlan", "ReattachExecute", "ExecutePlan"]);
+ assert!(
+ timeouts.iter().all(|(_, timeout)| timeout.is_some()),
+ "{timeouts:?}"
+ );
+ fixture.assert_release_all_count(1).await;
+}
+
+#[tokio::test]
+async fn stalled_reattach_is_reattached_again() {
+ // As the reference: a ReattachExecute that sends no headers past its
deadline is retried,
+ // the execution going on on the server, and the results after the last
response read are
+ // returned.
+ let fixture = Fixture::start(
+ vec![streamed(vec![response("r1", false)])],
+ vec![Step::Stall, streamed(vec![response("r2", true)])],
+ quick_retries(),
+ )
+ .await;
+ let mut stream =
bounded(fixture.client.execute_plan_reattachable(execute_request()))
+ .await
+ .unwrap();
+ let ids = drain(&mut stream).await.unwrap();
+ assert_eq!(ids, ["r1", "r2"]);
+
+ let reattaches = fixture.state.reattaches.lock().unwrap().clone();
+ let resumed: Vec<_> = reattaches
+ .iter()
+ .map(|r| r.last_response_id.clone())
+ .collect();
+ assert_eq!(resumed, [Some("r1".to_string()), Some("r1".to_string())]);
+ assert_eq!(*fixture.state.executes.lock().unwrap(), 1);
+ fixture.assert_release_all_count(1).await;
+}
+
+#[tokio::test]
+async fn unary_call_past_its_deadline_is_deadline_exceeded() {
+ // As the reference: a unary call stalled past its deadline raises
DEADLINE_EXCEEDED, with
+ // the message about the per-RPC timeouts.
+ let fixture = Fixture::start(vec![], vec![], quick_retries()).await;
+ let error = bounded(fixture.client.config(ConfigRequest::default()))
+ .await
+ .unwrap_err();
+ assert_eq!(
+ error.grpc_code,
+ Some(Code::DeadlineExceeded as i32),
+ "{error:?}"
+ );
+ assert!(
+ error.message.contains("RPC deadline exceeded"),
+ "unexpected error: {error:?}"
+ );
+ assert!(!error.message.contains("Timeout expired"), "{error:?}");
+}
+
+#[tokio::test]
+async fn execute_plan_past_its_deadline_is_bounded_as_the_reference_retries() {
+ // As the reference: the deadline of the initial ExecutePlan is a
RetryException, which is
+ // the error once the bound of the retries has passed. With no bound, the
error is the
+ // deadline, and nothing is reattached.
+ let fixture = Fixture::start(
+ vec![Step::Stall],
+ vec![],
+ RetryPolicy {
+ max_retry_exception_elapsed_time_ms: Some(0),
+ ..quick_retries()
+ },
+ )
+ .await;
+ let mut stream =
bounded(fixture.client.execute_plan_reattachable(execute_request()))
+ .await
+ .unwrap();
+ let error = bounded(stream.message()).await.unwrap_err();
+ assert_eq!(
+ error.grpc_code,
+ Some(Code::DeadlineExceeded as i32),
+ "{error:?}"
+ );
+ assert!(error.message.contains("RPC deadline exceeded"), "{error:?}");
+ assert!(fixture.state.reattaches.lock().unwrap().is_empty());
+ assert_eq!(*fixture.state.executes.lock().unwrap(), 1);
+ fixture.assert_release_all_count(1).await;
+}
+
+#[tokio::test]
+async fn reattach_past_its_deadline_is_the_error_once_the_bound_passes() {
+ // As the reference: a reattach past its deadline is retried, and past the
bound of those
+ // retries, the deadline is the error and the execution is released.
+ let fixture = Fixture::start(
+ vec![streamed(vec![response("r1", false)])],
+ vec![Step::Stall],
+ RetryPolicy {
+ max_retry_exception_elapsed_time_ms: Some(0),
+ ..quick_retries()
+ },
+ )
+ .await;
+ let mut stream =
bounded(fixture.client.execute_plan_reattachable(execute_request()))
+ .await
+ .unwrap();
+ assert_eq!(
+ bounded(stream.message())
+ .await
+ .unwrap()
+ .unwrap()
+ .response_id,
+ "r1"
+ );
+ let error = bounded(stream.message()).await.unwrap_err();
+ assert_eq!(
+ error.grpc_code,
+ Some(Code::DeadlineExceeded as i32),
+ "{error:?}"
+ );
+ assert_eq!(fixture.state.reattaches.lock().unwrap().len(), 1);
+ fixture.assert_release_all_count(1).await;
+}
+
+#[tokio::test]
+async fn operation_not_found_before_any_response_resends_the_execute_plan() {
+ // As the reference: the ExecutePlan of the execution streams no response,
so it is
+ // reattached, and the reattach reports that the server does not know the
operation, which
+ // is then sent again, its results returned.
+ let fixture = Fixture::start(
+ vec![
+ streamed(vec![]),
+ streamed(vec![response("r1", false), response("r2", true)]),
+ ],
+ vec![Step::Stream(vec![Err(operation_not_found())])],
+ quick_retries(),
+ )
+ .await;
+ let mut stream =
bounded(fixture.client.execute_plan_reattachable(execute_request()))
+ .await
+ .unwrap();
+ let ids = drain(&mut stream).await.unwrap();
+ assert_eq!(ids, ["r1", "r2"]);
+
+ assert_eq!(*fixture.state.executes.lock().unwrap(), 2);
+ let reattaches = fixture.state.reattaches.lock().unwrap().clone();
+ assert_eq!(reattaches.len(), 1);
+ assert_eq!(reattaches[0].last_response_id, None);
+ fixture.assert_release_all_count(1).await;
+}
+
+#[tokio::test]
+async fn operation_not_found_in_the_reattach_call_resends_the_execute_plan() {
+ // As the reference: the reattach call itself fails with
OPERATION_NOT_FOUND, before any
+ // response was read: the ExecutePlan is sent again.
+ let fixture = Fixture::start(
+ vec![
+ streamed(vec![]),
+ streamed(vec![response("r1", false), response("r2", true)]),
+ ],
+ vec![Step::Fail(operation_not_found())],
+ quick_retries(),
+ )
+ .await;
+ let mut stream =
bounded(fixture.client.execute_plan_reattachable(execute_request()))
+ .await
+ .unwrap();
+ let ids = drain(&mut stream).await.unwrap();
+ assert_eq!(ids, ["r1", "r2"]);
+ assert_eq!(*fixture.state.executes.lock().unwrap(), 2);
+ assert_eq!(fixture.state.reattaches.lock().unwrap().len(), 1);
+ fixture.assert_release_all_count(1).await;
+}
+
+#[tokio::test]
+async fn operation_not_found_after_a_response_is_the_error_and_is_not_resent()
{
+ // As the reference, which raises RESPONSE_ALREADY_RECEIVED here: once a
response was read,
+ // the execution is not sent again, and the stream fails.
+ let fixture = Fixture::start(
+ vec![streamed(vec![response("r1", false)])],
+ vec![Step::Fail(operation_not_found())],
+ quick_retries(),
+ )
+ .await;
+ let mut stream =
bounded(fixture.client.execute_plan_reattachable(execute_request()))
+ .await
+ .unwrap();
+ assert_eq!(
+ bounded(stream.message())
+ .await
+ .unwrap()
+ .unwrap()
+ .response_id,
+ "r1"
+ );
+ let error = bounded(stream.message()).await.unwrap_err();
+ assert!(
+ error.message.contains("INVALID_HANDLE.OPERATION_NOT_FOUND"),
+ "{error:?}"
+ );
+ assert_eq!(*fixture.state.executes.lock().unwrap(), 1);
+ fixture.assert_release_all_count(1).await;
+}
+
+/// The loopback service: ExecutePlan, ReattachExecute, ReleaseExecute and
Config.
+#[derive(Clone)]
+struct TestService(Arc<State>);
+
+impl NamedService for TestService {
+ const NAME: &'static str = "spark.connect.SparkConnectService";
+}
+
+impl Service<http::Request<Body>> for TestService {
+ type Response = http::Response<Body>;
+ type Error = Infallible;
+ type Future = BoxFuture<Self::Response, Self::Error>;
+
+ fn poll_ready(&mut self, _: &mut Context<'_>) -> Poll<Result<(),
Self::Error>> {
+ Poll::Ready(Ok(()))
+ }
+
+ fn call(&mut self, request: http::Request<Body>) -> Self::Future {
+ let state = self.0.clone();
+ match request.uri().path() {
+ "/spark.connect.SparkConnectService/ExecutePlan" => Box::pin(async
move {
+ let mut grpc =
tonic::server::Grpc::new(tonic_prost::ProstCodec::default());
+ Ok(grpc.server_streaming(Execute(state), request).await)
+ }),
+ "/spark.connect.SparkConnectService/ReattachExecute" =>
Box::pin(async move {
+ let mut grpc =
tonic::server::Grpc::new(tonic_prost::ProstCodec::default());
+ Ok(grpc.server_streaming(Reattach(state), request).await)
+ }),
+ "/spark.connect.SparkConnectService/ReleaseExecute" =>
Box::pin(async move {
+ let mut grpc =
tonic::server::Grpc::new(tonic_prost::ProstCodec::default());
+ Ok(grpc.unary(ReleaseRpc(state), request).await)
+ }),
+ "/spark.connect.SparkConnectService/Config" => Box::pin(async move
{
+ let mut grpc =
tonic::server::Grpc::new(tonic_prost::ProstCodec::default());
+ Ok(grpc.unary(Config(state), request).await)
+ }),
+ _ => Box::pin(async {
+ Ok(http::Response::builder()
+ .status(200)
+ .header("grpc-status", "12")
+ .header("content-type", "application/grpc")
+ .body(Body::empty())
+ .unwrap())
+ }),
+ }
+ }
+}
+
+/// Serves the next step of `script`, a call that stalls never returning its
headers.
+async fn serve(step: Step) -> Result<Response<Responses>, Status> {
+ match step {
+ Step::Stall => {
+ std::future::pending::<()>().await;
+ unreachable!("a stalled call never completes")
+ }
+ Step::Fail(status) => Err(status),
+ Step::Stream(items) => {
+ let responses: Responses = Box::pin(stream::iter(items));
+ Ok(Response::new(responses))
+ }
+ }
+}
+
+struct Execute(Arc<State>);
+
+impl ServerStreamingService<ExecutePlanRequest> for Execute {
+ type Response = ExecutePlanResponse;
+ type ResponseStream = Responses;
+ type Future = BoxFuture<Response<Responses>, Status>;
+
+ fn call(&mut self, request: Request<ExecutePlanRequest>) -> Self::Future {
+ let state = self.0.clone();
+ state.record_timeout("ExecutePlan", request.metadata());
+ *state.executes.lock().unwrap() += 1;
+ let operation =
request.get_ref().operation_id.clone().unwrap_or_default();
+ let step = state
+ .execute_script
+ .lock()
+ .unwrap()
+ .pop_front()
+ .unwrap_or_else(|| streamed(vec![response("complete", true)]));
+ if matches!(step, Step::Stream(_)) {
+ state.known.lock().unwrap().insert(operation);
+ }
+ Box::pin(serve(step))
+ }
+}
+
+struct Reattach(Arc<State>);
+
+impl ServerStreamingService<ReattachExecuteRequest> for Reattach {
+ type Response = ExecutePlanResponse;
+ type ResponseStream = Responses;
+ type Future = BoxFuture<Response<Responses>, Status>;
+
+ fn call(&mut self, request: Request<ReattachExecuteRequest>) ->
Self::Future {
+ let state = self.0.clone();
+ state.record_timeout("ReattachExecute", request.metadata());
+ let request = request.into_inner();
+ state.reattaches.lock().unwrap().push(request.clone());
+ let step = state
+ .reattach_script
+ .lock()
+ .unwrap()
+ .pop_front()
+ .unwrap_or_else(|| {
+ if state.known.lock().unwrap().contains(&request.operation_id)
{
+ streamed(vec![response("complete", true)])
+ } else {
+ Step::Fail(operation_not_found())
+ }
+ });
+ Box::pin(serve(step))
+ }
+}
+
+struct ReleaseRpc(Arc<State>);
+
+impl tonic::server::UnaryService<ReleaseExecuteRequest> for ReleaseRpc {
+ type Response = ReleaseExecuteResponse;
+ type Future = BoxFuture<Response<ReleaseExecuteResponse>, Status>;
+
+ fn call(&mut self, request: Request<ReleaseExecuteRequest>) ->
Self::Future {
+ let request = request.into_inner();
+ self.0.releases.lock().unwrap().push(CapturedRelease {
+ request: request.clone(),
+ });
+ Box::pin(async move {
+ Ok(Response::new(ReleaseExecuteResponse {
+ session_id: request.session_id,
+ operation_id: Some(request.operation_id),
+ ..Default::default()
+ }))
+ })
+ }
+}
+
+struct Config(Arc<State>);
+
+impl UnaryService<ConfigRequest> for Config {
+ type Response = ConfigResponse;
+ type Future = BoxFuture<Response<ConfigResponse>, Status>;
+
+ fn call(&mut self, request: Request<ConfigRequest>) -> Self::Future {
+ self.0.record_timeout("Config", request.metadata());
+ // Stalled before its headers, as a connection that never answers.
+ Box::pin(async {
+ std::future::pending::<()>().await;
+ unreachable!("a stalled call never completes")
+ })
+ }
+}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]