This is an automated email from the ASF dual-hosted git repository.

HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git


The following commit(s) were added to refs/heads/master by this push:
     new 91c1a37  [SPARK-61064][CONNECT] Raise DEADLINE_EXCEEDED and reattach 
when a per-RPC deadline passes before the response headers, and execute again 
on OPERATION_NOT_FOUND, as PySpark does
91c1a37 is described below

commit 91c1a3795d89d78b9c11d2449446ff7cd8563c64
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Sat Nov 7 11:41:53 2026 +0900

    [SPARK-61064][CONNECT] Raise DEADLINE_EXCEEDED and reattach when a per-RPC 
deadline passes before the response headers, and execute again on 
OPERATION_NOT_FOUND, as PySpark does
    
    ### What changes were proposed in this pull request?
    
    The per-RPC deadlines of PySpark Connect, which SPARK-60997 applied with 
tonic's `grpc-timeout`, now behave as PySpark's when they pass before the 
response headers (`crates/spark-connect-core/src/client.rs`):
    
    - `within_deadline` bounds each RPC as a whole by its deadline, and reads 
tonic's `CANCELLED` "Timeout expired", which tonic gives when its 
`grpc-timeout` passes before the headers, as the `DEADLINE_EXCEEDED` it is. A 
unary call stalled after its headers is now bounded as well.
    - A reattachable execution whose ExecutePlan or ReattachExecute passes its 
deadline reattaches, for the policy's bound of such retries from the first one, 
as the reference's `RetryException` does.
    - A reattach that finds `INVALID_HANDLE.OPERATION_NOT_FOUND` before any 
response, from the call or from its stream, sends the ExecutePlan again, as the 
reference iterator does.
    
    ### Why are the changes needed?
    
    [SPARK-61064](https://issues.apache.org/jira/browse/SPARK-61064): with a 
server that stalls before sending headers (an in-process mock that compares the 
two clients):
    
    | Scenario | Current master | This change and PySpark 4.3.0 |
    | --- | --- | --- |
    | First ExecutePlan of an execution stalls past its deadline | fails with 
`[The operation was cancelled] Timeout expired` after 10 minutes, no reattach | 
reattaches, executes again on OPERATION_NOT_FOUND, returns the rows |
    | ReattachExecute stalls past its deadline | fails the same way | 
reattaches again and returns the rows |
    | Config stalls past its deadline | `CANCELLED` | `DEADLINE_EXCEEDED`, with 
PySpark's message about the per-RPC timeouts |
    
    The existing tests stall response bodies, never headers, which the client 
already handled.
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes, in these failure cases, as in PySpark.
    
    ### How was this patch tested?
    
    - `crates/spark-connect-core/tests/deadlines.rs`, 8 tests against a 
loopback service that stalls before its headers: the three scenarios above, the 
bound of the retries past deadlines for the first call and for reattaches, and 
OPERATION_NOT_FOUND from the call, from the stream, and after a response (which 
still fails). Seven of them fail without the change.
    - The core tests (lib 119, reattach cleanup 11, user context 4, all passed) 
and the build of every integration test of `apache-spark-connect`.
    - The stalled-ExecutePlan scenario end to end through the Python drop-in: 
the rows are returned after the reattach and the second ExecutePlan.
    
    Unchanged and different from PySpark: PySpark also executes again on 
`INVALID_HANDLE.SESSION_NOT_FOUND`, and raises `RESPONSE_ALREADY_RECEIVED` for 
OPERATION_NOT_FOUND after a response.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Opus 5.5
    
    This pull request and its description were written by Isaac.
    
    Closes #255 from HyukjinKwon/SPARK-61064-deadlines.
    
    Authored-by: Hyukjin Kwon <[email protected]>
    Signed-off-by: Hyukjin Kwon <[email protected]>
---
 crates/spark-connect-core/src/client.rs      | 300 +++++++++----
 crates/spark-connect-core/tests/deadlines.rs | 614 +++++++++++++++++++++++++++
 2 files changed, 838 insertions(+), 76 deletions(-)

diff --git a/crates/spark-connect-core/src/client.rs 
b/crates/spark-connect-core/src/client.rs
index 786192a..7d68f87 100644
--- a/crates/spark-connect-core/src/client.rs
+++ b/crates/spark-connect-core/src/client.rs
@@ -160,6 +160,51 @@ fn with_deadline<T>(mut request: Request<T>, deadline: 
Option<Duration>) -> Requ
     request
 }
 
+/// `call` bounded as a whole by `deadline`, as gRPC bounds an RPC: past it, 
the call fails with
+/// `DEADLINE_EXCEEDED`, as PySpark's does, however far it got.
+///
+/// The `grpc-timeout` of [`with_deadline`] is also enforced by tonic, but 
only while the
+/// response headers are awaited, and an expiry there is a `CANCELLED` 
"Timeout expired". So
+/// the call is bounded here too, and that status is read as the deadline it 
is. A call stalled
+/// after its headers, as for a unary response, is then bounded as well.
+async fn within_deadline<T>(
+    deadline: Option<Duration>,
+    call: impl std::future::Future<Output = std::result::Result<T, 
tonic::Status>>,
+) -> std::result::Result<T, tonic::Status> {
+    let Some(deadline) = deadline else {
+        return call.await;
+    };
+    match tokio::time::timeout(deadline, call).await {
+        Ok(Err(status))
+            if status.code() == tonic::Code::Cancelled && status.message() == 
"Timeout expired" =>
+        {
+            Err(tonic::Status::deadline_exceeded("Deadline Exceeded"))
+        }
+        Ok(result) => result,
+        Err(_) => Err(tonic::Status::deadline_exceeded("Deadline Exceeded")),
+    }
+}
+
+/// Whether `status` says that the server does not know the operation of a 
reattach, which
+/// the reference reads as the initial ExecutePlan never having reached the 
server.
+fn is_operation_not_found(status: &tonic::Status) -> bool {
+    status
+        .message()
+        .contains("INVALID_HANDLE.OPERATION_NOT_FOUND")
+}
+
+/// Whether the retries that the reference makes as `RetryException`s have run 
for the policy's
+/// elapsed-time bound, measured from the first of them, which starts `since` 
when it is unset.
+fn retry_exception_bound_passed(
+    policy: &RetryPolicy,
+    since: &mut Option<std::time::Instant>,
+) -> bool {
+    let since = *since.get_or_insert_with(std::time::Instant::now);
+    policy
+        .max_retry_exception_elapsed_time_ms
+        .is_some_and(|ms| since.elapsed() >= Duration::from_millis(ms))
+}
+
 /// A Spark Connect client for communicating with a remote Spark server.
 ///
 /// Mirrors `pyspark.sql.connect.client.core.SparkConnectClient`.
@@ -420,15 +465,18 @@ impl SparkConnectClient {
             .max_decoding_message_size(GRPC_MAX_MESSAGE_LENGTH_DEFAULT)
             .max_encoding_message_size(GRPC_MAX_MESSAGE_LENGTH_DEFAULT);
         wait_ready(&mut grpc).await?;
-        let mut req = with_deadline(Request::new(request), 
self.rpc_deadlines.analyze_plan);
+        let deadline = self.rpc_deadlines.analyze_plan;
+        let mut req = with_deadline(Request::new(request), deadline);
         self._attach_metadata(&mut req);
         let path = tonic::codegen::http::uri::PathAndQuery::from_static(
             "/spark.connect.SparkConnectService/AnalyzePlan",
         );
-        let resp = grpc
-            .unary(req, path, crate::bytes_codec::BytesCodec)
-            .await
-            .map_err(SparkError::from_grpc_status)?;
+        let resp = within_deadline(
+            deadline,
+            grpc.unary(req, path, crate::bytes_codec::BytesCodec),
+        )
+        .await
+        .map_err(SparkError::from_grpc_status)?;
         Ok(resp.into_inner())
     }
 
@@ -441,13 +489,15 @@ impl SparkConnectClient {
     ) -> Result<AnalyzePlanResponse> {
         self.fill_user_context(&mut request.user_context);
         self.with_retry(|| {
-            let mut req = with_deadline(
-                Request::new(request.clone()),
-                self.rpc_deadlines.analyze_plan,
-            );
+            let deadline = self.rpc_deadlines.analyze_plan;
+            let mut req = with_deadline(Request::new(request.clone()), 
deadline);
             self._attach_metadata(&mut req);
             let mut stub = self.stub.clone();
-            async move { 
stub.analyze_plan(req).await.map(Response::into_inner) }
+            async move {
+                within_deadline(deadline, stub.analyze_plan(req))
+                    .await
+                    .map(Response::into_inner)
+            }
         })
         .await
     }
@@ -502,7 +552,11 @@ impl SparkConnectClient {
                 self._attach_metadata(&mut req);
                 stream_deadline = deadline.map(|d| tokio::time::Instant::now() 
+ d);
                 let mut stub = self.stub.clone();
-                async move { 
stub.execute_plan(req).await.map(Response::into_inner) }
+                async move {
+                    within_deadline(deadline, stub.execute_plan(req))
+                        .await
+                        .map(Response::into_inner)
+                }
             })
             .await;
         let stream = match stream {
@@ -525,6 +579,7 @@ impl SparkConnectClient {
             iter,
             stream,
             stream_deadline,
+            execute_again: false,
             done: false,
             released: false,
         })
@@ -539,12 +594,10 @@ impl SparkConnectClient {
         mut request: FetchErrorDetailsRequest,
     ) -> Result<FetchErrorDetailsResponse> {
         self.fill_user_context(&mut request.user_context);
-        let mut req = with_deadline(
-            Request::new(request),
-            self.rpc_deadlines.fetch_error_details,
-        );
+        let deadline = self.rpc_deadlines.fetch_error_details;
+        let mut req = with_deadline(Request::new(request), deadline);
         self._attach_metadata(&mut req);
-        let resp = self.stub.clone().fetch_error_details(req).await;
+        let resp = within_deadline(deadline, 
self.stub.clone().fetch_error_details(req)).await;
         resp.map(Response::into_inner)
             .map_err(SparkError::from_grpc_status)
     }
@@ -580,12 +633,12 @@ impl SparkConnectClient {
             }),
             ..Default::default()
         };
-        let mut req = with_deadline(
-            Request::new(request),
-            self.rpc_deadlines.fetch_error_details,
-        );
+        let deadline = self.rpc_deadlines.fetch_error_details;
+        let mut req = with_deadline(Request::new(request), deadline);
         self._attach_metadata(&mut req);
-        let response = self.stub.clone().fetch_error_details(req).await.ok()?;
+        let response = within_deadline(deadline, 
self.stub.clone().fetch_error_details(req))
+            .await
+            .ok()?;
         Some(response.into_inner())
     }
 
@@ -635,10 +688,15 @@ impl SparkConnectClient {
     pub async fn config(&self, mut request: ConfigRequest) -> 
Result<ConfigResponse> {
         self.fill_user_context(&mut request.user_context);
         self.with_retry(|| {
-            let mut req = with_deadline(Request::new(request.clone()), 
self.rpc_deadlines.config);
+            let deadline = self.rpc_deadlines.config;
+            let mut req = with_deadline(Request::new(request.clone()), 
deadline);
             self._attach_metadata(&mut req);
             let mut stub = self.stub.clone();
-            async move { stub.config(req).await.map(Response::into_inner) }
+            async move {
+                within_deadline(deadline, stub.config(req))
+                    .await
+                    .map(Response::into_inner)
+            }
         })
         .await
     }
@@ -649,11 +707,15 @@ impl SparkConnectClient {
     pub async fn interrupt(&self, mut request: InterruptRequest) -> 
Result<InterruptResponse> {
         self.fill_user_context(&mut request.user_context);
         self.with_retry(|| {
-            let mut req =
-                with_deadline(Request::new(request.clone()), 
self.rpc_deadlines.interrupt);
+            let deadline = self.rpc_deadlines.interrupt;
+            let mut req = with_deadline(Request::new(request.clone()), 
deadline);
             self._attach_metadata(&mut req);
             let mut stub = self.stub.clone();
-            async move { stub.interrupt(req).await.map(Response::into_inner) }
+            async move {
+                within_deadline(deadline, stub.interrupt(req))
+                    .await
+                    .map(Response::into_inner)
+            }
         })
         .await
     }
@@ -957,13 +1019,15 @@ impl SparkConnectClient {
         };
         let response = self
             .with_retry(|| {
-                let mut req = with_deadline(
-                    Request::new(request.clone()),
-                    self.rpc_deadlines.clone_session,
-                );
+                let deadline = self.rpc_deadlines.clone_session;
+                let mut req = with_deadline(Request::new(request.clone()), 
deadline);
                 self._attach_metadata(&mut req);
                 let mut stub = self.stub.clone();
-                async move { 
stub.clone_session(req).await.map(Response::into_inner) }
+                async move {
+                    within_deadline(deadline, stub.clone_session(req))
+                        .await
+                        .map(Response::into_inner)
+                }
             })
             .await?;
         if let Some(requested) = new_session_id {
@@ -987,9 +1051,10 @@ impl SparkConnectClient {
             ..Default::default()
         });
 
-        let mut req = with_deadline(Request::new(request), 
self.rpc_deadlines.release_session);
+        let deadline = self.rpc_deadlines.release_session;
+        let mut req = with_deadline(Request::new(request), deadline);
         self._attach_metadata(&mut req);
-        match self.stub.clone().release_session(req).await {
+        match within_deadline(deadline, 
self.stub.clone().release_session(req)).await {
             Ok(_) => Ok(()),
             Err(status) => Err(self.error_from_status(status).await),
         }
@@ -1138,14 +1203,10 @@ impl SparkConnectClient {
                 request
             })
             .collect();
-        let mut req = with_deadline(
-            Request::new(futures::stream::iter(requests)),
-            self.rpc_deadlines.add_artifacts,
-        );
+        let deadline = self.rpc_deadlines.add_artifacts;
+        let mut req = 
with_deadline(Request::new(futures::stream::iter(requests)), deadline);
         self._attach_metadata(&mut req);
-        self.stub
-            .clone()
-            .add_artifacts(req)
+        within_deadline(deadline, self.stub.clone().add_artifacts(req))
             .await
             .map(Response::into_inner)
     }
@@ -1175,9 +1236,12 @@ impl SparkConnectClient {
         request.client_type = Some(self.user_agent.clone());
         request.names = names.to_vec();
 
-        let mut req = with_deadline(Request::new(request), 
self.rpc_deadlines.artifact_status);
+        let deadline = self.rpc_deadlines.artifact_status;
+        let mut req = with_deadline(Request::new(request), deadline);
         self._attach_metadata(&mut req);
-        let response = 
self.stub.clone().artifact_status(req).await?.into_inner();
+        let response = within_deadline(deadline, 
self.stub.clone().artifact_status(req))
+            .await?
+            .into_inner();
         Ok(response
             .statuses
             .into_iter()
@@ -1235,6 +1299,9 @@ pub struct ReattachableResponseStream {
     stream: Option<Streaming<ExecutePlanResponse>>,
     /// When the deadline of the stream passes, as gRPC's of the call that 
opened it.
     stream_deadline: Option<tokio::time::Instant>,
+    /// Whether the next stream is opened by sending the ExecutePlan again, 
rather than by
+    /// reattaching: the server does not know the execution, which never 
reached it.
+    execute_again: bool,
     done: bool,
     /// Whether `ReleaseAll` was sent, after which nothing else is released.
     released: bool,
@@ -1321,12 +1388,23 @@ impl ReattachableResponseStream {
         // As the reference iterator's `_has_next` retries with a `Retrying` 
of its own, the
         // reattaches of each response have the attempts and the elapsed time 
of the policy.
         let mut retry_state = RetryPolicyState::new(self.retry_policy.clone());
-        // When the first stream of this response passed its deadline, for the 
bound of the
-        // reattaches after deadlines, as of the reference `RetryException`s.
-        let mut past_deadlines_since: Option<std::time::Instant> = None;
+        // The clock of the retries that the reference makes as 
`RetryException`s, bounded by the
+        // policy's elapsed time: it starts at the first deadline that passes, 
the deadline of
+        // the initial ExecutePlan included, and the reattaches after a 
deadline or an unknown
+        // execution are its retries.
+        let mut retry_exception_since: Option<std::time::Instant> = None;
+        if self.stream.is_none()
+            && retry_exception_bound_passed(&self.retry_policy, &mut 
retry_exception_since)
+        {
+            // The initial ExecutePlan passed its deadline, a RetryException 
that fails at once
+            // when the bound of the retries has passed, as it has when the 
bound is 0.
+            let status = tonic::Status::deadline_exceeded("Deadline Exceeded");
+            return Err(self.fail_with_status(status).await);
+        }
         loop {
             let Some(stream) = self.stream.as_mut() else {
-                self.reattach(&mut retry_state).await?;
+                self.reattach(&mut retry_state, &mut retry_exception_since)
+                    .await?;
                 continue;
             };
             let next = match self.stream_deadline {
@@ -1349,7 +1427,10 @@ impl ReattachableResponseStream {
                 // The server completes a reattachable stream before 
`ResultComplete` on
                 // purpose, after `senderMaxStreamDuration` or 
`senderMaxStreamSize`: the rest
                 // of the result is read by reattaching.
-                Ok(None) => self.reattach(&mut retry_state).await?,
+                Ok(None) => {
+                    self.reattach(&mut retry_state, &mut retry_exception_since)
+                        .await?
+                }
                 // As the reference iterator: the per-RPC deadline of the 
stream passed, and the
                 // execution goes on on the server, so the iteration 
reattaches at once, for up
                 // to the policy's bound of such retries.
@@ -1357,55 +1438,122 @@ impl ReattachableResponseStream {
                     if status.code() == tonic::Code::DeadlineExceeded
                         && !self.iter.is_completed().await =>
                 {
-                    let since = 
*past_deadlines_since.get_or_insert_with(std::time::Instant::now);
-                    let bound = 
self.retry_policy.max_retry_exception_elapsed_time_ms;
-                    if bound.is_some_and(|ms| since.elapsed() >= 
Duration::from_millis(ms)) {
-                        let error = 
self.client.error_from_status(status).await;
-                        return Err(self.fail(error));
+                    if retry_exception_bound_passed(&self.retry_policy, &mut 
retry_exception_since)
+                    {
+                        return Err(self.fail_with_status(status).await);
                     }
                     self.stream = None;
                 }
+                // As the reference iterator: the server does not know the 
execution, so the
+                // initial ExecutePlan never reached it, and it is sent again, 
if no response was
+                // read yet.
+                Err(status)
+                    if is_operation_not_found(&status)
+                        && self.iter.last_response_id().await.is_none()
+                        && !self.iter.is_completed().await =>
+                {
+                    if retry_exception_bound_passed(&self.retry_policy, &mut 
retry_exception_since)
+                    {
+                        return Err(self.fail_with_status(status).await);
+                    }
+                    self.execute_again = true;
+                    self.stream = None;
+                }
                 Err(status) => {
                     // Only reattach a still-running execution on a retriable 
drop.
                     if self.iter.is_completed().await || 
!self.retry_policy.can_retry(&status) {
-                        let error = 
self.client.error_from_status(status).await;
-                        return Err(self.fail(error));
+                        return Err(self.fail_with_status(status).await);
                     }
-                    self.reattach(&mut retry_state).await?;
+                    self.reattach(&mut retry_state, &mut retry_exception_since)
+                        .await?;
                 }
             }
         }
     }
 
-    /// Resumes the execution from the last response read, retrying transient 
failures under
-    /// `retry_state`.
-    async fn reattach(&mut self, retry_state: &mut RetryPolicyState) -> 
Result<()> {
-        let deadline = self.client.rpc_deadlines.reattach_execute;
+    /// Opens the stream of the execution again: by reattaching to it from the 
last response
+    /// read, or, when the server does not know it, by sending the ExecutePlan 
again. Transient
+    /// failures are retried under `retry_state`, and a deadline or an unknown 
execution as the
+    /// reference's `RetryException`s, bounded by `retry_exception_since`.
+    async fn reattach(
+        &mut self,
+        retry_state: &mut RetryPolicyState,
+        retry_exception_since: &mut Option<std::time::Instant>,
+    ) -> Result<()> {
         loop {
-            let mut req = 
Request::new(self.iter.create_reattach_request().await);
-            self.attach_metadata(&mut req);
-            // After the metadata, which the deadline is a header of.
-            let req = with_deadline(req, deadline);
-            self.stream_deadline = deadline.map(|d| 
tokio::time::Instant::now() + d);
-            match self.client.stub.clone().reattach_execute(req).await {
-                Ok(r) => {
-                    self.stream = Some(r.into_inner());
+            let opened = if self.execute_again {
+                self.open_execute_plan_again().await
+            } else {
+                self.open_reattach().await
+            };
+            let status = match opened {
+                Ok(stream) => {
+                    self.stream = Some(stream);
+                    self.execute_again = false;
                     return Ok(());
                 }
-                Err(e) => {
-                    if self.retry_policy.can_retry(&e) {
-                        if let Some(w) = retry_state.next_attempt(None) {
-                            tokio::time::sleep(Duration::from_millis(w)).await;
-                            continue;
-                        }
-                    }
-                    let error = self.client.error_from_status(e).await;
-                    return Err(self.fail(error));
+                Err(status) => status,
+            };
+            if is_operation_not_found(&status) && 
self.iter.last_response_id().await.is_none() {
+                if retry_exception_bound_passed(&self.retry_policy, 
retry_exception_since) {
+                    return Err(self.fail_with_status(status).await);
                 }
+                self.execute_again = true;
+                continue;
             }
+            if status.code() == tonic::Code::DeadlineExceeded {
+                // As the reference: a call past its deadline, whether it is 
sent again or
+                // reattached, is retried at once, the execution going on on 
the server.
+                if retry_exception_bound_passed(&self.retry_policy, 
retry_exception_since) {
+                    return Err(self.fail_with_status(status).await);
+                }
+                self.execute_again = false;
+                continue;
+            }
+            if self.retry_policy.can_retry(&status) {
+                if let Some(w) = retry_state.next_attempt(None) {
+                    tokio::time::sleep(Duration::from_millis(w)).await;
+                    continue;
+                }
+            }
+            return Err(self.fail_with_status(status).await);
         }
     }
 
+    /// ReattachExecute from the last response read, bounded by the deadline 
of the call.
+    async fn open_reattach(
+        &mut self,
+    ) -> std::result::Result<Streaming<ExecutePlanResponse>, tonic::Status> {
+        let deadline = self.client.rpc_deadlines.reattach_execute;
+        let mut req = Request::new(self.iter.create_reattach_request().await);
+        self.attach_metadata(&mut req);
+        // After the metadata, which the deadline is a header of.
+        let req = with_deadline(req, deadline);
+        self.stream_deadline = deadline.map(|d| tokio::time::Instant::now() + 
d);
+        let response =
+            within_deadline(deadline, 
self.client.stub.clone().reattach_execute(req)).await?;
+        Ok(response.into_inner())
+    }
+
+    /// ExecutePlan of the request again, as the initial one, bounded by its 
deadline.
+    async fn open_execute_plan_again(
+        &mut self,
+    ) -> std::result::Result<Streaming<ExecutePlanResponse>, tonic::Status> {
+        let deadline = self.client.rpc_deadlines.reattachable_execute_plan;
+        let mut req = with_deadline(Request::new(self.iter.request().clone()), 
deadline);
+        self.client._attach_metadata(&mut req);
+        self.stream_deadline = deadline.map(|d| tokio::time::Instant::now() + 
d);
+        let response =
+            within_deadline(deadline, 
self.client.stub.clone().execute_plan(req)).await?;
+        Ok(response.into_inner())
+    }
+
+    /// Ends the stream with the error of `status`, as [`Self::fail`].
+    async fn fail_with_status(&mut self, status: tonic::Status) -> SparkError {
+        let error = self.client.error_from_status(status).await;
+        self.fail(error)
+    }
+
     /// Ends the stream with `error`, releasing the execution as the reference 
client does
     /// when an execution fails.
     /// The error carries the ExecutePlan operation id 
(`SparkConnectException.operation_id`).
diff --git a/crates/spark-connect-core/tests/deadlines.rs 
b/crates/spark-connect-core/tests/deadlines.rs
new file mode 100644
index 0000000..0a4282f
--- /dev/null
+++ b/crates/spark-connect-core/tests/deadlines.rs
@@ -0,0 +1,614 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Per-RPC deadlines that expire before the response headers, and the 
reattach to an
+//! execution that the server does not know. No Spark server required: a 
loopback tonic
+//! service stalls its calls before they send headers, as a stalled connection 
does.
+
+use std::collections::{HashSet, VecDeque};
+use std::convert::Infallible;
+use std::future::Future;
+use std::pin::Pin;
+use std::sync::{Arc, Mutex};
+use std::task::{Context, Poll};
+use std::time::Duration;
+
+use futures::{stream, Stream};
+use spark_connect_core::{ChannelBuilder, RetryPolicy, RpcDeadlines, 
SparkConnectClient};
+use spark_connect_proto::execute_plan_response::{ResponseType, ResultComplete};
+use spark_connect_proto::release_execute_request::Release;
+use spark_connect_proto::{
+    ConfigRequest, ConfigResponse, ExecutePlanRequest, ExecutePlanResponse, 
ReattachExecuteRequest,
+    ReleaseExecuteRequest, ReleaseExecuteResponse, UserContext,
+};
+use tokio::net::TcpListener;
+use tonic::body::Body;
+use tonic::codegen::{http, BoxFuture, Service};
+use tonic::metadata::MetadataMap;
+use tonic::server::{NamedService, ServerStreamingService, UnaryService};
+use tonic::transport::Server;
+use tonic::{Code, Request, Response, Status};
+
+/// The per-RPC deadline of the tests: short, so that the stalls end quickly.
+const DEADLINE: Duration = Duration::from_millis(200);
+/// Bound of every RPC of a test, so that a call that waits forever fails the 
test.
+const WAIT: Duration = Duration::from_secs(5);
+const OPERATION: &str = "00000000-0000-4000-8000-000000000002";
+type Responses = Pin<Box<dyn Stream<Item = Result<ExecutePlanResponse, 
Status>> + Send>>;
+
+async fn bounded<T>(future: impl Future<Output = T>) -> T {
+    tokio::time::timeout(WAIT, future)
+        .await
+        .expect("test RPC waited past the bound")
+}
+
+/// What the server does with one call: stall before its headers, fail the 
call, or stream
+/// these items (an error item ends the stream with that status).
+enum Step {
+    Stall,
+    Fail(Status),
+    Stream(Vec<Result<ExecutePlanResponse, Status>>),
+}
+
+fn operation_not_found() -> Status {
+    // As the server reports an execution it never started: an INTERNAL status 
whose message
+    // carries the error class.
+    Status::internal(format!(
+        "[INVALID_HANDLE.OPERATION_NOT_FOUND] Operation not found: {OPERATION}"
+    ))
+}
+
+#[derive(Clone)]
+struct CapturedRelease {
+    request: ReleaseExecuteRequest,
+}
+
+struct State {
+    /// What each ExecutePlan call does, in order. When none is left, it 
completes the result.
+    execute_script: Mutex<VecDeque<Step>>,
+    /// What each ReattachExecute call does, in order. When none is left, it 
completes the
+    /// result of an execution that the server knows, and fails with 
OPERATION_NOT_FOUND for
+    /// one it does not.
+    reattach_script: Mutex<VecDeque<Step>>,
+    /// The operations that the server started, that is, whose ExecutePlan 
streamed.
+    known: Mutex<HashSet<String>>,
+    /// The `grpc-timeout` of each call, in order.
+    timeouts: Mutex<Vec<(&'static str, Option<String>)>>,
+    executes: Mutex<usize>,
+    reattaches: Mutex<Vec<ReattachExecuteRequest>>,
+    releases: Mutex<Vec<CapturedRelease>>,
+}
+
+impl State {
+    fn new(execute: Vec<Step>, reattach: Vec<Step>) -> Self {
+        State {
+            execute_script: Mutex::new(execute.into()),
+            reattach_script: Mutex::new(reattach.into()),
+            known: Mutex::new(HashSet::new()),
+            timeouts: Mutex::new(Vec::new()),
+            executes: Mutex::new(0),
+            reattaches: Mutex::new(Vec::new()),
+            releases: Mutex::new(Vec::new()),
+        }
+    }
+
+    fn record_timeout(&self, call: &'static str, metadata: &MetadataMap) {
+        let timeout = metadata
+            .get("grpc-timeout")
+            .map(|value| value.to_str().unwrap().to_string());
+        self.timeouts.lock().unwrap().push((call, timeout));
+    }
+
+    fn release_all_count(&self) -> usize {
+        self.releases
+            .lock()
+            .unwrap()
+            .iter()
+            .filter(|r| matches!(r.request.release, 
Some(Release::ReleaseAll(_))))
+            .count()
+    }
+}
+
+struct Fixture {
+    state: Arc<State>,
+    server: tokio::task::JoinHandle<()>,
+    client: SparkConnectClient,
+}
+
+impl Fixture {
+    /// A client with `deadline` for the calls of a reattachable execution and 
for Config, and
+    /// the given retry policy.
+    async fn start(execute: Vec<Step>, reattach: Vec<Step>, retry_policy: 
RetryPolicy) -> Self {
+        let state = Arc::new(State::new(execute, reattach));
+        let listener = TcpListener::bind("127.0.0.1:0").await.unwrap();
+        let address = listener.local_addr().unwrap();
+        let incoming = stream::unfold(listener, |listener| async {
+            Some((listener.accept().await.map(|(socket, _)| socket), listener))
+        });
+        let service = TestService(state.clone());
+        let server = tokio::spawn(async move {
+            Server::builder()
+                .add_service(service)
+                .serve_with_incoming(incoming)
+                .await
+                .unwrap();
+        });
+        let builder = ChannelBuilder::parse(&format!(
+            "sc://{address}/;user_id=transport-user;token=fixture-token"
+        ))
+        .unwrap();
+        let mut client = bounded(SparkConnectClient::connect(&builder))
+            .await
+            .unwrap();
+        client.set_retry_policy(retry_policy);
+        client.set_rpc_deadlines(RpcDeadlines {
+            reattachable_execute_plan: Some(DEADLINE),
+            reattach_execute: Some(DEADLINE),
+            config: Some(DEADLINE),
+            ..RpcDeadlines::default()
+        });
+        Self {
+            state,
+            server,
+            client,
+        }
+    }
+
+    /// Waits until the release of the whole execution is sent once, and 
checks that it is not
+    /// sent again.
+    async fn assert_release_all_count(&self, expected: usize) {
+        let state = self.state.clone();
+        bounded(async {
+            while state.release_all_count() < expected {
+                tokio::time::sleep(Duration::from_millis(5)).await;
+            }
+        })
+        .await;
+        tokio::time::sleep(Duration::from_millis(100)).await;
+        assert_eq!(self.state.release_all_count(), expected);
+    }
+}
+
+impl Drop for Fixture {
+    fn drop(&mut self) {
+        self.server.abort();
+    }
+}
+
+fn execute_request() -> ExecutePlanRequest {
+    ExecutePlanRequest {
+        session_id: "00000000-0000-4000-8000-000000000001".into(),
+        user_context: Some(UserContext {
+            user_id: "request-user".into(),
+            ..Default::default()
+        }),
+        operation_id: Some(OPERATION.into()),
+        client_type: Some("deadline-regression".into()),
+        ..Default::default()
+    }
+}
+
+fn response(id: &str, complete: bool) -> ExecutePlanResponse {
+    ExecutePlanResponse {
+        response_id: id.into(),
+        response_type: 
complete.then_some(ResponseType::ResultComplete(ResultComplete {})),
+        ..Default::default()
+    }
+}
+
+/// A stream of `items`, as the server sends them.
+fn streamed(items: Vec<ExecutePlanResponse>) -> Step {
+    Step::Stream(items.into_iter().map(Ok).collect())
+}
+
+/// The retry policy of the tests: as the cleanup tests, with short backoffs.
+fn quick_retries() -> RetryPolicy {
+    RetryPolicy {
+        max_retries: Some(2),
+        initial_backoff_ms: 1,
+        max_backoff_ms: Some(2),
+        ..RetryPolicy::default()
+    }
+}
+
+/// Drains `stream`, returning the ids of its responses, or the error that 
ends it.
+async fn drain(
+    stream: &mut spark_connect_core::client::ReattachableResponseStream,
+) -> Result<Vec<String>, spark_connect_core::SparkError> {
+    let mut ids = Vec::new();
+    while let Some(response) = bounded(stream.message()).await? {
+        ids.push(response.response_id);
+    }
+    Ok(ids)
+}
+
+#[tokio::test]
+async fn stalled_first_execute_plan_reattaches_then_resends_the_execution() {
+    // As the reference: the first ExecutePlan sends no headers, so its 
deadline passes. The
+    // execution is reattached, and as the server does not know it, it is sent 
again, and its
+    // results are returned.
+    let fixture = Fixture::start(
+        vec![
+            Step::Stall,
+            streamed(vec![response("r1", false), response("r2", true)]),
+        ],
+        vec![],
+        quick_retries(),
+    )
+    .await;
+    let mut stream = 
bounded(fixture.client.execute_plan_reattachable(execute_request()))
+        .await
+        .unwrap();
+    let ids = drain(&mut stream).await.unwrap();
+    assert_eq!(ids, ["r1", "r2"]);
+
+    assert_eq!(*fixture.state.executes.lock().unwrap(), 2);
+    let reattaches = fixture.state.reattaches.lock().unwrap().clone();
+    assert_eq!(reattaches.len(), 1);
+    assert_eq!(reattaches[0].last_response_id, None);
+    let timeouts = fixture.state.timeouts.lock().unwrap().clone();
+    let calls: Vec<_> = timeouts.iter().map(|(call, _)| *call).collect();
+    assert_eq!(calls, ["ExecutePlan", "ReattachExecute", "ExecutePlan"]);
+    assert!(
+        timeouts.iter().all(|(_, timeout)| timeout.is_some()),
+        "{timeouts:?}"
+    );
+    fixture.assert_release_all_count(1).await;
+}
+
+#[tokio::test]
+async fn stalled_reattach_is_reattached_again() {
+    // As the reference: a ReattachExecute that sends no headers past its 
deadline is retried,
+    // the execution going on on the server, and the results after the last 
response read are
+    // returned.
+    let fixture = Fixture::start(
+        vec![streamed(vec![response("r1", false)])],
+        vec![Step::Stall, streamed(vec![response("r2", true)])],
+        quick_retries(),
+    )
+    .await;
+    let mut stream = 
bounded(fixture.client.execute_plan_reattachable(execute_request()))
+        .await
+        .unwrap();
+    let ids = drain(&mut stream).await.unwrap();
+    assert_eq!(ids, ["r1", "r2"]);
+
+    let reattaches = fixture.state.reattaches.lock().unwrap().clone();
+    let resumed: Vec<_> = reattaches
+        .iter()
+        .map(|r| r.last_response_id.clone())
+        .collect();
+    assert_eq!(resumed, [Some("r1".to_string()), Some("r1".to_string())]);
+    assert_eq!(*fixture.state.executes.lock().unwrap(), 1);
+    fixture.assert_release_all_count(1).await;
+}
+
+#[tokio::test]
+async fn unary_call_past_its_deadline_is_deadline_exceeded() {
+    // As the reference: a unary call stalled past its deadline raises 
DEADLINE_EXCEEDED, with
+    // the message about the per-RPC timeouts.
+    let fixture = Fixture::start(vec![], vec![], quick_retries()).await;
+    let error = bounded(fixture.client.config(ConfigRequest::default()))
+        .await
+        .unwrap_err();
+    assert_eq!(
+        error.grpc_code,
+        Some(Code::DeadlineExceeded as i32),
+        "{error:?}"
+    );
+    assert!(
+        error.message.contains("RPC deadline exceeded"),
+        "unexpected error: {error:?}"
+    );
+    assert!(!error.message.contains("Timeout expired"), "{error:?}");
+}
+
+#[tokio::test]
+async fn execute_plan_past_its_deadline_is_bounded_as_the_reference_retries() {
+    // As the reference: the deadline of the initial ExecutePlan is a 
RetryException, which is
+    // the error once the bound of the retries has passed. With no bound, the 
error is the
+    // deadline, and nothing is reattached.
+    let fixture = Fixture::start(
+        vec![Step::Stall],
+        vec![],
+        RetryPolicy {
+            max_retry_exception_elapsed_time_ms: Some(0),
+            ..quick_retries()
+        },
+    )
+    .await;
+    let mut stream = 
bounded(fixture.client.execute_plan_reattachable(execute_request()))
+        .await
+        .unwrap();
+    let error = bounded(stream.message()).await.unwrap_err();
+    assert_eq!(
+        error.grpc_code,
+        Some(Code::DeadlineExceeded as i32),
+        "{error:?}"
+    );
+    assert!(error.message.contains("RPC deadline exceeded"), "{error:?}");
+    assert!(fixture.state.reattaches.lock().unwrap().is_empty());
+    assert_eq!(*fixture.state.executes.lock().unwrap(), 1);
+    fixture.assert_release_all_count(1).await;
+}
+
+#[tokio::test]
+async fn reattach_past_its_deadline_is_the_error_once_the_bound_passes() {
+    // As the reference: a reattach past its deadline is retried, and past the 
bound of those
+    // retries, the deadline is the error and the execution is released.
+    let fixture = Fixture::start(
+        vec![streamed(vec![response("r1", false)])],
+        vec![Step::Stall],
+        RetryPolicy {
+            max_retry_exception_elapsed_time_ms: Some(0),
+            ..quick_retries()
+        },
+    )
+    .await;
+    let mut stream = 
bounded(fixture.client.execute_plan_reattachable(execute_request()))
+        .await
+        .unwrap();
+    assert_eq!(
+        bounded(stream.message())
+            .await
+            .unwrap()
+            .unwrap()
+            .response_id,
+        "r1"
+    );
+    let error = bounded(stream.message()).await.unwrap_err();
+    assert_eq!(
+        error.grpc_code,
+        Some(Code::DeadlineExceeded as i32),
+        "{error:?}"
+    );
+    assert_eq!(fixture.state.reattaches.lock().unwrap().len(), 1);
+    fixture.assert_release_all_count(1).await;
+}
+
+#[tokio::test]
+async fn operation_not_found_before_any_response_resends_the_execute_plan() {
+    // As the reference: the ExecutePlan of the execution streams no response, 
so it is
+    // reattached, and the reattach reports that the server does not know the 
operation, which
+    // is then sent again, its results returned.
+    let fixture = Fixture::start(
+        vec![
+            streamed(vec![]),
+            streamed(vec![response("r1", false), response("r2", true)]),
+        ],
+        vec![Step::Stream(vec![Err(operation_not_found())])],
+        quick_retries(),
+    )
+    .await;
+    let mut stream = 
bounded(fixture.client.execute_plan_reattachable(execute_request()))
+        .await
+        .unwrap();
+    let ids = drain(&mut stream).await.unwrap();
+    assert_eq!(ids, ["r1", "r2"]);
+
+    assert_eq!(*fixture.state.executes.lock().unwrap(), 2);
+    let reattaches = fixture.state.reattaches.lock().unwrap().clone();
+    assert_eq!(reattaches.len(), 1);
+    assert_eq!(reattaches[0].last_response_id, None);
+    fixture.assert_release_all_count(1).await;
+}
+
+#[tokio::test]
+async fn operation_not_found_in_the_reattach_call_resends_the_execute_plan() {
+    // As the reference: the reattach call itself fails with 
OPERATION_NOT_FOUND, before any
+    // response was read: the ExecutePlan is sent again.
+    let fixture = Fixture::start(
+        vec![
+            streamed(vec![]),
+            streamed(vec![response("r1", false), response("r2", true)]),
+        ],
+        vec![Step::Fail(operation_not_found())],
+        quick_retries(),
+    )
+    .await;
+    let mut stream = 
bounded(fixture.client.execute_plan_reattachable(execute_request()))
+        .await
+        .unwrap();
+    let ids = drain(&mut stream).await.unwrap();
+    assert_eq!(ids, ["r1", "r2"]);
+    assert_eq!(*fixture.state.executes.lock().unwrap(), 2);
+    assert_eq!(fixture.state.reattaches.lock().unwrap().len(), 1);
+    fixture.assert_release_all_count(1).await;
+}
+
+#[tokio::test]
+async fn operation_not_found_after_a_response_is_the_error_and_is_not_resent() 
{
+    // As the reference, which raises RESPONSE_ALREADY_RECEIVED here: once a 
response was read,
+    // the execution is not sent again, and the stream fails.
+    let fixture = Fixture::start(
+        vec![streamed(vec![response("r1", false)])],
+        vec![Step::Fail(operation_not_found())],
+        quick_retries(),
+    )
+    .await;
+    let mut stream = 
bounded(fixture.client.execute_plan_reattachable(execute_request()))
+        .await
+        .unwrap();
+    assert_eq!(
+        bounded(stream.message())
+            .await
+            .unwrap()
+            .unwrap()
+            .response_id,
+        "r1"
+    );
+    let error = bounded(stream.message()).await.unwrap_err();
+    assert!(
+        error.message.contains("INVALID_HANDLE.OPERATION_NOT_FOUND"),
+        "{error:?}"
+    );
+    assert_eq!(*fixture.state.executes.lock().unwrap(), 1);
+    fixture.assert_release_all_count(1).await;
+}
+
+/// The loopback service: ExecutePlan, ReattachExecute, ReleaseExecute and 
Config.
+#[derive(Clone)]
+struct TestService(Arc<State>);
+
+impl NamedService for TestService {
+    const NAME: &'static str = "spark.connect.SparkConnectService";
+}
+
+impl Service<http::Request<Body>> for TestService {
+    type Response = http::Response<Body>;
+    type Error = Infallible;
+    type Future = BoxFuture<Self::Response, Self::Error>;
+
+    fn poll_ready(&mut self, _: &mut Context<'_>) -> Poll<Result<(), 
Self::Error>> {
+        Poll::Ready(Ok(()))
+    }
+
+    fn call(&mut self, request: http::Request<Body>) -> Self::Future {
+        let state = self.0.clone();
+        match request.uri().path() {
+            "/spark.connect.SparkConnectService/ExecutePlan" => Box::pin(async 
move {
+                let mut grpc = 
tonic::server::Grpc::new(tonic_prost::ProstCodec::default());
+                Ok(grpc.server_streaming(Execute(state), request).await)
+            }),
+            "/spark.connect.SparkConnectService/ReattachExecute" => 
Box::pin(async move {
+                let mut grpc = 
tonic::server::Grpc::new(tonic_prost::ProstCodec::default());
+                Ok(grpc.server_streaming(Reattach(state), request).await)
+            }),
+            "/spark.connect.SparkConnectService/ReleaseExecute" => 
Box::pin(async move {
+                let mut grpc = 
tonic::server::Grpc::new(tonic_prost::ProstCodec::default());
+                Ok(grpc.unary(ReleaseRpc(state), request).await)
+            }),
+            "/spark.connect.SparkConnectService/Config" => Box::pin(async move 
{
+                let mut grpc = 
tonic::server::Grpc::new(tonic_prost::ProstCodec::default());
+                Ok(grpc.unary(Config(state), request).await)
+            }),
+            _ => Box::pin(async {
+                Ok(http::Response::builder()
+                    .status(200)
+                    .header("grpc-status", "12")
+                    .header("content-type", "application/grpc")
+                    .body(Body::empty())
+                    .unwrap())
+            }),
+        }
+    }
+}
+
+/// Serves the next step of `script`, a call that stalls never returning its 
headers.
+async fn serve(step: Step) -> Result<Response<Responses>, Status> {
+    match step {
+        Step::Stall => {
+            std::future::pending::<()>().await;
+            unreachable!("a stalled call never completes")
+        }
+        Step::Fail(status) => Err(status),
+        Step::Stream(items) => {
+            let responses: Responses = Box::pin(stream::iter(items));
+            Ok(Response::new(responses))
+        }
+    }
+}
+
+struct Execute(Arc<State>);
+
+impl ServerStreamingService<ExecutePlanRequest> for Execute {
+    type Response = ExecutePlanResponse;
+    type ResponseStream = Responses;
+    type Future = BoxFuture<Response<Responses>, Status>;
+
+    fn call(&mut self, request: Request<ExecutePlanRequest>) -> Self::Future {
+        let state = self.0.clone();
+        state.record_timeout("ExecutePlan", request.metadata());
+        *state.executes.lock().unwrap() += 1;
+        let operation = 
request.get_ref().operation_id.clone().unwrap_or_default();
+        let step = state
+            .execute_script
+            .lock()
+            .unwrap()
+            .pop_front()
+            .unwrap_or_else(|| streamed(vec![response("complete", true)]));
+        if matches!(step, Step::Stream(_)) {
+            state.known.lock().unwrap().insert(operation);
+        }
+        Box::pin(serve(step))
+    }
+}
+
+struct Reattach(Arc<State>);
+
+impl ServerStreamingService<ReattachExecuteRequest> for Reattach {
+    type Response = ExecutePlanResponse;
+    type ResponseStream = Responses;
+    type Future = BoxFuture<Response<Responses>, Status>;
+
+    fn call(&mut self, request: Request<ReattachExecuteRequest>) -> 
Self::Future {
+        let state = self.0.clone();
+        state.record_timeout("ReattachExecute", request.metadata());
+        let request = request.into_inner();
+        state.reattaches.lock().unwrap().push(request.clone());
+        let step = state
+            .reattach_script
+            .lock()
+            .unwrap()
+            .pop_front()
+            .unwrap_or_else(|| {
+                if state.known.lock().unwrap().contains(&request.operation_id) 
{
+                    streamed(vec![response("complete", true)])
+                } else {
+                    Step::Fail(operation_not_found())
+                }
+            });
+        Box::pin(serve(step))
+    }
+}
+
+struct ReleaseRpc(Arc<State>);
+
+impl tonic::server::UnaryService<ReleaseExecuteRequest> for ReleaseRpc {
+    type Response = ReleaseExecuteResponse;
+    type Future = BoxFuture<Response<ReleaseExecuteResponse>, Status>;
+
+    fn call(&mut self, request: Request<ReleaseExecuteRequest>) -> 
Self::Future {
+        let request = request.into_inner();
+        self.0.releases.lock().unwrap().push(CapturedRelease {
+            request: request.clone(),
+        });
+        Box::pin(async move {
+            Ok(Response::new(ReleaseExecuteResponse {
+                session_id: request.session_id,
+                operation_id: Some(request.operation_id),
+                ..Default::default()
+            }))
+        })
+    }
+}
+
+struct Config(Arc<State>);
+
+impl UnaryService<ConfigRequest> for Config {
+    type Response = ConfigResponse;
+    type Future = BoxFuture<Response<ConfigResponse>, Status>;
+
+    fn call(&mut self, request: Request<ConfigRequest>) -> Self::Future {
+        self.0.record_timeout("Config", request.metadata());
+        // Stalled before its headers, as a connection that never answers.
+        Box::pin(async {
+            std::future::pending::<()>().await;
+            unreachable!("a stalled call never completes")
+        })
+    }
+}


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to