This is an automated email from the ASF dual-hosted git repository.

dongjoon-hyun pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/spark-connect-swift.git


The following commit(s) were added to refs/heads/main by this push:
     new 972101c  [SPARK-60464] Fix completed executions to not fail when the 
final `ReleaseExecute` fails
972101c is described below

commit 972101c622f7658a9a2d8cf3c35adad011fb3eda
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Thu Oct 22 08:05:09 2026 -0700

    [SPARK-60464] Fix completed executions to not fail when the final 
`ReleaseExecute` fails
    
    ### What changes were proposed in this pull request?
    
    This PR aims to send the final `ReleaseExecute(release_all)` of 
reattachable executions in a best-effort way, like the Scala and Python clients.
    
    Previously, it was sent inside the reattach loop right after 
`ResultComplete` arrived, so its failure was handled like a broken response 
stream. Now, it is sent after the loop by the existing `releaseAll(_:)` helper, 
which already ignores failures when an execution is abandoned after an error.
    
    ### Why are the changes needed?
    
    `ReleaseExecute` only asks the server to free the buffered responses, and 
the server removes abandoned executions by itself after 
`spark.connect.execute.manager.detachedTimeout` (5 minutes by default). 
However, its failure failed the whole execution even after all responses were 
received and processed.
    
    - `INVALID_HANDLE.SESSION_NOT_FOUND`, e.g., after the server restarted, was 
reported as `invalidState: OPERATION_NOT_FOUND on the server but responses were 
already received from it.`
    - `INVALID_HANDLE.SESSION_CLOSED`, e.g., after another task released the 
session, was thrown as `sessionClosed`.
    - Only retriable errors like `UNAVAILABLE` succeeded, after a needless 
backoff sleep.
    
    Since `collect()`, `count()`, and the command APIs like 
`DataFrameWriter.save()` use this path, a finished write could be reported as 
failed, and retrying it could repeat its side effects. The Scala and Python 
clients send this request asynchronously and only log its failure.
    
    ### Does this PR introduce _any_ user-facing change?
    
    No for the released versions because the reattachable execution 
(SPARK-58543) is not released yet. The latest release is 
[0.7.0](https://github.com/apache/spark-connect-swift/releases/tag/0.7.0) 
(2026-07-25). In the `main` branch, executions which received all responses no 
longer fail when the final `ReleaseExecute` fails.
    
    ### How was this patch tested?
    
    Pass the CIs with the newly added test case, 
`ReattachableExecuteTests.releaseAllFailureAfterResultComplete`. It releases 
the session with `ReleaseSession` when `ResultComplete` arrives, so that the 
final `ReleaseExecute` fails with `SESSION_NOT_FOUND` (`allow_reconnect=true`, 
like after a server restart) and `SESSION_CLOSED`.
    
    Without this fix, the test case fails like the following.
    
    ```
    Test releaseAllFailureAfterResultComplete() recorded an issue at 
ReattachableExecuteTests.swift:130:4: Caught error: invalidState: 
OPERATION_NOT_FOUND on the server but responses were already received from it.
    ```
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Opus 5.5
    
    Closes #616 from dongjoon-hyun/SPARK-60464.
    
    Authored-by: Dongjoon Hyun <[email protected]>
    Signed-off-by: Dongjoon Hyun <[email protected]>
---
 .../SparkConnectClient+ReattachableExecute.swift   | 10 +++---
 .../ReattachableExecuteTests.swift                 | 36 +++++++++++++++++++++-
 2 files changed, 39 insertions(+), 7 deletions(-)

diff --git a/Sources/SparkConnect/SparkConnectClient+ReattachableExecute.swift 
b/Sources/SparkConnect/SparkConnectClient+ReattachableExecute.swift
index d318f6b..ab136ee 100644
--- a/Sources/SparkConnect/SparkConnectClient+ReattachableExecute.swift
+++ b/Sources/SparkConnect/SparkConnectClient+ReattachableExecute.swift
@@ -110,7 +110,7 @@ extension SparkConnectClient {
   /// broken by a retriable RPC error, or ends without a `ResultComplete` 
message, the stream is
   /// resumed with `ReattachExecute` after the last received response. The 
retriable errors are
   /// governed by ``RetryPolicy/defaultPolicy`` whose backoff is reset 
whenever an attempt
-  /// receives a new response. The server side buffer is released with a 
`release_all`
+  /// receives a new response. The server side buffer is released with a 
best-effort `release_all`
   /// `ReleaseExecute` at the end.
   /// - Parameters:
   ///   - request: An ``ExecutePlanRequest`` to execute.
@@ -148,9 +148,6 @@ extension SparkConnectClient {
           } else {
             try await service.executePlan(initialRequest, onResponse: consume)
           }
-          if state.resultComplete {
-            _ = try await 
service.releaseExecute(Self.getReleaseAllRequest(initialRequest))
-          }
         }
         // When the stream ended gracefully without `ResultComplete`, the 
server has more
         // responses and the loop continues with `ReattachExecute`.
@@ -184,10 +181,11 @@ extension SparkConnectClient {
         try await Task.sleep(for: wait)
       }
     }
+    await releaseAll(initialRequest)
   }
 
-  /// Inform the server to release the operation of `request` after giving up, 
ignoring failures
-  /// because the server releases abandoned executions by itself.
+  /// Inform the server to release the operation of `request` after receiving 
all responses or
+  /// giving up, ignoring failures because the server releases abandoned 
executions by itself.
   private func releaseAll(_ request: ExecutePlanRequest) async {
     try? await withGPRC(retryable: false) { client in
       let service = SparkConnectService.Client(wrapping: client)
diff --git a/Tests/SparkConnectTests/ReattachableExecuteTests.swift 
b/Tests/SparkConnectTests/ReattachableExecuteTests.swift
index 02e857d..60acfd1 100644
--- a/Tests/SparkConnectTests/ReattachableExecuteTests.swift
+++ b/Tests/SparkConnectTests/ReattachableExecuteTests.swift
@@ -26,9 +26,11 @@ import Testing
 
 @testable import SparkConnect
 
-/// A test suite for the reattachable execution request builders
+/// A test suite for the reattachable execution
 @Suite(.serialized)
 struct ReattachableExecuteTests {
+  let TEST_REMOTE = ProcessInfo.processInfo.environment["SPARK_REMOTE"] ?? 
"sc://localhost"
+
   static func executePlanRequest() -> ExecutePlanRequest {
     var request = ExecutePlanRequest()
     request.clientType = "swift"
@@ -108,4 +110,36 @@ struct ReattachableExecuteTests {
     #expect(SparkConnectClient.getReattachExecuteRequest(request, 
nil).operationID == operationID)
     #expect(SparkConnectClient.getReleaseAllRequest(request).operationID == 
operationID)
   }
+
+  /// Release the session of `client` on the server without stopping `client`. 
With
+  /// `allowReconnect`, the server forgets the session like after a restart, 
and the next
+  /// `ReleaseExecute` fails with `SESSION_NOT_FOUND` instead of 
`SESSION_CLOSED`.
+  static func releaseSession(_ client: isolated SparkConnectClient, 
allowReconnect: Bool)
+    async throws
+  {
+    var request = Spark_Connect_ReleaseSessionRequest()
+    request.sessionID = client.sessionID!
+    request.userContext = client.userContext
+    request.clientType = client.clientType
+    request.allowReconnect = allowReconnect
+    try await client.withGPRC { grpc in
+      _ = try await SparkConnectService.Client(wrapping: 
grpc).releaseSession(request)
+    }
+  }
+
+  @Test
+  func releaseAllFailureAfterResultComplete() async throws {
+    for allowReconnect in [true, false] {
+      let client = try SparkConnectClient(remote: TEST_REMOTE)
+      try await client.connect(UUID().uuidString)
+      let request = await client.getExecutePlanRequest(await 
client.getPlanRange(0, 1, 1))
+      // Make the final `release_all` fail after receiving all responses.
+      try await client.executePlanWithReattach(request) { response in
+        if case .resultComplete = response.responseType {
+          try await Self.releaseSession(client, allowReconnect: allowReconnect)
+        }
+      }
+      await client.stop()
+    }
+  }
 }


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to