This is an automated email from the ASF dual-hosted git repository.
dongjoon-hyun pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/spark-connect-swift.git
The following commit(s) were added to refs/heads/main by this push:
new 972101c [SPARK-60464] Fix completed executions to not fail when the
final `ReleaseExecute` fails
972101c is described below
commit 972101c622f7658a9a2d8cf3c35adad011fb3eda
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Thu Oct 22 08:05:09 2026 -0700
[SPARK-60464] Fix completed executions to not fail when the final
`ReleaseExecute` fails
### What changes were proposed in this pull request?
This PR aims to send the final `ReleaseExecute(release_all)` of
reattachable executions in a best-effort way, like the Scala and Python clients.
Previously, it was sent inside the reattach loop right after
`ResultComplete` arrived, so its failure was handled like a broken response
stream. Now, it is sent after the loop by the existing `releaseAll(_:)` helper,
which already ignores failures when an execution is abandoned after an error.
### Why are the changes needed?
`ReleaseExecute` only asks the server to free the buffered responses, and
the server removes abandoned executions by itself after
`spark.connect.execute.manager.detachedTimeout` (5 minutes by default).
However, its failure failed the whole execution even after all responses were
received and processed.
- `INVALID_HANDLE.SESSION_NOT_FOUND`, e.g., after the server restarted, was
reported as `invalidState: OPERATION_NOT_FOUND on the server but responses were
already received from it.`
- `INVALID_HANDLE.SESSION_CLOSED`, e.g., after another task released the
session, was thrown as `sessionClosed`.
- Only retriable errors like `UNAVAILABLE` succeeded, after a needless
backoff sleep.
Since `collect()`, `count()`, and the command APIs like
`DataFrameWriter.save()` use this path, a finished write could be reported as
failed, and retrying it could repeat its side effects. The Scala and Python
clients send this request asynchronously and only log its failure.
### Does this PR introduce _any_ user-facing change?
No for the released versions because the reattachable execution
(SPARK-58543) is not released yet. The latest release is
[0.7.0](https://github.com/apache/spark-connect-swift/releases/tag/0.7.0)
(2026-07-25). In the `main` branch, executions which received all responses no
longer fail when the final `ReleaseExecute` fails.
### How was this patch tested?
Pass the CIs with the newly added test case,
`ReattachableExecuteTests.releaseAllFailureAfterResultComplete`. It releases
the session with `ReleaseSession` when `ResultComplete` arrives, so that the
final `ReleaseExecute` fails with `SESSION_NOT_FOUND` (`allow_reconnect=true`,
like after a server restart) and `SESSION_CLOSED`.
Without this fix, the test case fails like the following.
```
Test releaseAllFailureAfterResultComplete() recorded an issue at
ReattachableExecuteTests.swift:130:4: Caught error: invalidState:
OPERATION_NOT_FOUND on the server but responses were already received from it.
```
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
Closes #616 from dongjoon-hyun/SPARK-60464.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
---
.../SparkConnectClient+ReattachableExecute.swift | 10 +++---
.../ReattachableExecuteTests.swift | 36 +++++++++++++++++++++-
2 files changed, 39 insertions(+), 7 deletions(-)
diff --git a/Sources/SparkConnect/SparkConnectClient+ReattachableExecute.swift
b/Sources/SparkConnect/SparkConnectClient+ReattachableExecute.swift
index d318f6b..ab136ee 100644
--- a/Sources/SparkConnect/SparkConnectClient+ReattachableExecute.swift
+++ b/Sources/SparkConnect/SparkConnectClient+ReattachableExecute.swift
@@ -110,7 +110,7 @@ extension SparkConnectClient {
/// broken by a retriable RPC error, or ends without a `ResultComplete`
message, the stream is
/// resumed with `ReattachExecute` after the last received response. The
retriable errors are
/// governed by ``RetryPolicy/defaultPolicy`` whose backoff is reset
whenever an attempt
- /// receives a new response. The server side buffer is released with a
`release_all`
+ /// receives a new response. The server side buffer is released with a
best-effort `release_all`
/// `ReleaseExecute` at the end.
/// - Parameters:
/// - request: An ``ExecutePlanRequest`` to execute.
@@ -148,9 +148,6 @@ extension SparkConnectClient {
} else {
try await service.executePlan(initialRequest, onResponse: consume)
}
- if state.resultComplete {
- _ = try await
service.releaseExecute(Self.getReleaseAllRequest(initialRequest))
- }
}
// When the stream ended gracefully without `ResultComplete`, the
server has more
// responses and the loop continues with `ReattachExecute`.
@@ -184,10 +181,11 @@ extension SparkConnectClient {
try await Task.sleep(for: wait)
}
}
+ await releaseAll(initialRequest)
}
- /// Inform the server to release the operation of `request` after giving up,
ignoring failures
- /// because the server releases abandoned executions by itself.
+ /// Inform the server to release the operation of `request` after receiving
all responses or
+ /// giving up, ignoring failures because the server releases abandoned
executions by itself.
private func releaseAll(_ request: ExecutePlanRequest) async {
try? await withGPRC(retryable: false) { client in
let service = SparkConnectService.Client(wrapping: client)
diff --git a/Tests/SparkConnectTests/ReattachableExecuteTests.swift
b/Tests/SparkConnectTests/ReattachableExecuteTests.swift
index 02e857d..60acfd1 100644
--- a/Tests/SparkConnectTests/ReattachableExecuteTests.swift
+++ b/Tests/SparkConnectTests/ReattachableExecuteTests.swift
@@ -26,9 +26,11 @@ import Testing
@testable import SparkConnect
-/// A test suite for the reattachable execution request builders
+/// A test suite for the reattachable execution
@Suite(.serialized)
struct ReattachableExecuteTests {
+ let TEST_REMOTE = ProcessInfo.processInfo.environment["SPARK_REMOTE"] ??
"sc://localhost"
+
static func executePlanRequest() -> ExecutePlanRequest {
var request = ExecutePlanRequest()
request.clientType = "swift"
@@ -108,4 +110,36 @@ struct ReattachableExecuteTests {
#expect(SparkConnectClient.getReattachExecuteRequest(request,
nil).operationID == operationID)
#expect(SparkConnectClient.getReleaseAllRequest(request).operationID ==
operationID)
}
+
+ /// Release the session of `client` on the server without stopping `client`.
With
+ /// `allowReconnect`, the server forgets the session like after a restart,
and the next
+ /// `ReleaseExecute` fails with `SESSION_NOT_FOUND` instead of
`SESSION_CLOSED`.
+ static func releaseSession(_ client: isolated SparkConnectClient,
allowReconnect: Bool)
+ async throws
+ {
+ var request = Spark_Connect_ReleaseSessionRequest()
+ request.sessionID = client.sessionID!
+ request.userContext = client.userContext
+ request.clientType = client.clientType
+ request.allowReconnect = allowReconnect
+ try await client.withGPRC { grpc in
+ _ = try await SparkConnectService.Client(wrapping:
grpc).releaseSession(request)
+ }
+ }
+
+ @Test
+ func releaseAllFailureAfterResultComplete() async throws {
+ for allowReconnect in [true, false] {
+ let client = try SparkConnectClient(remote: TEST_REMOTE)
+ try await client.connect(UUID().uuidString)
+ let request = await client.getExecutePlanRequest(await
client.getPlanRange(0, 1, 1))
+ // Make the final `release_all` fail after receiving all responses.
+ try await client.executePlanWithReattach(request) { response in
+ if case .resultComplete = response.responseType {
+ try await Self.releaseSession(client, allowReconnect: allowReconnect)
+ }
+ }
+ await client.stop()
+ }
+ }
}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]