This is an automated email from the ASF dual-hosted git repository.
dongjoon-hyun pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/spark-connect-swift.git
The following commit(s) were added to refs/heads/main by this push:
new 2384f38 [SPARK-60434] Send `client_observed_server_side_session_id`
to detect server-side session changes
2384f38 is described below
commit 2384f38764a4bfcfa4a640c46252eb4efa58ac6d
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Wed Oct 21 11:42:21 2026 -0700
[SPARK-60434] Send `client_observed_server_side_session_id` to detect
server-side session changes
### What changes were proposed in this pull request?
This PR aims to send the server-side session ID observed by the client as
`client_observed_server_side_session_id` in requests, like the Scala and Python
clients. With it, the server rejects requests with
`INVALID_HANDLE.SESSION_CHANGED` when the server-side session has changed,
e.g., after the server restarted.
- Add `SparkConnectClient.serverSideSessionID`, which `connect(_:)` records
from the `server_side_session_id` of its `AnalyzePlan` response.
- Set `client_observed_server_side_session_id` in all requests which have
the field: `ExecutePlan`, `AnalyzePlan`, `Config`, `Interrupt`, `GetStatus`,
`CloneSession`, `AddArtifacts`, and `ArtifactStatuses`. `ReattachExecute` and
`ReleaseExecute` copy it from the initial `ExecutePlan` request.
- Remove the unused `SparkSession.serverSideSessionID`.
`FetchErrorDetails` is not changed, like the Scala client, because it only
fetches the details of an error which the server already returned.
### Why are the changes needed?
When the Spark Connect server restarts, e.g., due to a driver failure, the
server-side session is lost. Since the Swift client sends only `session_id`,
the server silently creates a new empty session for it. The existing
`SparkSession` keeps working without the session state like SQL configurations,
temporary views, registered functions, and artifacts.
```swift
let spark = try await SparkSession.builder.getOrCreate()
try await spark.conf.set("spark.sql.session.timeZone", "Asia/Seoul")
try await spark.range(3).createTempView("v")
// The server restarts here.
try await spark.conf.get("spark.sql.session.timeZone")
try await spark.table("v").count()
try await spark.range(3).count()
```
- Before: `Etc/UTC` (the server default), `TABLE_OR_VIEW_NOT_FOUND`, and
`3`.
- After: All of them fail with `[INVALID_HANDLE.SESSION_CHANGED] The handle
... is invalid. The existing Spark server driver instance has restarted. Please
reconnect.`
The Scala and Python clients have sent this field since Apache Spark
[v4.0.0](https://github.com/apache/spark/releases/tag/v4.0.0) (2025-05-19) via
SPARK-47380, so all servers supported by this client validate it. The server
checks it in
[SparkConnectSessionManager](https://github.com/apache/spark/blob/v4.2.0/sql/connect/server/src/main/scala/org/apache/spark/sql/connect/service/SparkConnectSessionManager.scala#L78-L106)
of `v4.2.0` (2026-07-11) before running a request, so a request [...]
### Does this PR introduce _any_ user-facing change?
Yes. After the server-side session changes, e.g., due to a server restart,
the requests of the existing `SparkSession` fail with
`INVALID_HANDLE.SESSION_CHANGED` instead of silently running in a new empty
session. Users need to create a new `SparkSession` to continue, like the Scala
and Python clients. There is no change while the server-side session stays the
same.
### How was this patch tested?
Pass the CIs with the newly added test cases.
- `SparkConnectClientTests.clientObservedServerSideSessionID` and
`ReattachableExecuteTests.clientObservedServerSideSessionID` check the request
builders. They don't need a server.
- `SparkConnectClientTests.serverSideSessionID` checks that `connect(_:)`
records the server-side session ID.
- `SparkConnectClientTests.serverSideSessionChanged` changes the observed
server-side session ID to simulate a server restart. It checks that the server
rejects every kind of request with `INVALID_HANDLE.SESSION_CHANGED` without
running it.
I also checked the above example against `apache/spark:4.2.0` by restarting
the server container before and after this PR.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
Closes #613 from dongjoon-hyun/SPARK-60434.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
---
.../SparkConnect/SparkConnectClient+Artifact.swift | 12 +++
.../SparkConnectClient+ReattachableExecute.swift | 6 ++
Sources/SparkConnect/SparkConnectClient.swift | 50 ++++++++++++
Sources/SparkConnect/SparkSession.swift | 3 -
.../ReattachableExecuteTests.swift | 14 ++++
.../SparkConnectClientTests.swift | 90 ++++++++++++++++++++++
6 files changed, 172 insertions(+), 3 deletions(-)
diff --git a/Sources/SparkConnect/SparkConnectClient+Artifact.swift
b/Sources/SparkConnect/SparkConnectClient+Artifact.swift
index 92e114f..c8c2294 100644
--- a/Sources/SparkConnect/SparkConnectClient+Artifact.swift
+++ b/Sources/SparkConnect/SparkConnectClient+Artifact.swift
@@ -56,6 +56,9 @@ extension SparkConnectClient {
addArtifactsRequest.sessionID = self.sessionID!
addArtifactsRequest.userContext = self.userContext
addArtifactsRequest.clientType = self.clientType
+ if let serverSideSessionID {
+ addArtifactsRequest.clientObservedServerSideSessionID =
serverSideSessionID
+ }
addArtifactsRequest.batch = batch
let request = addArtifactsRequest
_ = try await service.addArtifacts(
@@ -75,6 +78,9 @@ extension SparkConnectClient {
request.sessionID = self.sessionID!
request.userContext = self.userContext
request.clientType = self.clientType
+ if let serverSideSessionID {
+ request.clientObservedServerSideSessionID = serverSideSessionID
+ }
request.names = [name]
let response = try await service.artifactStatus(request)
return response.statuses[name]?.exists ?? false
@@ -112,6 +118,9 @@ extension SparkConnectClient {
request.sessionID = self.sessionID!
request.userContext = self.userContext
request.clientType = self.clientType
+ if let serverSideSessionID {
+ request.clientObservedServerSideSessionID = serverSideSessionID
+ }
request.batch = batch
requests.append(request)
} else {
@@ -126,6 +135,9 @@ extension SparkConnectClient {
request.sessionID = self.sessionID!
request.userContext = self.userContext
request.clientType = self.clientType
+ if let serverSideSessionID {
+ request.clientObservedServerSideSessionID = serverSideSessionID
+ }
if i == 0 {
var beginChunk =
Spark_Connect_AddArtifactsRequest.BeginChunkedArtifact()
beginChunk.name = name
diff --git a/Sources/SparkConnect/SparkConnectClient+ReattachableExecute.swift
b/Sources/SparkConnect/SparkConnectClient+ReattachableExecute.swift
index c1b3817..d318f6b 100644
--- a/Sources/SparkConnect/SparkConnectClient+ReattachableExecute.swift
+++ b/Sources/SparkConnect/SparkConnectClient+ReattachableExecute.swift
@@ -80,6 +80,9 @@ extension SparkConnectClient {
reattach.userContext = request.userContext
reattach.operationID = request.operationID
reattach.clientType = request.clientType
+ if request.hasClientObservedServerSideSessionID {
+ reattach.clientObservedServerSideSessionID =
request.clientObservedServerSideSessionID
+ }
if let lastResponseID {
reattach.lastResponseID = lastResponseID
}
@@ -93,6 +96,9 @@ extension SparkConnectClient {
release.userContext = request.userContext
release.operationID = request.operationID
release.clientType = request.clientType
+ if request.hasClientObservedServerSideSessionID {
+ release.clientObservedServerSideSessionID =
request.clientObservedServerSideSessionID
+ }
release.releaseAll = ReleaseExecuteRequest.ReleaseAll()
return release
}
diff --git a/Sources/SparkConnect/SparkConnectClient.swift
b/Sources/SparkConnect/SparkConnectClient.swift
index 050f449..8dfd1e8 100644
--- a/Sources/SparkConnect/SparkConnectClient.swift
+++ b/Sources/SparkConnect/SparkConnectClient.swift
@@ -42,6 +42,11 @@ public actor SparkConnectClient {
let serviceConfig: ServiceConfig
var intercepters: [ClientInterceptor] = []
let userContext: UserContext
+ /// The server-side session ID observed by ``connect(_:)``. It is sent back
as
+ /// `client_observed_server_side_session_id` in requests in order to make
the server reject them
+ /// with `INVALID_HANDLE.SESSION_CHANGED` when the server-side session
changed, e.g., after the
+ /// server restarted, instead of silently running them in a new session.
+ var serverSideSessionID: String? = nil
var sessionID: String? = nil
var tags = Set<String>()
@@ -169,6 +174,9 @@ public actor SparkConnectClient {
return OneOf_Analyze.sparkVersion(AnalyzePlanRequest.SparkVersion())
})
let response = try await service.analyzePlan(request)
+ if !response.serverSideSessionID.isEmpty {
+ self.serverSideSessionID = response.serverSideSessionID
+ }
return response
}
}
@@ -238,6 +246,9 @@ public actor SparkConnectClient {
request.clientType = clientType
request.userContext = userContext
request.sessionID = self.sessionID!
+ if let serverSideSessionID {
+ request.clientObservedServerSideSessionID = serverSideSessionID
+ }
_ = try await service.config(request)
return true
}
@@ -266,6 +277,9 @@ public actor SparkConnectClient {
request.clientType = clientType
request.userContext = userContext
request.sessionID = self.sessionID!
+ if let serverSideSessionID {
+ request.clientObservedServerSideSessionID = serverSideSessionID
+ }
let _ = try await service.config(request)
return true
}
@@ -293,6 +307,9 @@ public actor SparkConnectClient {
request.clientType = clientType
request.userContext = userContext
request.sessionID = self.sessionID!
+ if let serverSideSessionID {
+ request.clientObservedServerSideSessionID = serverSideSessionID
+ }
let response = try await service.config(request)
return try Self.getConfValue(key, from: response)
}
@@ -335,6 +352,9 @@ public actor SparkConnectClient {
request.clientType = clientType
request.userContext = userContext
request.sessionID = self.sessionID!
+ if let serverSideSessionID {
+ request.clientObservedServerSideSessionID = serverSideSessionID
+ }
let response = try await service.config(request)
let result =
if let pair = response.pairs.first, pair.hasValue {
@@ -368,6 +388,9 @@ public actor SparkConnectClient {
request.clientType = clientType
request.userContext = userContext
request.sessionID = self.sessionID!
+ if let serverSideSessionID {
+ request.clientObservedServerSideSessionID = serverSideSessionID
+ }
let response = try await service.config(request)
let result: String? =
if let pair = response.pairs.first, pair.hasValue {
@@ -398,6 +421,9 @@ public actor SparkConnectClient {
request.clientType = clientType
request.userContext = userContext
request.sessionID = self.sessionID!
+ if let serverSideSessionID {
+ request.clientObservedServerSideSessionID = serverSideSessionID
+ }
let response = try await service.config(request)
var map = [String: String]()
for pair in response.pairs {
@@ -431,6 +457,9 @@ public actor SparkConnectClient {
request.clientType = clientType
request.userContext = userContext
request.sessionID = self.sessionID!
+ if let serverSideSessionID {
+ request.clientObservedServerSideSessionID = serverSideSessionID
+ }
let response = try await service.config(request)
return response.pairs.first?.value == "true"
}
@@ -500,6 +529,9 @@ public actor SparkConnectClient {
request.clientType = clientType
request.userContext = userContext
request.sessionID = self.sessionID!
+ if let serverSideSessionID {
+ request.clientObservedServerSideSessionID = serverSideSessionID
+ }
// The server normalizes the operation ID of `ExecutePlan` to a lowercase
UUID string, but
// looks up the operation ID of `ReattachExecute` and `ReleaseExecute` as
it is.
request.operationID = UUID().uuidString.lowercased()
@@ -535,6 +567,9 @@ public actor SparkConnectClient {
request.clientType = clientType
request.userContext = userContext
request.sessionID = self.sessionID!
+ if let serverSideSessionID {
+ request.clientObservedServerSideSessionID = serverSideSessionID
+ }
request.analyze = f()
return request
}
@@ -1026,6 +1061,9 @@ public actor SparkConnectClient {
request.sessionID = self.sessionID!
request.userContext = self.userContext
request.clientType = self.clientType
+ if let serverSideSessionID {
+ request.clientObservedServerSideSessionID = serverSideSessionID
+ }
request.interruptType = .all
return try await withGPRC { client in
@@ -1040,6 +1078,9 @@ public actor SparkConnectClient {
request.sessionID = self.sessionID!
request.userContext = self.userContext
request.clientType = self.clientType
+ if let serverSideSessionID {
+ request.clientObservedServerSideSessionID = serverSideSessionID
+ }
request.interruptType = .tag
request.operationTag = tag
@@ -1055,6 +1096,9 @@ public actor SparkConnectClient {
request.sessionID = self.sessionID!
request.userContext = self.userContext
request.clientType = self.clientType
+ if let serverSideSessionID {
+ request.clientObservedServerSideSessionID = serverSideSessionID
+ }
request.interruptType = .operationID
request.operationID = operationId
@@ -1072,6 +1116,9 @@ public actor SparkConnectClient {
request.sessionID = self.sessionID!
request.userContext = self.userContext
request.clientType = self.clientType
+ if let serverSideSessionID {
+ request.clientObservedServerSideSessionID = serverSideSessionID
+ }
if !operationIds.isEmpty {
var operationStatusRequest =
Spark_Connect_GetStatusRequest.OperationStatusRequest()
operationStatusRequest.operationIds = operationIds
@@ -1094,6 +1141,9 @@ public actor SparkConnectClient {
request.sessionID = self.sessionID!
request.userContext = self.userContext
request.clientType = self.clientType
+ if let serverSideSessionID {
+ request.clientObservedServerSideSessionID = serverSideSessionID
+ }
if let newSessionID {
request.newSessionID = newSessionID
}
diff --git a/Sources/SparkConnect/SparkSession.swift
b/Sources/SparkConnect/SparkSession.swift
index c62dc26..afcc4b4 100644
--- a/Sources/SparkConnect/SparkSession.swift
+++ b/Sources/SparkConnect/SparkSession.swift
@@ -108,9 +108,6 @@ public actor SparkSession {
sessionID
}
- /// A server-side generated session ID. This is supposed to be overwritten
during establishing connections.
- var serverSideSessionID: String = ""
-
/// A variable for ``SparkContext``. This is designed to throw exceptions by
Apache Spark.
public var sparkContext: SparkContext {
get throws {
diff --git a/Tests/SparkConnectTests/ReattachableExecuteTests.swift
b/Tests/SparkConnectTests/ReattachableExecuteTests.swift
index 04f8af8..02e857d 100644
--- a/Tests/SparkConnectTests/ReattachableExecuteTests.swift
+++ b/Tests/SparkConnectTests/ReattachableExecuteTests.swift
@@ -75,6 +75,20 @@ struct ReattachableExecuteTests {
#expect(request.release == .releaseAll(ReleaseExecuteRequest.ReleaseAll()))
}
+ @Test
+ func clientObservedServerSideSessionID() {
+ var request = Self.executePlanRequest()
+ let reattachWithoutID =
SparkConnectClient.getReattachExecuteRequest(request, nil)
+ #expect(!reattachWithoutID.hasClientObservedServerSideSessionID)
+
#expect(!SparkConnectClient.getReleaseAllRequest(request).hasClientObservedServerSideSessionID)
+
+ request.clientObservedServerSideSessionID = "server-side-session-id"
+ let reattach = SparkConnectClient.getReattachExecuteRequest(request, nil)
+ #expect(reattach.clientObservedServerSideSessionID ==
"server-side-session-id")
+ let release = SparkConnectClient.getReleaseAllRequest(request)
+ #expect(release.clientObservedServerSideSessionID ==
"server-side-session-id")
+ }
+
/// Create an ``ExecutePlanRequest`` with `client` without connecting to the
server.
static func executePlanRequest(_ client: isolated SparkConnectClient) async
-> ExecutePlanRequest
diff --git a/Tests/SparkConnectTests/SparkConnectClientTests.swift
b/Tests/SparkConnectTests/SparkConnectClientTests.swift
index 485007b..89e6939 100644
--- a/Tests/SparkConnectTests/SparkConnectClientTests.swift
+++ b/Tests/SparkConnectTests/SparkConnectClientTests.swift
@@ -132,6 +132,96 @@ struct SparkConnectClientTests {
await client.stop()
}
+ /// Set the session IDs of `client` without connecting to the server.
+ static func setSessionIDs(
+ _ client: isolated SparkConnectClient, _ sessionID: String?, _
serverSideSessionID: String?
+ ) {
+ client.sessionID = sessionID
+ client.serverSideSessionID = serverSideSessionID
+ }
+
+ @Test
+ func clientObservedServerSideSessionID() async throws {
+ let client = try SparkConnectClient(remote: "sc://localhost")
+ let sessionID = UUID().uuidString.lowercased()
+ let plan = await client.getPlanRange(0, 1, 1)
+
+ await Self.setSessionIDs(client, sessionID, nil)
+ #expect(await
!client.getExecutePlanRequest(plan).hasClientObservedServerSideSessionID)
+ let analyzePlanRequest = await client.getAnalyzePlanRequest(sessionID,
plan)
+ #expect(!analyzePlanRequest.hasClientObservedServerSideSessionID)
+
+ let serverSideSessionID = UUID().uuidString.lowercased()
+ await Self.setSessionIDs(client, sessionID, serverSideSessionID)
+ let executePlanRequestWithID = await client.getExecutePlanRequest(plan)
+ #expect(executePlanRequestWithID.clientObservedServerSideSessionID ==
serverSideSessionID)
+ let analyzePlanRequestWithID = await
client.getAnalyzePlanRequest(sessionID, plan)
+ #expect(analyzePlanRequestWithID.clientObservedServerSideSessionID ==
serverSideSessionID)
+ }
+
+ @Test
+ func serverSideSessionID() async throws {
+ let client = try SparkConnectClient(remote: TEST_REMOTE)
+ #expect(await client.serverSideSessionID == nil)
+ let response = try await client.connect(UUID().uuidString)
+ #expect(!response.serverSideSessionID.isEmpty)
+ #expect(await client.serverSideSessionID == response.serverSideSessionID)
+ await client.stop()
+ }
+
+ @Test
+ func serverSideSessionChanged() async throws {
+ let client = try SparkConnectClient(remote: TEST_REMOTE)
+ let response = try await client.connect(UUID().uuidString)
+ let version = response.sparkVersion.version
+ let sessionID = try #require(await client.sessionID)
+ let serverSideSessionID = try #require(await client.serverSideSessionID)
+ let key = "spark.test.serverSideSessionChanged"
+ let path = "server-side-session-changed.jar"
+ #expect(FileManager().createFile(atPath: path, contents: "abc".data(using:
.utf8)))
+ defer { try? FileManager().removeItem(atPath: path) }
+
+ // Simulate that the server-side session changed, e.g., after the server
restarted.
+ await Self.setSessionIDs(client, sessionID, UUID().uuidString.lowercased())
+ var calls: [() async throws -> Void] = [
+ { _ = try await client.ddlParse("a int") },
+ {
+ try await client.executePlanWithReattach(
+ client.getExecutePlanRequest(client.getPlanRange(0, 1, 1))
+ ) { _ in }
+ },
+ { _ = try await client.setConf(map: [key: "true"]) },
+ { _ = try await client.unsetConf(keys: [key]) },
+ { _ = try await client.getConf(key) },
+ { _ = try await client.getConfWithDefault(key, "false") },
+ { _ = try await client.getConfOption(key) },
+ { _ = try await client.getConfAll() },
+ { _ = try await client.isModifiable(key) },
+ { _ = try await client.interruptAll() },
+ { _ = try await client.interruptTag("tag") },
+ { _ = try await
client.interruptOperation(UUID().uuidString.lowercased()) },
+ { _ = try await client.artifactExists("cache/nonexistent") },
+ { try await client.addArtifact(URL(fileURLWithPath: path)) },
+ ]
+ if isSparkVersionAtLeast(version, "4.1") {
+ calls.append { _ = try await client.cloneSession() }
+ }
+ if isSparkVersionAtLeast(version, "4.2") {
+ calls.append { _ = try await client.getOperationStatuses() }
+ }
+ for call in calls {
+ let error = try await #require(throws: Error.self) {
+ try await call()
+ }
+ #expect("\(error)".contains("INVALID_HANDLE.SESSION_CHANGED"))
+ }
+
+ // The server rejected the requests without running them in the session.
+ await Self.setSessionIDs(client, sessionID, serverSideSessionID)
+ #expect(try await client.getConfOption(key) == nil)
+ await client.stop()
+ }
+
@Test
func resultChunking() async throws {
let client = try SparkConnectClient(remote: TEST_REMOTE)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]