dongjoon-hyun opened a new pull request, #613:
URL: https://github.com/apache/spark-connect-swift/pull/613
### What changes were proposed in this pull request?
This PR aims to send the server-side session ID observed by the client as
`client_observed_server_side_session_id` in requests, like the Scala and Python
clients. With it, the server rejects requests with
`INVALID_HANDLE.SESSION_CHANGED` when the server-side session has changed,
e.g., after the server restarted.
- Add `SparkConnectClient.serverSideSessionID`, which `connect(_:)` records
from the `server_side_session_id` of its `AnalyzePlan` response.
- Set `client_observed_server_side_session_id` in all requests which have
the field: `ExecutePlan`, `AnalyzePlan`, `Config`, `Interrupt`, `GetStatus`,
`CloneSession`, `AddArtifacts`, and `ArtifactStatuses`. `ReattachExecute` and
`ReleaseExecute` copy it from the initial `ExecutePlan` request.
- Remove the unused `SparkSession.serverSideSessionID`.
`FetchErrorDetails` is not changed, like the Scala client, because it only
fetches the details of an error which the server already returned.
### Why are the changes needed?
When the Spark Connect server restarts, e.g., due to a driver failure, the
server-side session is lost. Since the Swift client sends only `session_id`,
the server silently creates a new empty session for it. The existing
`SparkSession` keeps working without the session state like SQL configurations,
temporary views, registered functions, and artifacts.
```swift
let spark = try await SparkSession.builder.getOrCreate()
try await spark.conf.set("spark.sql.session.timeZone", "Asia/Seoul")
try await spark.range(3).createTempView("v")
// The server restarts here.
try await spark.conf.get("spark.sql.session.timeZone")
try await spark.table("v").count()
try await spark.range(3).count()
```
- Before: `Etc/UTC` (the server default), `TABLE_OR_VIEW_NOT_FOUND`, and `3`.
- After: All of them fail with `[INVALID_HANDLE.SESSION_CHANGED] The handle
... is invalid. The existing Spark server driver instance has restarted. Please
reconnect.`
The Scala and Python clients have sent this field since Apache Spark
[v4.0.0](https://github.com/apache/spark/releases/tag/v4.0.0) (2025-05-19) via
SPARK-47380, so all servers supported by this client validate it. The server
checks it in
[SparkConnectSessionManager](https://github.com/apache/spark/blob/v4.2.0/sql/connect/server/src/main/scala/org/apache/spark/sql/connect/service/SparkConnectSessionManager.scala#L78-L106)
of `v4.2.0` (2026-07-11) before running a request, so a request is never
executed in the new session.
### Does this PR introduce _any_ user-facing change?
Yes. After the server-side session changes, e.g., due to a server restart,
the requests of the existing `SparkSession` fail with
`INVALID_HANDLE.SESSION_CHANGED` instead of silently running in a new empty
session. Users need to create a new `SparkSession` to continue, like the Scala
and Python clients. There is no change while the server-side session stays the
same.
### How was this patch tested?
Pass the CIs with the newly added test cases.
- `SparkConnectClientTests.clientObservedServerSideSessionID` and
`ReattachableExecuteTests.clientObservedServerSideSessionID` check the request
builders. They don't need a server.
- `SparkConnectClientTests.serverSideSessionID` checks that `connect(_:)`
records the server-side session ID.
- `SparkConnectClientTests.serverSideSessionChanged` changes the observed
server-side session ID to simulate a server restart. It checks that the server
rejects every kind of request with `INVALID_HANDLE.SESSION_CHANGED` without
running it.
I also checked the above example against `apache/spark:4.2.0` by restarting
the server container before and after this PR.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
