This is an automated email from the ASF dual-hosted git repository.

dongjoon-hyun pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/spark-connect-swift.git


The following commit(s) were added to refs/heads/main by this push:
     new 2384f38  [SPARK-60434] Send `client_observed_server_side_session_id` 
to detect server-side session changes
2384f38 is described below

commit 2384f38764a4bfcfa4a640c46252eb4efa58ac6d
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Wed Oct 21 11:42:21 2026 -0700

    [SPARK-60434] Send `client_observed_server_side_session_id` to detect 
server-side session changes
    
    ### What changes were proposed in this pull request?
    
    This PR aims to send the server-side session ID observed by the client as 
`client_observed_server_side_session_id` in requests, like the Scala and Python 
clients. With it, the server rejects requests with 
`INVALID_HANDLE.SESSION_CHANGED` when the server-side session has changed, 
e.g., after the server restarted.
    
    - Add `SparkConnectClient.serverSideSessionID`, which `connect(_:)` records 
from the `server_side_session_id` of its `AnalyzePlan` response.
    - Set `client_observed_server_side_session_id` in all requests which have 
the field: `ExecutePlan`, `AnalyzePlan`, `Config`, `Interrupt`, `GetStatus`, 
`CloneSession`, `AddArtifacts`, and `ArtifactStatuses`. `ReattachExecute` and 
`ReleaseExecute` copy it from the initial `ExecutePlan` request.
    - Remove the unused `SparkSession.serverSideSessionID`.
    
    `FetchErrorDetails` is not changed, like the Scala client, because it only 
fetches the details of an error which the server already returned.
    
    ### Why are the changes needed?
    
    When the Spark Connect server restarts, e.g., due to a driver failure, the 
server-side session is lost. Since the Swift client sends only `session_id`, 
the server silently creates a new empty session for it. The existing 
`SparkSession` keeps working without the session state like SQL configurations, 
temporary views, registered functions, and artifacts.
    
    ```swift
    let spark = try await SparkSession.builder.getOrCreate()
    try await spark.conf.set("spark.sql.session.timeZone", "Asia/Seoul")
    try await spark.range(3).createTempView("v")
    // The server restarts here.
    try await spark.conf.get("spark.sql.session.timeZone")
    try await spark.table("v").count()
    try await spark.range(3).count()
    ```
    
    - Before: `Etc/UTC` (the server default), `TABLE_OR_VIEW_NOT_FOUND`, and 
`3`.
    - After: All of them fail with `[INVALID_HANDLE.SESSION_CHANGED] The handle 
... is invalid. The existing Spark server driver instance has restarted. Please 
reconnect.`
    
    The Scala and Python clients have sent this field since Apache Spark 
[v4.0.0](https://github.com/apache/spark/releases/tag/v4.0.0) (2025-05-19) via 
SPARK-47380, so all servers supported by this client validate it. The server 
checks it in 
[SparkConnectSessionManager](https://github.com/apache/spark/blob/v4.2.0/sql/connect/server/src/main/scala/org/apache/spark/sql/connect/service/SparkConnectSessionManager.scala#L78-L106)
 of `v4.2.0` (2026-07-11) before running a request, so a request  [...]
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes. After the server-side session changes, e.g., due to a server restart, 
the requests of the existing `SparkSession` fail with 
`INVALID_HANDLE.SESSION_CHANGED` instead of silently running in a new empty 
session. Users need to create a new `SparkSession` to continue, like the Scala 
and Python clients. There is no change while the server-side session stays the 
same.
    
    ### How was this patch tested?
    
    Pass the CIs with the newly added test cases.
    
    - `SparkConnectClientTests.clientObservedServerSideSessionID` and 
`ReattachableExecuteTests.clientObservedServerSideSessionID` check the request 
builders. They don't need a server.
    - `SparkConnectClientTests.serverSideSessionID` checks that `connect(_:)` 
records the server-side session ID.
    - `SparkConnectClientTests.serverSideSessionChanged` changes the observed 
server-side session ID to simulate a server restart. It checks that the server 
rejects every kind of request with `INVALID_HANDLE.SESSION_CHANGED` without 
running it.
    
    I also checked the above example against `apache/spark:4.2.0` by restarting 
the server container before and after this PR.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Opus 5.5
    
    Closes #613 from dongjoon-hyun/SPARK-60434.
    
    Authored-by: Dongjoon Hyun <[email protected]>
    Signed-off-by: Dongjoon Hyun <[email protected]>
---
 .../SparkConnect/SparkConnectClient+Artifact.swift | 12 +++
 .../SparkConnectClient+ReattachableExecute.swift   |  6 ++
 Sources/SparkConnect/SparkConnectClient.swift      | 50 ++++++++++++
 Sources/SparkConnect/SparkSession.swift            |  3 -
 .../ReattachableExecuteTests.swift                 | 14 ++++
 .../SparkConnectClientTests.swift                  | 90 ++++++++++++++++++++++
 6 files changed, 172 insertions(+), 3 deletions(-)

diff --git a/Sources/SparkConnect/SparkConnectClient+Artifact.swift 
b/Sources/SparkConnect/SparkConnectClient+Artifact.swift
index 92e114f..c8c2294 100644
--- a/Sources/SparkConnect/SparkConnectClient+Artifact.swift
+++ b/Sources/SparkConnect/SparkConnectClient+Artifact.swift
@@ -56,6 +56,9 @@ extension SparkConnectClient {
       addArtifactsRequest.sessionID = self.sessionID!
       addArtifactsRequest.userContext = self.userContext
       addArtifactsRequest.clientType = self.clientType
+      if let serverSideSessionID {
+        addArtifactsRequest.clientObservedServerSideSessionID = 
serverSideSessionID
+      }
       addArtifactsRequest.batch = batch
       let request = addArtifactsRequest
       _ = try await service.addArtifacts(
@@ -75,6 +78,9 @@ extension SparkConnectClient {
       request.sessionID = self.sessionID!
       request.userContext = self.userContext
       request.clientType = self.clientType
+      if let serverSideSessionID {
+        request.clientObservedServerSideSessionID = serverSideSessionID
+      }
       request.names = [name]
       let response = try await service.artifactStatus(request)
       return response.statuses[name]?.exists ?? false
@@ -112,6 +118,9 @@ extension SparkConnectClient {
         request.sessionID = self.sessionID!
         request.userContext = self.userContext
         request.clientType = self.clientType
+        if let serverSideSessionID {
+          request.clientObservedServerSideSessionID = serverSideSessionID
+        }
         request.batch = batch
         requests.append(request)
       } else {
@@ -126,6 +135,9 @@ extension SparkConnectClient {
           request.sessionID = self.sessionID!
           request.userContext = self.userContext
           request.clientType = self.clientType
+          if let serverSideSessionID {
+            request.clientObservedServerSideSessionID = serverSideSessionID
+          }
           if i == 0 {
             var beginChunk = 
Spark_Connect_AddArtifactsRequest.BeginChunkedArtifact()
             beginChunk.name = name
diff --git a/Sources/SparkConnect/SparkConnectClient+ReattachableExecute.swift 
b/Sources/SparkConnect/SparkConnectClient+ReattachableExecute.swift
index c1b3817..d318f6b 100644
--- a/Sources/SparkConnect/SparkConnectClient+ReattachableExecute.swift
+++ b/Sources/SparkConnect/SparkConnectClient+ReattachableExecute.swift
@@ -80,6 +80,9 @@ extension SparkConnectClient {
     reattach.userContext = request.userContext
     reattach.operationID = request.operationID
     reattach.clientType = request.clientType
+    if request.hasClientObservedServerSideSessionID {
+      reattach.clientObservedServerSideSessionID = 
request.clientObservedServerSideSessionID
+    }
     if let lastResponseID {
       reattach.lastResponseID = lastResponseID
     }
@@ -93,6 +96,9 @@ extension SparkConnectClient {
     release.userContext = request.userContext
     release.operationID = request.operationID
     release.clientType = request.clientType
+    if request.hasClientObservedServerSideSessionID {
+      release.clientObservedServerSideSessionID = 
request.clientObservedServerSideSessionID
+    }
     release.releaseAll = ReleaseExecuteRequest.ReleaseAll()
     return release
   }
diff --git a/Sources/SparkConnect/SparkConnectClient.swift 
b/Sources/SparkConnect/SparkConnectClient.swift
index 050f449..8dfd1e8 100644
--- a/Sources/SparkConnect/SparkConnectClient.swift
+++ b/Sources/SparkConnect/SparkConnectClient.swift
@@ -42,6 +42,11 @@ public actor SparkConnectClient {
   let serviceConfig: ServiceConfig
   var intercepters: [ClientInterceptor] = []
   let userContext: UserContext
+  /// The server-side session ID observed by ``connect(_:)``. It is sent back 
as
+  /// `client_observed_server_side_session_id` in requests in order to make 
the server reject them
+  /// with `INVALID_HANDLE.SESSION_CHANGED` when the server-side session 
changed, e.g., after the
+  /// server restarted, instead of silently running them in a new session.
+  var serverSideSessionID: String? = nil
   var sessionID: String? = nil
   var tags = Set<String>()
 
@@ -169,6 +174,9 @@ public actor SparkConnectClient {
           return OneOf_Analyze.sparkVersion(AnalyzePlanRequest.SparkVersion())
         })
       let response = try await service.analyzePlan(request)
+      if !response.serverSideSessionID.isEmpty {
+        self.serverSideSessionID = response.serverSideSessionID
+      }
       return response
     }
   }
@@ -238,6 +246,9 @@ public actor SparkConnectClient {
       request.clientType = clientType
       request.userContext = userContext
       request.sessionID = self.sessionID!
+      if let serverSideSessionID {
+        request.clientObservedServerSideSessionID = serverSideSessionID
+      }
       _ = try await service.config(request)
       return true
     }
@@ -266,6 +277,9 @@ public actor SparkConnectClient {
       request.clientType = clientType
       request.userContext = userContext
       request.sessionID = self.sessionID!
+      if let serverSideSessionID {
+        request.clientObservedServerSideSessionID = serverSideSessionID
+      }
       let _ = try await service.config(request)
       return true
     }
@@ -293,6 +307,9 @@ public actor SparkConnectClient {
       request.clientType = clientType
       request.userContext = userContext
       request.sessionID = self.sessionID!
+      if let serverSideSessionID {
+        request.clientObservedServerSideSessionID = serverSideSessionID
+      }
       let response = try await service.config(request)
       return try Self.getConfValue(key, from: response)
     }
@@ -335,6 +352,9 @@ public actor SparkConnectClient {
       request.clientType = clientType
       request.userContext = userContext
       request.sessionID = self.sessionID!
+      if let serverSideSessionID {
+        request.clientObservedServerSideSessionID = serverSideSessionID
+      }
       let response = try await service.config(request)
       let result =
         if let pair = response.pairs.first, pair.hasValue {
@@ -368,6 +388,9 @@ public actor SparkConnectClient {
       request.clientType = clientType
       request.userContext = userContext
       request.sessionID = self.sessionID!
+      if let serverSideSessionID {
+        request.clientObservedServerSideSessionID = serverSideSessionID
+      }
       let response = try await service.config(request)
       let result: String? =
         if let pair = response.pairs.first, pair.hasValue {
@@ -398,6 +421,9 @@ public actor SparkConnectClient {
       request.clientType = clientType
       request.userContext = userContext
       request.sessionID = self.sessionID!
+      if let serverSideSessionID {
+        request.clientObservedServerSideSessionID = serverSideSessionID
+      }
       let response = try await service.config(request)
       var map = [String: String]()
       for pair in response.pairs {
@@ -431,6 +457,9 @@ public actor SparkConnectClient {
       request.clientType = clientType
       request.userContext = userContext
       request.sessionID = self.sessionID!
+      if let serverSideSessionID {
+        request.clientObservedServerSideSessionID = serverSideSessionID
+      }
       let response = try await service.config(request)
       return response.pairs.first?.value == "true"
     }
@@ -500,6 +529,9 @@ public actor SparkConnectClient {
     request.clientType = clientType
     request.userContext = userContext
     request.sessionID = self.sessionID!
+    if let serverSideSessionID {
+      request.clientObservedServerSideSessionID = serverSideSessionID
+    }
     // The server normalizes the operation ID of `ExecutePlan` to a lowercase 
UUID string, but
     // looks up the operation ID of `ReattachExecute` and `ReleaseExecute` as 
it is.
     request.operationID = UUID().uuidString.lowercased()
@@ -535,6 +567,9 @@ public actor SparkConnectClient {
     request.clientType = clientType
     request.userContext = userContext
     request.sessionID = self.sessionID!
+    if let serverSideSessionID {
+      request.clientObservedServerSideSessionID = serverSideSessionID
+    }
     request.analyze = f()
     return request
   }
@@ -1026,6 +1061,9 @@ public actor SparkConnectClient {
     request.sessionID = self.sessionID!
     request.userContext = self.userContext
     request.clientType = self.clientType
+    if let serverSideSessionID {
+      request.clientObservedServerSideSessionID = serverSideSessionID
+    }
     request.interruptType = .all
 
     return try await withGPRC { client in
@@ -1040,6 +1078,9 @@ public actor SparkConnectClient {
     request.sessionID = self.sessionID!
     request.userContext = self.userContext
     request.clientType = self.clientType
+    if let serverSideSessionID {
+      request.clientObservedServerSideSessionID = serverSideSessionID
+    }
     request.interruptType = .tag
     request.operationTag = tag
 
@@ -1055,6 +1096,9 @@ public actor SparkConnectClient {
     request.sessionID = self.sessionID!
     request.userContext = self.userContext
     request.clientType = self.clientType
+    if let serverSideSessionID {
+      request.clientObservedServerSideSessionID = serverSideSessionID
+    }
     request.interruptType = .operationID
     request.operationID = operationId
 
@@ -1072,6 +1116,9 @@ public actor SparkConnectClient {
     request.sessionID = self.sessionID!
     request.userContext = self.userContext
     request.clientType = self.clientType
+    if let serverSideSessionID {
+      request.clientObservedServerSideSessionID = serverSideSessionID
+    }
     if !operationIds.isEmpty {
       var operationStatusRequest = 
Spark_Connect_GetStatusRequest.OperationStatusRequest()
       operationStatusRequest.operationIds = operationIds
@@ -1094,6 +1141,9 @@ public actor SparkConnectClient {
     request.sessionID = self.sessionID!
     request.userContext = self.userContext
     request.clientType = self.clientType
+    if let serverSideSessionID {
+      request.clientObservedServerSideSessionID = serverSideSessionID
+    }
     if let newSessionID {
       request.newSessionID = newSessionID
     }
diff --git a/Sources/SparkConnect/SparkSession.swift 
b/Sources/SparkConnect/SparkSession.swift
index c62dc26..afcc4b4 100644
--- a/Sources/SparkConnect/SparkSession.swift
+++ b/Sources/SparkConnect/SparkSession.swift
@@ -108,9 +108,6 @@ public actor SparkSession {
     sessionID
   }
 
-  /// A server-side generated session ID. This is supposed to be overwritten 
during establishing connections.
-  var serverSideSessionID: String = ""
-
   /// A variable for ``SparkContext``. This is designed to throw exceptions by 
Apache Spark.
   public var sparkContext: SparkContext {
     get throws {
diff --git a/Tests/SparkConnectTests/ReattachableExecuteTests.swift 
b/Tests/SparkConnectTests/ReattachableExecuteTests.swift
index 04f8af8..02e857d 100644
--- a/Tests/SparkConnectTests/ReattachableExecuteTests.swift
+++ b/Tests/SparkConnectTests/ReattachableExecuteTests.swift
@@ -75,6 +75,20 @@ struct ReattachableExecuteTests {
     #expect(request.release == .releaseAll(ReleaseExecuteRequest.ReleaseAll()))
   }
 
+  @Test
+  func clientObservedServerSideSessionID() {
+    var request = Self.executePlanRequest()
+    let reattachWithoutID = 
SparkConnectClient.getReattachExecuteRequest(request, nil)
+    #expect(!reattachWithoutID.hasClientObservedServerSideSessionID)
+    
#expect(!SparkConnectClient.getReleaseAllRequest(request).hasClientObservedServerSideSessionID)
+
+    request.clientObservedServerSideSessionID = "server-side-session-id"
+    let reattach = SparkConnectClient.getReattachExecuteRequest(request, nil)
+    #expect(reattach.clientObservedServerSideSessionID == 
"server-side-session-id")
+    let release = SparkConnectClient.getReleaseAllRequest(request)
+    #expect(release.clientObservedServerSideSessionID == 
"server-side-session-id")
+  }
+
   /// Create an ``ExecutePlanRequest`` with `client` without connecting to the 
server.
   static func executePlanRequest(_ client: isolated SparkConnectClient) async
     -> ExecutePlanRequest
diff --git a/Tests/SparkConnectTests/SparkConnectClientTests.swift 
b/Tests/SparkConnectTests/SparkConnectClientTests.swift
index 485007b..89e6939 100644
--- a/Tests/SparkConnectTests/SparkConnectClientTests.swift
+++ b/Tests/SparkConnectTests/SparkConnectClientTests.swift
@@ -132,6 +132,96 @@ struct SparkConnectClientTests {
     await client.stop()
   }
 
+  /// Set the session IDs of `client` without connecting to the server.
+  static func setSessionIDs(
+    _ client: isolated SparkConnectClient, _ sessionID: String?, _ 
serverSideSessionID: String?
+  ) {
+    client.sessionID = sessionID
+    client.serverSideSessionID = serverSideSessionID
+  }
+
+  @Test
+  func clientObservedServerSideSessionID() async throws {
+    let client = try SparkConnectClient(remote: "sc://localhost")
+    let sessionID = UUID().uuidString.lowercased()
+    let plan = await client.getPlanRange(0, 1, 1)
+
+    await Self.setSessionIDs(client, sessionID, nil)
+    #expect(await 
!client.getExecutePlanRequest(plan).hasClientObservedServerSideSessionID)
+    let analyzePlanRequest = await client.getAnalyzePlanRequest(sessionID, 
plan)
+    #expect(!analyzePlanRequest.hasClientObservedServerSideSessionID)
+
+    let serverSideSessionID = UUID().uuidString.lowercased()
+    await Self.setSessionIDs(client, sessionID, serverSideSessionID)
+    let executePlanRequestWithID = await client.getExecutePlanRequest(plan)
+    #expect(executePlanRequestWithID.clientObservedServerSideSessionID == 
serverSideSessionID)
+    let analyzePlanRequestWithID = await 
client.getAnalyzePlanRequest(sessionID, plan)
+    #expect(analyzePlanRequestWithID.clientObservedServerSideSessionID == 
serverSideSessionID)
+  }
+
+  @Test
+  func serverSideSessionID() async throws {
+    let client = try SparkConnectClient(remote: TEST_REMOTE)
+    #expect(await client.serverSideSessionID == nil)
+    let response = try await client.connect(UUID().uuidString)
+    #expect(!response.serverSideSessionID.isEmpty)
+    #expect(await client.serverSideSessionID == response.serverSideSessionID)
+    await client.stop()
+  }
+
+  @Test
+  func serverSideSessionChanged() async throws {
+    let client = try SparkConnectClient(remote: TEST_REMOTE)
+    let response = try await client.connect(UUID().uuidString)
+    let version = response.sparkVersion.version
+    let sessionID = try #require(await client.sessionID)
+    let serverSideSessionID = try #require(await client.serverSideSessionID)
+    let key = "spark.test.serverSideSessionChanged"
+    let path = "server-side-session-changed.jar"
+    #expect(FileManager().createFile(atPath: path, contents: "abc".data(using: 
.utf8)))
+    defer { try? FileManager().removeItem(atPath: path) }
+
+    // Simulate that the server-side session changed, e.g., after the server 
restarted.
+    await Self.setSessionIDs(client, sessionID, UUID().uuidString.lowercased())
+    var calls: [() async throws -> Void] = [
+      { _ = try await client.ddlParse("a int") },
+      {
+        try await client.executePlanWithReattach(
+          client.getExecutePlanRequest(client.getPlanRange(0, 1, 1))
+        ) { _ in }
+      },
+      { _ = try await client.setConf(map: [key: "true"]) },
+      { _ = try await client.unsetConf(keys: [key]) },
+      { _ = try await client.getConf(key) },
+      { _ = try await client.getConfWithDefault(key, "false") },
+      { _ = try await client.getConfOption(key) },
+      { _ = try await client.getConfAll() },
+      { _ = try await client.isModifiable(key) },
+      { _ = try await client.interruptAll() },
+      { _ = try await client.interruptTag("tag") },
+      { _ = try await 
client.interruptOperation(UUID().uuidString.lowercased()) },
+      { _ = try await client.artifactExists("cache/nonexistent") },
+      { try await client.addArtifact(URL(fileURLWithPath: path)) },
+    ]
+    if isSparkVersionAtLeast(version, "4.1") {
+      calls.append { _ = try await client.cloneSession() }
+    }
+    if isSparkVersionAtLeast(version, "4.2") {
+      calls.append { _ = try await client.getOperationStatuses() }
+    }
+    for call in calls {
+      let error = try await #require(throws: Error.self) {
+        try await call()
+      }
+      #expect("\(error)".contains("INVALID_HANDLE.SESSION_CHANGED"))
+    }
+
+    // The server rejected the requests without running them in the session.
+    await Self.setSessionIDs(client, sessionID, serverSideSessionID)
+    #expect(try await client.getConfOption(key) == nil)
+    await client.stop()
+  }
+
   @Test
   func resultChunking() async throws {
     let client = try SparkConnectClient(remote: TEST_REMOTE)


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to