This is an automated email from the ASF dual-hosted git repository.

dongjoon-hyun pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/spark-connect-swift.git


The following commit(s) were added to refs/heads/main by this push:
     new 7dc5f0b  [SPARK-60511] Make SparkSession.Builder.create public
7dc5f0b is described below

commit 7dc5f0b0d13bfb280dae517812a319123d68970e
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Fri Oct 23 10:04:28 2026 -0700

    [SPARK-60511] Make SparkSession.Builder.create public
    
    ### What changes were proposed in this pull request?
    
    This PR aims to make `SparkSession.Builder.create()` `public` like Scala 
and Python clients. In addition, the documentation of 
`SparkSession.Builder.getOrCreate()` is fixed because it also always creates a 
new session.
    
    ### Why are the changes needed?
    
    Since SPARK-51483, `getOrCreate()` has been `return try await create()`, 
while `create()` is `internal`. So, every `getOrCreate()` call creates a new 
session in all releases from 
[v0.1.0](https://github.com/apache/spark-connect-swift/releases/tag/v0.1.0) 
(2025-05-04) to 
[0.7.0](https://github.com/apache/spark-connect-swift/releases/tag/0.7.0) 
(2026-07-22). However, the documentation added by SPARK-51991 says the 
following.
    
    > If a session with the same configuration already exists, it will be 
returned. Otherwise, a new session is created with the specified configuration.
    
    In [v4.2.0](https://github.com/apache/spark/releases/tag/v4.2.0) 
(2026-07-11), the other Spark Connect clients provide both APIs with different 
semantics.
    
    | | `create()` | `getOrCreate()` |
    | --- | --- | --- |
    | Scala 
([Builder](https://github.com/apache/spark/blob/v4.2.0/sql/connect/common/src/main/scala/org/apache/spark/sql/connect/SparkSession.scala#L1089-L1133))
 | A new session | A cached session for the same client configuration |
    | Python 
([Builder](https://github.com/apache/spark/blob/v4.2.0/python/pyspark/sql/connect/session.py#L227-L260))
 | A new session | The active or default session |
    
    This PR provides `create()` so that users can explicitly request a new 
session like the other clients. `getOrCreate()` keeps its behavior because 
existing applications and tests depend on it, e.g., each caller can `stop()` 
its own session without affecting the others. Its documentation now says that, 
unlike the other clients, it doesn't return an existing session.
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes, `SparkSession.builder.create()` becomes available as a public API. 
There is no behavior change.
    
    ### How was this patch tested?
    
    Pass the CIs with the newly added test case, `BuilderTests.create`.
    
    Since `BuilderTests` uses `testable import SparkConnect`, the visibility 
was verified manually with a temporary test file which uses `import 
SparkConnect`. It fails to compile without this change.
    
    ```
    error: 'create' is inaccessible due to 'internal' protection level
    ```
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Opus 5.5
    
    Closes #621 from dongjoon-hyun/SPARK-60511.
    
    Authored-by: Dongjoon Hyun <[email protected]>
    Signed-off-by: Dongjoon Hyun <[email protected]>
---
 Sources/SparkConnect/SparkSession.swift    | 23 +++++++++++++++++------
 Tests/SparkConnectTests/BuilderTests.swift |  9 +++++++++
 2 files changed, 26 insertions(+), 6 deletions(-)

diff --git a/Sources/SparkConnect/SparkSession.swift 
b/Sources/SparkConnect/SparkSession.swift
index afcc4b4..43f52b8 100644
--- a/Sources/SparkConnect/SparkSession.swift
+++ b/Sources/SparkConnect/SparkSession.swift
@@ -798,9 +798,19 @@ public actor SparkSession {
       sparkConf.filter { $0.key != "spark.remote" && $0.key != "spark.master" }
     }
 
-    /// Create a new ``SparkSession``. If `spark.remote` is not given, 
`sc://localhost:15002` is used.
+    /// Creates a new ``SparkSession``.
+    ///
+    /// Like Scala and Python clients, this always returns a newly created 
session. If
+    /// `spark.remote` is not given, the `SPARK_REMOTE` environment variable or
+    /// `sc://localhost:15002` is used.
+    ///
+    /// ```swift
+    /// let spark = try await SparkSession.builder.create()
+    /// ```
+    ///
     /// - Returns: A newly created ``SparkSession``.
-    func create() async throws -> SparkSession {
+    /// - Throws: ``SparkConnectError`` if connection fails or configuration 
is invalid
+    public func create() async throws -> SparkSession {
       let remote = ProcessInfo.processInfo.environment["SPARK_REMOTE"] ?? 
"sc://localhost:15002"
       let session = try SparkSession(sparkConf["spark.remote"] ?? remote)
       let response = try await session.client.connect(session.sessionID)
@@ -812,11 +822,12 @@ public actor SparkSession {
       return session
     }
 
-    /// Creates or retrieves an existing ``SparkSession``.
+    /// Creates a new ``SparkSession``.
     ///
-    /// This is the primary method for obtaining a ``SparkSession`` instance. 
If a session with
-    /// the same configuration already exists, it will be returned. Otherwise, 
a new session
-    /// is created with the specified configuration.
+    /// This is the primary method for obtaining a ``SparkSession`` instance. 
Unlike Scala and
+    /// Python clients, this does not return an existing session. Like 
``create()``, every call
+    /// creates a new session with the specified configuration, so reuse the 
returned session and
+    /// call ``SparkSession/stop()`` when it is no longer needed.
     ///
     /// The method will:
     /// 1. Check for the `SPARK_REMOTE` environment variable if no remote URL 
is set
diff --git a/Tests/SparkConnectTests/BuilderTests.swift 
b/Tests/SparkConnectTests/BuilderTests.swift
index 7717a37..5c60b95 100644
--- a/Tests/SparkConnectTests/BuilderTests.swift
+++ b/Tests/SparkConnectTests/BuilderTests.swift
@@ -77,4 +77,13 @@ struct BuilderTests {
     #expect(await builder.sparkConf["spark.app.name"] == "TestApp")
     try await builder.getOrCreate().stop()
   }
+
+  @Test
+  func create() async throws {
+    let spark1 = try await SparkSession.builder.create()
+    let spark2 = try await SparkSession.builder.create()
+    #expect(spark1 != spark2)
+    await spark1.stop()
+    await spark2.stop()
+  }
 }


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to