This is an automated email from the ASF dual-hosted git repository.
dongjoon-hyun pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/spark-connect-swift.git
The following commit(s) were added to refs/heads/main by this push:
new 7dc5f0b [SPARK-60511] Make SparkSession.Builder.create public
7dc5f0b is described below
commit 7dc5f0b0d13bfb280dae517812a319123d68970e
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Fri Oct 23 10:04:28 2026 -0700
[SPARK-60511] Make SparkSession.Builder.create public
### What changes were proposed in this pull request?
This PR aims to make `SparkSession.Builder.create()` `public` like Scala
and Python clients. In addition, the documentation of
`SparkSession.Builder.getOrCreate()` is fixed because it also always creates a
new session.
### Why are the changes needed?
Since SPARK-51483, `getOrCreate()` has been `return try await create()`,
while `create()` is `internal`. So, every `getOrCreate()` call creates a new
session in all releases from
[v0.1.0](https://github.com/apache/spark-connect-swift/releases/tag/v0.1.0)
(2025-05-04) to
[0.7.0](https://github.com/apache/spark-connect-swift/releases/tag/0.7.0)
(2026-07-22). However, the documentation added by SPARK-51991 says the
following.
> If a session with the same configuration already exists, it will be
returned. Otherwise, a new session is created with the specified configuration.
In [v4.2.0](https://github.com/apache/spark/releases/tag/v4.2.0)
(2026-07-11), the other Spark Connect clients provide both APIs with different
semantics.
| | `create()` | `getOrCreate()` |
| --- | --- | --- |
| Scala
([Builder](https://github.com/apache/spark/blob/v4.2.0/sql/connect/common/src/main/scala/org/apache/spark/sql/connect/SparkSession.scala#L1089-L1133))
| A new session | A cached session for the same client configuration |
| Python
([Builder](https://github.com/apache/spark/blob/v4.2.0/python/pyspark/sql/connect/session.py#L227-L260))
| A new session | The active or default session |
This PR provides `create()` so that users can explicitly request a new
session like the other clients. `getOrCreate()` keeps its behavior because
existing applications and tests depend on it, e.g., each caller can `stop()`
its own session without affecting the others. Its documentation now says that,
unlike the other clients, it doesn't return an existing session.
### Does this PR introduce _any_ user-facing change?
Yes, `SparkSession.builder.create()` becomes available as a public API.
There is no behavior change.
### How was this patch tested?
Pass the CIs with the newly added test case, `BuilderTests.create`.
Since `BuilderTests` uses `testable import SparkConnect`, the visibility
was verified manually with a temporary test file which uses `import
SparkConnect`. It fails to compile without this change.
```
error: 'create' is inaccessible due to 'internal' protection level
```
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
Closes #621 from dongjoon-hyun/SPARK-60511.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
---
Sources/SparkConnect/SparkSession.swift | 23 +++++++++++++++++------
Tests/SparkConnectTests/BuilderTests.swift | 9 +++++++++
2 files changed, 26 insertions(+), 6 deletions(-)
diff --git a/Sources/SparkConnect/SparkSession.swift
b/Sources/SparkConnect/SparkSession.swift
index afcc4b4..43f52b8 100644
--- a/Sources/SparkConnect/SparkSession.swift
+++ b/Sources/SparkConnect/SparkSession.swift
@@ -798,9 +798,19 @@ public actor SparkSession {
sparkConf.filter { $0.key != "spark.remote" && $0.key != "spark.master" }
}
- /// Create a new ``SparkSession``. If `spark.remote` is not given,
`sc://localhost:15002` is used.
+ /// Creates a new ``SparkSession``.
+ ///
+ /// Like Scala and Python clients, this always returns a newly created
session. If
+ /// `spark.remote` is not given, the `SPARK_REMOTE` environment variable or
+ /// `sc://localhost:15002` is used.
+ ///
+ /// ```swift
+ /// let spark = try await SparkSession.builder.create()
+ /// ```
+ ///
/// - Returns: A newly created ``SparkSession``.
- func create() async throws -> SparkSession {
+ /// - Throws: ``SparkConnectError`` if connection fails or configuration
is invalid
+ public func create() async throws -> SparkSession {
let remote = ProcessInfo.processInfo.environment["SPARK_REMOTE"] ??
"sc://localhost:15002"
let session = try SparkSession(sparkConf["spark.remote"] ?? remote)
let response = try await session.client.connect(session.sessionID)
@@ -812,11 +822,12 @@ public actor SparkSession {
return session
}
- /// Creates or retrieves an existing ``SparkSession``.
+ /// Creates a new ``SparkSession``.
///
- /// This is the primary method for obtaining a ``SparkSession`` instance.
If a session with
- /// the same configuration already exists, it will be returned. Otherwise,
a new session
- /// is created with the specified configuration.
+ /// This is the primary method for obtaining a ``SparkSession`` instance.
Unlike Scala and
+ /// Python clients, this does not return an existing session. Like
``create()``, every call
+ /// creates a new session with the specified configuration, so reuse the
returned session and
+ /// call ``SparkSession/stop()`` when it is no longer needed.
///
/// The method will:
/// 1. Check for the `SPARK_REMOTE` environment variable if no remote URL
is set
diff --git a/Tests/SparkConnectTests/BuilderTests.swift
b/Tests/SparkConnectTests/BuilderTests.swift
index 7717a37..5c60b95 100644
--- a/Tests/SparkConnectTests/BuilderTests.swift
+++ b/Tests/SparkConnectTests/BuilderTests.swift
@@ -77,4 +77,13 @@ struct BuilderTests {
#expect(await builder.sparkConf["spark.app.name"] == "TestApp")
try await builder.getOrCreate().stop()
}
+
+ @Test
+ func create() async throws {
+ let spark1 = try await SparkSession.builder.create()
+ let spark2 = try await SparkSession.builder.create()
+ #expect(spark1 != spark2)
+ await spark1.stop()
+ await spark2.stop()
+ }
}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]