This is an automated email from the ASF dual-hosted git repository.
dongjoon-hyun pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/spark-connect-swift.git
The following commit(s) were added to refs/heads/main by this push:
new 6ab8a43 [SPARK-60471] Fix `DataFrame.withColumnRenamed` to not crash
on duplicate column names
6ab8a43 is described below
commit 6ab8a4342e04666275914a7911dc59bd8b317eba
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Thu Oct 22 08:06:29 2026 -0700
[SPARK-60471] Fix `DataFrame.withColumnRenamed` to not crash on duplicate
column names
### What changes were proposed in this pull request?
This PR aims to fix `DataFrame.withColumnRenamed(_ colNames: [String], _
newColNames: [String])` to not crash on duplicate column names. It now sends
the renames in the given order via the `renames` field of `WithColumnsRenamed`,
like the Scala and Python Spark Connect clients.
- Add `SparkConnectClient.getWithColumnRenamed(_:_:_:)`, which builds
`WithColumnsRenamed.renames` from the given column name lists.
- `withColumnRenamed(_ colNames: [String], _ newColNames: [String])` uses
it instead of `Dictionary(uniqueKeysWithValues:)` and the deprecated
`rename_columns_map` field. If the lists have different lengths, the extra
names are ignored as before.
`withColumnRenamed(_ existingName: String, _ newName: String)` and
`withColumnRenamed(_ colsMap: [String: String])` are not changed.
### Why are the changes needed?
`Dictionary(uniqueKeysWithValues:)` traps on duplicate keys, so the
following crashes the client process although the API does not throw.
```swift
// Fatal error: Duplicate values for key: 'a'
df.withColumnRenamed(["a", "a"], ["x", "y"])
```
In addition, `rename_columns_map` is an unordered `map`, and
`swift-protobuf` serializes it in the iteration order of `Dictionary`, which
differs across processes. Since the server applies the renames one by one, the
result of order-sensitive renames was not deterministic. For example,
`df.withColumnRenamed(["a", "x"], ["x", "y"])` returned `["x", "b", "c", "d"]`
or `["y", "b", "c", "d"]` depending on the process.
The `renames` field was added by SPARK-46261 in Apache Spark
[v4.0.0](https://github.com/apache/spark/releases/tag/v4.0.0) (2025-05-19), so
all servers supported by this client accept it. In `v4.2.0` (2026-07-11), the
server reads it in
[SparkConnectPlanner](https://github.com/apache/spark/blob/v4.2.0/sql/connect/server/src/main/scala/org/apache/spark/sql/connect/planner/SparkConnectPlanner.scala#L1287-L1310)
and applies the renames in order in [UnresolvedStarWithColumnsRenames](https
[...]
### Does this PR introduce _any_ user-facing change?
Yes, it fixes a bug. Duplicate column names no longer crash, and the
renames are applied in the given order, the same as the Scala and Python
clients. The API signatures do not change.
- `df.withColumnRenamed(["a", "a"], ["x", "y"])`: crash → `["x", "b", "c",
"d"]`. The second `a` is ignored because `a` is already renamed to `x`, like
`df.withColumnRenamed("a", "x").withColumnRenamed("a", "y")`.
- `df.withColumnRenamed(["a", "x"], ["x", "y"])`: `["x", "b", "c", "d"]` or
`["y", "b", "c", "d"]` → always `["y", "b", "c", "d"]`.
### How was this patch tested?
Pass the CIs with the newly added test cases.
- `DataFrameInternalTests.withColumnRenamedPlan`, which only inspects the
generated plan, so it does not require a running server.
- `DataFrameTests.withColumnRenamedWithDuplicateNames`
Both crashed with `Fatal error: Duplicate values for key: 'a'` before this
fix. In addition, `DataFrameTests` and `DataFrameInternalTests` passed against
`apache/spark:4.2.0` and `apache/spark:4.0.4-java21-scala` Spark Connect
servers, and the Scala Spark Connect client returned the same results against
`apache/spark:4.2.0`.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
Closes #617 from dongjoon-hyun/SPARK-60471.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
---
Sources/SparkConnect/DataFrame+Transformations.swift | 4 ++--
Sources/SparkConnect/SparkConnectClient.swift | 14 ++++++++++++++
Tests/SparkConnectTests/DataFrameInternalTests.swift | 13 +++++++++++++
Tests/SparkConnectTests/DataFrameTests.swift | 10 ++++++++++
4 files changed, 39 insertions(+), 2 deletions(-)
diff --git a/Sources/SparkConnect/DataFrame+Transformations.swift
b/Sources/SparkConnect/DataFrame+Transformations.swift
index e40e25f..14a2973 100644
--- a/Sources/SparkConnect/DataFrame+Transformations.swift
+++ b/Sources/SparkConnect/DataFrame+Transformations.swift
@@ -287,9 +287,9 @@ extension DataFrame {
/// - newColNames: A list of new column names.
/// - Returns: A ``DataFrame`` with the renamed columns.
public func withColumnRenamed(_ colNames: [String], _ newColNames: [String])
-> DataFrame {
- let dic = Dictionary(uniqueKeysWithValues: zip(colNames, newColNames))
return DataFrame(
- spark: self.spark, plan:
SparkConnectClient.getWithColumnRenamed(self.plan.root, dic))
+ spark: self.spark,
+ plan: SparkConnectClient.getWithColumnRenamed(self.plan.root, colNames,
newColNames))
}
/// Returns a new Dataset with columns renamed. This is a no-op if schema
doesn't contain existingName.
diff --git a/Sources/SparkConnect/SparkConnectClient.swift
b/Sources/SparkConnect/SparkConnectClient.swift
index 0822cd9..4bbbfb0 100644
--- a/Sources/SparkConnect/SparkConnectClient.swift
+++ b/Sources/SparkConnect/SparkConnectClient.swift
@@ -748,6 +748,20 @@ public actor SparkConnectClient {
return createPlan { $0.withColumnsRenamed = withColumnsRenamed }
}
+ static func getWithColumnRenamed(
+ _ child: Relation, _ colNames: [String], _ newColNames: [String]
+ ) -> Plan {
+ var withColumnsRenamed = WithColumnsRenamed()
+ withColumnsRenamed.input = child
+ withColumnsRenamed.renames = zip(colNames, newColNames).map { (colName,
newColName) in
+ var rename = WithColumnsRenamed.Rename()
+ rename.colName = colName
+ rename.newColName = newColName
+ return rename
+ }
+ return createPlan { $0.withColumnsRenamed = withColumnsRenamed }
+ }
+
static func getWithColumns(_ child: Relation, _ colsMap: [String: String])
-> Plan {
var withColumns = WithColumns()
withColumns.input = child
diff --git a/Tests/SparkConnectTests/DataFrameInternalTests.swift
b/Tests/SparkConnectTests/DataFrameInternalTests.swift
index 2b54315..ee091ec 100644
--- a/Tests/SparkConnectTests/DataFrameInternalTests.swift
+++ b/Tests/SparkConnectTests/DataFrameInternalTests.swift
@@ -236,4 +236,17 @@ struct DataFrameInternalTests {
let other = await df.stat.sampleBy("key", [[1]: 0.5],
0).plan.root.sampleBy.fractions
#expect(other[0].stratum.literalType == .string("[1]"))
}
+
+ @Test
+ func withColumnRenamedPlan() async throws {
+ // This test doesn't require a running server because it only inspects the
plan.
+ let spark = try SparkSession("sc://localhost")
+ var plan = Plan()
+ plan.root.range.end = 10
+ let df = DataFrame(spark: spark, plan: plan)
+ let renames = await df.withColumnRenamed(["a", "b", "a"], ["b", "c",
"d"]).plan.root
+ .withColumnsRenamed.renames
+ #expect(renames.map { $0.colName } == ["a", "b", "a"])
+ #expect(renames.map { $0.newColName } == ["b", "c", "d"])
+ }
}
diff --git a/Tests/SparkConnectTests/DataFrameTests.swift
b/Tests/SparkConnectTests/DataFrameTests.swift
index 063a359..8687a1b 100644
--- a/Tests/SparkConnectTests/DataFrameTests.swift
+++ b/Tests/SparkConnectTests/DataFrameTests.swift
@@ -635,6 +635,16 @@ struct DataFrameTests {
await spark.stop()
}
+ @Test
+ func withColumnRenamedWithDuplicateNames() async throws {
+ let spark = try await SparkSession.builder.getOrCreate()
+ let df = try await spark.sql("SELECT 1 a, 2 b, 3 c, 4 d")
+ // Renames are applied in the given order. The second `a` is ignored
because `a` is renamed.
+ #expect(try await df.withColumnRenamed(["a", "a"], ["x", "y"]).columns ==
["x", "b", "c", "d"])
+ #expect(try await df.withColumnRenamed(["a", "x"], ["x", "y"]).columns ==
["y", "b", "c", "d"])
+ await spark.stop()
+ }
+
@Test
func withColumn() async throws {
let spark = try await SparkSession.builder.getOrCreate()
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]