This is an automated email from the ASF dual-hosted git repository.

dongjoon-hyun pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/spark-connect-swift.git


The following commit(s) were added to refs/heads/main by this push:
     new 6ab8a43  [SPARK-60471] Fix `DataFrame.withColumnRenamed` to not crash 
on duplicate column names
6ab8a43 is described below

commit 6ab8a4342e04666275914a7911dc59bd8b317eba
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Thu Oct 22 08:06:29 2026 -0700

    [SPARK-60471] Fix `DataFrame.withColumnRenamed` to not crash on duplicate 
column names
    
    ### What changes were proposed in this pull request?
    
    This PR aims to fix `DataFrame.withColumnRenamed(_ colNames: [String], _ 
newColNames: [String])` to not crash on duplicate column names. It now sends 
the renames in the given order via the `renames` field of `WithColumnsRenamed`, 
like the Scala and Python Spark Connect clients.
    
    - Add `SparkConnectClient.getWithColumnRenamed(_:_:_:)`, which builds 
`WithColumnsRenamed.renames` from the given column name lists.
    - `withColumnRenamed(_ colNames: [String], _ newColNames: [String])` uses 
it instead of `Dictionary(uniqueKeysWithValues:)` and the deprecated 
`rename_columns_map` field. If the lists have different lengths, the extra 
names are ignored as before.
    
    `withColumnRenamed(_ existingName: String, _ newName: String)` and 
`withColumnRenamed(_ colsMap: [String: String])` are not changed.
    
    ### Why are the changes needed?
    
    `Dictionary(uniqueKeysWithValues:)` traps on duplicate keys, so the 
following crashes the client process although the API does not throw.
    
    ```swift
    // Fatal error: Duplicate values for key: 'a'
    df.withColumnRenamed(["a", "a"], ["x", "y"])
    ```
    
    In addition, `rename_columns_map` is an unordered `map`, and 
`swift-protobuf` serializes it in the iteration order of `Dictionary`, which 
differs across processes. Since the server applies the renames one by one, the 
result of order-sensitive renames was not deterministic. For example, 
`df.withColumnRenamed(["a", "x"], ["x", "y"])` returned `["x", "b", "c", "d"]` 
or `["y", "b", "c", "d"]` depending on the process.
    
    The `renames` field was added by SPARK-46261 in Apache Spark 
[v4.0.0](https://github.com/apache/spark/releases/tag/v4.0.0) (2025-05-19), so 
all servers supported by this client accept it. In `v4.2.0` (2026-07-11), the 
server reads it in 
[SparkConnectPlanner](https://github.com/apache/spark/blob/v4.2.0/sql/connect/server/src/main/scala/org/apache/spark/sql/connect/planner/SparkConnectPlanner.scala#L1287-L1310)
 and applies the renames in order in [UnresolvedStarWithColumnsRenames](https 
[...]
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes, it fixes a bug. Duplicate column names no longer crash, and the 
renames are applied in the given order, the same as the Scala and Python 
clients. The API signatures do not change.
    
    - `df.withColumnRenamed(["a", "a"], ["x", "y"])`: crash → `["x", "b", "c", 
"d"]`. The second `a` is ignored because `a` is already renamed to `x`, like 
`df.withColumnRenamed("a", "x").withColumnRenamed("a", "y")`.
    - `df.withColumnRenamed(["a", "x"], ["x", "y"])`: `["x", "b", "c", "d"]` or 
`["y", "b", "c", "d"]` → always `["y", "b", "c", "d"]`.
    
    ### How was this patch tested?
    
    Pass the CIs with the newly added test cases.
    
    - `DataFrameInternalTests.withColumnRenamedPlan`, which only inspects the 
generated plan, so it does not require a running server.
    - `DataFrameTests.withColumnRenamedWithDuplicateNames`
    
    Both crashed with `Fatal error: Duplicate values for key: 'a'` before this 
fix. In addition, `DataFrameTests` and `DataFrameInternalTests` passed against 
`apache/spark:4.2.0` and `apache/spark:4.0.4-java21-scala` Spark Connect 
servers, and the Scala Spark Connect client returned the same results against 
`apache/spark:4.2.0`.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Opus 5.5
    
    Closes #617 from dongjoon-hyun/SPARK-60471.
    
    Authored-by: Dongjoon Hyun <[email protected]>
    Signed-off-by: Dongjoon Hyun <[email protected]>
---
 Sources/SparkConnect/DataFrame+Transformations.swift |  4 ++--
 Sources/SparkConnect/SparkConnectClient.swift        | 14 ++++++++++++++
 Tests/SparkConnectTests/DataFrameInternalTests.swift | 13 +++++++++++++
 Tests/SparkConnectTests/DataFrameTests.swift         | 10 ++++++++++
 4 files changed, 39 insertions(+), 2 deletions(-)

diff --git a/Sources/SparkConnect/DataFrame+Transformations.swift 
b/Sources/SparkConnect/DataFrame+Transformations.swift
index e40e25f..14a2973 100644
--- a/Sources/SparkConnect/DataFrame+Transformations.swift
+++ b/Sources/SparkConnect/DataFrame+Transformations.swift
@@ -287,9 +287,9 @@ extension DataFrame {
   ///   - newColNames: A list of new column names.
   /// - Returns: A ``DataFrame`` with the renamed columns.
   public func withColumnRenamed(_ colNames: [String], _ newColNames: [String]) 
-> DataFrame {
-    let dic = Dictionary(uniqueKeysWithValues: zip(colNames, newColNames))
     return DataFrame(
-      spark: self.spark, plan: 
SparkConnectClient.getWithColumnRenamed(self.plan.root, dic))
+      spark: self.spark,
+      plan: SparkConnectClient.getWithColumnRenamed(self.plan.root, colNames, 
newColNames))
   }
 
   /// Returns a new Dataset with columns renamed. This is a no-op if schema 
doesn't contain existingName.
diff --git a/Sources/SparkConnect/SparkConnectClient.swift 
b/Sources/SparkConnect/SparkConnectClient.swift
index 0822cd9..4bbbfb0 100644
--- a/Sources/SparkConnect/SparkConnectClient.swift
+++ b/Sources/SparkConnect/SparkConnectClient.swift
@@ -748,6 +748,20 @@ public actor SparkConnectClient {
     return createPlan { $0.withColumnsRenamed = withColumnsRenamed }
   }
 
+  static func getWithColumnRenamed(
+    _ child: Relation, _ colNames: [String], _ newColNames: [String]
+  ) -> Plan {
+    var withColumnsRenamed = WithColumnsRenamed()
+    withColumnsRenamed.input = child
+    withColumnsRenamed.renames = zip(colNames, newColNames).map { (colName, 
newColName) in
+      var rename = WithColumnsRenamed.Rename()
+      rename.colName = colName
+      rename.newColName = newColName
+      return rename
+    }
+    return createPlan { $0.withColumnsRenamed = withColumnsRenamed }
+  }
+
   static func getWithColumns(_ child: Relation, _ colsMap: [String: String]) 
-> Plan {
     var withColumns = WithColumns()
     withColumns.input = child
diff --git a/Tests/SparkConnectTests/DataFrameInternalTests.swift 
b/Tests/SparkConnectTests/DataFrameInternalTests.swift
index 2b54315..ee091ec 100644
--- a/Tests/SparkConnectTests/DataFrameInternalTests.swift
+++ b/Tests/SparkConnectTests/DataFrameInternalTests.swift
@@ -236,4 +236,17 @@ struct DataFrameInternalTests {
     let other = await df.stat.sampleBy("key", [[1]: 0.5], 
0).plan.root.sampleBy.fractions
     #expect(other[0].stratum.literalType == .string("[1]"))
   }
+
+  @Test
+  func withColumnRenamedPlan() async throws {
+    // This test doesn't require a running server because it only inspects the 
plan.
+    let spark = try SparkSession("sc://localhost")
+    var plan = Plan()
+    plan.root.range.end = 10
+    let df = DataFrame(spark: spark, plan: plan)
+    let renames = await df.withColumnRenamed(["a", "b", "a"], ["b", "c", 
"d"]).plan.root
+      .withColumnsRenamed.renames
+    #expect(renames.map { $0.colName } == ["a", "b", "a"])
+    #expect(renames.map { $0.newColName } == ["b", "c", "d"])
+  }
 }
diff --git a/Tests/SparkConnectTests/DataFrameTests.swift 
b/Tests/SparkConnectTests/DataFrameTests.swift
index 063a359..8687a1b 100644
--- a/Tests/SparkConnectTests/DataFrameTests.swift
+++ b/Tests/SparkConnectTests/DataFrameTests.swift
@@ -635,6 +635,16 @@ struct DataFrameTests {
     await spark.stop()
   }
 
+  @Test
+  func withColumnRenamedWithDuplicateNames() async throws {
+    let spark = try await SparkSession.builder.getOrCreate()
+    let df = try await spark.sql("SELECT 1 a, 2 b, 3 c, 4 d")
+    // Renames are applied in the given order. The second `a` is ignored 
because `a` is renamed.
+    #expect(try await df.withColumnRenamed(["a", "a"], ["x", "y"]).columns == 
["x", "b", "c", "d"])
+    #expect(try await df.withColumnRenamed(["a", "x"], ["x", "y"]).columns == 
["y", "b", "c", "d"])
+    await spark.stop()
+  }
+
   @Test
   func withColumn() async throws {
     let spark = try await SparkSession.builder.getOrCreate()


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to