dongjoon-hyun opened a new pull request, #617: URL: https://github.com/apache/spark-connect-swift/pull/617
### What changes were proposed in this pull request? This PR aims to fix `DataFrame.withColumnRenamed(_ colNames: [String], _ newColNames: [String])` to not crash on duplicate column names. It now sends the renames in the given order via the `renames` field of `WithColumnsRenamed`, like the Scala and Python Spark Connect clients. - Add `SparkConnectClient.getWithColumnRenamed(_:_:_:)`, which builds `WithColumnsRenamed.renames` from the given column name lists. - `withColumnRenamed(_ colNames: [String], _ newColNames: [String])` uses it instead of `Dictionary(uniqueKeysWithValues:)` and the deprecated `rename_columns_map` field. If the lists have different lengths, the extra names are ignored as before. `withColumnRenamed(_ existingName: String, _ newName: String)` and `withColumnRenamed(_ colsMap: [String: String])` are not changed. ### Why are the changes needed? `Dictionary(uniqueKeysWithValues:)` traps on duplicate keys, so the following crashes the client process although the API does not throw. ```swift // Fatal error: Duplicate values for key: 'a' df.withColumnRenamed(["a", "a"], ["x", "y"]) ``` In addition, `rename_columns_map` is an unordered `map`, and `swift-protobuf` serializes it in the iteration order of `Dictionary`, which differs across processes. Since the server applies the renames one by one, the result of order-sensitive renames was not deterministic. For example, `df.withColumnRenamed(["a", "x"], ["x", "y"])` returned `["x", "b", "c", "d"]` or `["y", "b", "c", "d"]` depending on the process. The `renames` field was added by SPARK-46261 in Apache Spark [v4.0.0](https://github.com/apache/spark/releases/tag/v4.0.0) (2025-05-19), so all servers supported by this client accept it. In `v4.2.0` (2026-07-11), the server reads it in [SparkConnectPlanner](https://github.com/apache/spark/blob/v4.2.0/sql/connect/server/src/main/scala/org/apache/spark/sql/connect/planner/SparkConnectPlanner.scala#L1287-L1310) and applies the renames in order in [UnresolvedStarWithColumnsRenames](https://github.com/apache/spark/blob/v4.2.0/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/unresolved.scala#L856-L865). The Scala client ([Dataset.withColumnsRenamed](https://github.com/apache/spark/blob/v4.2.0/sql/connect/common/src/main/scala/org/apache/spark/sql/connect/Dataset.scala#L828-L846)) and the Python client ([WithColumnsRenamed](https://github.com/apache/spark/blob/v4.2.0/python/pyspark/sql/connect/plan.py#L1687-L1702)) also use `renames`. ### Does this PR introduce _any_ user-facing change? Yes, it fixes a bug. Duplicate column names no longer crash, and the renames are applied in the given order, the same as the Scala and Python clients. The API signatures do not change. - `df.withColumnRenamed(["a", "a"], ["x", "y"])`: crash → `["x", "b", "c", "d"]`. The second `a` is ignored because `a` is already renamed to `x`, like `df.withColumnRenamed("a", "x").withColumnRenamed("a", "y")`. - `df.withColumnRenamed(["a", "x"], ["x", "y"])`: `["x", "b", "c", "d"]` or `["y", "b", "c", "d"]` → always `["y", "b", "c", "d"]`. ### How was this patch tested? Pass the CIs with the newly added test cases. - `DataFrameInternalTests.withColumnRenamedPlan`, which only inspects the generated plan, so it does not require a running server. - `DataFrameTests.withColumnRenamedWithDuplicateNames` Both crashed with `Fatal error: Duplicate values for key: 'a'` before this fix. In addition, `DataFrameTests` and `DataFrameInternalTests` passed against `apache/spark:4.2.0` and `apache/spark:4.0.4-java21-scala` Spark Connect servers, and the Scala Spark Connect client returned the same results against `apache/spark:4.2.0`. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Opus 5.5 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
