This is an automated email from the ASF dual-hosted git repository.

dongjoon-hyun pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/spark-connect-swift.git


The following commit(s) were added to refs/heads/main by this push:
     new a8d5ff5  [SPARK-60516] Fix `DataFrame.toDF` to rename columns instead 
of selecting them
a8d5ff5 is described below

commit a8d5ff58d657dd08621801a44acadac247e18143
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Fri Oct 23 10:05:17 2026 -0700

    [SPARK-60516] Fix `DataFrame.toDF` to rename columns instead of selecting 
them
    
    ### What changes were proposed in this pull request?
    
    This PR aims to fix `DataFrame.toDF(_:)` to rename the columns by position 
with the `ToDF` relation, like the Scala and PySpark Spark Connect clients.
    
    ```swift
    -        DataFrame(spark: self.spark, plan: 
SparkConnectClient.getProject(self.plan.root, cols))
    +        DataFrame(spark: self.spark, plan: 
SparkConnectClient.getToDF(self.plan.root, cols))
    ```
    
    `toDF()` without column names still returns the `DataFrame` as is, like 
`Dataset.toDF()` in Scala.
    
    ### Why are the changes needed?
    
    `toDF` was added for feature parity in SPARK-51995 
([#111](https://github.com/apache/spark-connect-swift/pull/111)), but it sends 
a `Project` relation with the given names, which is the same as `select(_:)`. 
Its documentation was also copied from `select(_:)`.
    
    In Scala and PySpark, `toDF` renames all columns by position, and both 
Spark Connect clients send the `ToDF` relation.
    
    - [Apache Spark 4.2.0 (2026-07-11) 
`Dataset.toDF`](https://github.com/apache/spark/blob/v4.2.0/sql/connect/common/src/main/scala/org/apache/spark/sql/connect/Dataset.scala#L186-L190)
    
    ```scala
    def toDF(colNames: String*): DataFrame = sparkSession.newDataFrame { 
builder =>
      builder.getToDfBuilder
        .setInput(plan.getRoot)
        .addAllColumnNames(colNames.asJava)
    }
    ```
    
    - [Apache Spark 4.2.0 (2026-07-11) PySpark 
`DataFrame.toDF`](https://github.com/apache/spark/blob/v4.2.0/python/pyspark/sql/connect/dataframe.py#L2080-L2091)
    
    ```python
    return DataFrame(plan.ToDF(self._plan, list(cols)), self._session)
    ```
    
    As a result, the same code behaves differently in the Swift client. 
`spark.range(1).toDF("x")` fails only in Swift, and `toDF("b", "a")` on the 
columns `a` and `b` swaps the values without an error.
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes. `toDF` renames the columns by position like Scala and PySpark. This is 
a user-facing change compared to all released versions from [v0.1.0 
(2025-05-04)](https://github.com/apache/spark-connect-swift/releases/tag/v0.1.0)
 to [0.7.0 
(2026-07-22)](https://github.com/apache/spark-connect-swift/releases/tag/0.7.0).
    
    The number of column names must be the same as the number of columns. To 
select or reorder columns by name, use `select(_:)` instead. `toDF()` and 
`toDF` with all column names in the same order return the same results as 
before.
    
    **BEFORE**
    
    ```swift
    let df = try await spark.sql("SELECT 1 a, 2 b")
    try await spark.range(1).toDF("x").columns  // throws 
[UNRESOLVED_COLUMN.WITH_SUGGESTION]
    try await df.toDF("b", "a").collect()       // [Row(2, 1)]
    try await df.toDF("a").columns              // ["a"]
    ```
    
    **AFTER**
    
    ```swift
    let df = try await spark.sql("SELECT 1 a, 2 b")
    try await spark.range(1).toDF("x").columns  // ["x"]
    try await df.toDF("b", "a").collect()       // [Row(1, 2)]
    try await df.toDF("a").columns              // throws 
[ASSIGNMENT_ARITY_MISMATCH]
    ```
    
    ### How was this patch tested?
    
    Pass the CIs with the updated test case.
    
    I also tested manually with Apache Spark 4.2.0 and 4.0.4 Spark Connect 
servers. The Scala Spark Connect client of Apache Spark 4.2.0 returns the same 
results as the AFTER example.
    
    ```
    $ swift test --no-parallel --filter "DataFrameTests|DataFrameInternalTests"
    ...
    Test run with 178 tests in 3 suites passed after 31.553 seconds.
    ```
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Opus 5.5
    
    Closes #622 from dongjoon-hyun/SPARK-60516.
    
    Authored-by: Dongjoon Hyun <[email protected]>
    Signed-off-by: Dongjoon Hyun <[email protected]>
---
 Sources/SparkConnect/DataFrame+Transformations.swift | 17 +++++++++++++----
 Sources/SparkConnect/SparkConnectClient.swift        |  7 +++++++
 Tests/SparkConnectTests/DataFrameTests.swift         |  9 +++++++++
 3 files changed, 29 insertions(+), 4 deletions(-)

diff --git a/Sources/SparkConnect/DataFrame+Transformations.swift 
b/Sources/SparkConnect/DataFrame+Transformations.swift
index 14a2973..83222a4 100644
--- a/Sources/SparkConnect/DataFrame+Transformations.swift
+++ b/Sources/SparkConnect/DataFrame+Transformations.swift
@@ -123,15 +123,24 @@ extension DataFrame {
     return Column(expr)
   }
 
-  /// Selects a subset of existing columns using column names.
-  /// - Parameter cols: Column names
-  /// - Returns: A ``DataFrame`` with subset of columns.
+  /// Returns a new ``DataFrame`` with the columns renamed by position.
+  ///
+  /// The number of column names must be the same as the number of columns. 
Otherwise, an error is
+  /// thrown when the returned ``DataFrame`` is analyzed or executed. To 
select or reorder columns
+  /// by name, use ``select(_:)``.
+  ///
+  /// ```swift
+  /// let df = try await spark.sql("SELECT 1 a, 2 b")
+  /// try await df.toDF("x", "y").columns  // ["x", "y"]
+  /// ```
+  /// - Parameter cols: New column names. If empty, the columns are not 
renamed.
+  /// - Returns: A ``DataFrame`` with the new column names.
   public func toDF(_ cols: String...) -> DataFrame {
     let df =
       if cols.isEmpty {
         DataFrame(spark: self.spark, plan: self.plan)
       } else {
-        DataFrame(spark: self.spark, plan: 
SparkConnectClient.getProject(self.plan.root, cols))
+        DataFrame(spark: self.spark, plan: 
SparkConnectClient.getToDF(self.plan.root, cols))
       }
     return df
   }
diff --git a/Sources/SparkConnect/SparkConnectClient.swift 
b/Sources/SparkConnect/SparkConnectClient.swift
index 4bbbfb0..62d71cf 100644
--- a/Sources/SparkConnect/SparkConnectClient.swift
+++ b/Sources/SparkConnect/SparkConnectClient.swift
@@ -723,6 +723,13 @@ public actor SparkConnectClient {
     return createPlan { $0.toSchema = toSchema }
   }
 
+  static func getToDF(_ child: Relation, _ columnNames: [String]) -> Plan {
+    var toDF = Spark_Connect_ToDF()
+    toDF.input = child
+    toDF.columnNames = columnNames
+    return createPlan { $0.toDf = toDF }
+  }
+
   static func getProjectExprs(_ child: Relation, _ exprs: [String]) -> Plan {
     var project = Project()
     project.input = child
diff --git a/Tests/SparkConnectTests/DataFrameTests.swift 
b/Tests/SparkConnectTests/DataFrameTests.swift
index 7710d7a..71a119d 100644
--- a/Tests/SparkConnectTests/DataFrameTests.swift
+++ b/Tests/SparkConnectTests/DataFrameTests.swift
@@ -472,6 +472,15 @@ struct DataFrameTests {
     let spark = try await SparkSession.builder.getOrCreate()
     #expect(try await spark.range(1).toDF().columns == ["id"])
     #expect(try await spark.range(1).toDF("id").columns == ["id"])
+    #expect(try await spark.range(1).toDF("x").columns == ["x"])
+
+    // Renames the columns by position instead of selecting them by name.
+    let df = try await spark.sql("SELECT 1 a, 2 b")
+    #expect(try await df.toDF("b", "a").columns == ["b", "a"])
+    #expect(try await df.toDF("b", "a").collect() == [Row(1, 2)])
+    try await #require(throws: Error.self) {
+      try await df.toDF("a").columns
+    }
     await spark.stop()
   }
 


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to