This is an automated email from the ASF dual-hosted git repository.
dongjoon-hyun pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/spark-connect-swift.git
The following commit(s) were added to refs/heads/main by this push:
new a8d5ff5 [SPARK-60516] Fix `DataFrame.toDF` to rename columns instead
of selecting them
a8d5ff5 is described below
commit a8d5ff58d657dd08621801a44acadac247e18143
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Fri Oct 23 10:05:17 2026 -0700
[SPARK-60516] Fix `DataFrame.toDF` to rename columns instead of selecting
them
### What changes were proposed in this pull request?
This PR aims to fix `DataFrame.toDF(_:)` to rename the columns by position
with the `ToDF` relation, like the Scala and PySpark Spark Connect clients.
```swift
- DataFrame(spark: self.spark, plan:
SparkConnectClient.getProject(self.plan.root, cols))
+ DataFrame(spark: self.spark, plan:
SparkConnectClient.getToDF(self.plan.root, cols))
```
`toDF()` without column names still returns the `DataFrame` as is, like
`Dataset.toDF()` in Scala.
### Why are the changes needed?
`toDF` was added for feature parity in SPARK-51995
([#111](https://github.com/apache/spark-connect-swift/pull/111)), but it sends
a `Project` relation with the given names, which is the same as `select(_:)`.
Its documentation was also copied from `select(_:)`.
In Scala and PySpark, `toDF` renames all columns by position, and both
Spark Connect clients send the `ToDF` relation.
- [Apache Spark 4.2.0 (2026-07-11)
`Dataset.toDF`](https://github.com/apache/spark/blob/v4.2.0/sql/connect/common/src/main/scala/org/apache/spark/sql/connect/Dataset.scala#L186-L190)
```scala
def toDF(colNames: String*): DataFrame = sparkSession.newDataFrame {
builder =>
builder.getToDfBuilder
.setInput(plan.getRoot)
.addAllColumnNames(colNames.asJava)
}
```
- [Apache Spark 4.2.0 (2026-07-11) PySpark
`DataFrame.toDF`](https://github.com/apache/spark/blob/v4.2.0/python/pyspark/sql/connect/dataframe.py#L2080-L2091)
```python
return DataFrame(plan.ToDF(self._plan, list(cols)), self._session)
```
As a result, the same code behaves differently in the Swift client.
`spark.range(1).toDF("x")` fails only in Swift, and `toDF("b", "a")` on the
columns `a` and `b` swaps the values without an error.
### Does this PR introduce _any_ user-facing change?
Yes. `toDF` renames the columns by position like Scala and PySpark. This is
a user-facing change compared to all released versions from [v0.1.0
(2025-05-04)](https://github.com/apache/spark-connect-swift/releases/tag/v0.1.0)
to [0.7.0
(2026-07-22)](https://github.com/apache/spark-connect-swift/releases/tag/0.7.0).
The number of column names must be the same as the number of columns. To
select or reorder columns by name, use `select(_:)` instead. `toDF()` and
`toDF` with all column names in the same order return the same results as
before.
**BEFORE**
```swift
let df = try await spark.sql("SELECT 1 a, 2 b")
try await spark.range(1).toDF("x").columns // throws
[UNRESOLVED_COLUMN.WITH_SUGGESTION]
try await df.toDF("b", "a").collect() // [Row(2, 1)]
try await df.toDF("a").columns // ["a"]
```
**AFTER**
```swift
let df = try await spark.sql("SELECT 1 a, 2 b")
try await spark.range(1).toDF("x").columns // ["x"]
try await df.toDF("b", "a").collect() // [Row(1, 2)]
try await df.toDF("a").columns // throws
[ASSIGNMENT_ARITY_MISMATCH]
```
### How was this patch tested?
Pass the CIs with the updated test case.
I also tested manually with Apache Spark 4.2.0 and 4.0.4 Spark Connect
servers. The Scala Spark Connect client of Apache Spark 4.2.0 returns the same
results as the AFTER example.
```
$ swift test --no-parallel --filter "DataFrameTests|DataFrameInternalTests"
...
Test run with 178 tests in 3 suites passed after 31.553 seconds.
```
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
Closes #622 from dongjoon-hyun/SPARK-60516.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
---
Sources/SparkConnect/DataFrame+Transformations.swift | 17 +++++++++++++----
Sources/SparkConnect/SparkConnectClient.swift | 7 +++++++
Tests/SparkConnectTests/DataFrameTests.swift | 9 +++++++++
3 files changed, 29 insertions(+), 4 deletions(-)
diff --git a/Sources/SparkConnect/DataFrame+Transformations.swift
b/Sources/SparkConnect/DataFrame+Transformations.swift
index 14a2973..83222a4 100644
--- a/Sources/SparkConnect/DataFrame+Transformations.swift
+++ b/Sources/SparkConnect/DataFrame+Transformations.swift
@@ -123,15 +123,24 @@ extension DataFrame {
return Column(expr)
}
- /// Selects a subset of existing columns using column names.
- /// - Parameter cols: Column names
- /// - Returns: A ``DataFrame`` with subset of columns.
+ /// Returns a new ``DataFrame`` with the columns renamed by position.
+ ///
+ /// The number of column names must be the same as the number of columns.
Otherwise, an error is
+ /// thrown when the returned ``DataFrame`` is analyzed or executed. To
select or reorder columns
+ /// by name, use ``select(_:)``.
+ ///
+ /// ```swift
+ /// let df = try await spark.sql("SELECT 1 a, 2 b")
+ /// try await df.toDF("x", "y").columns // ["x", "y"]
+ /// ```
+ /// - Parameter cols: New column names. If empty, the columns are not
renamed.
+ /// - Returns: A ``DataFrame`` with the new column names.
public func toDF(_ cols: String...) -> DataFrame {
let df =
if cols.isEmpty {
DataFrame(spark: self.spark, plan: self.plan)
} else {
- DataFrame(spark: self.spark, plan:
SparkConnectClient.getProject(self.plan.root, cols))
+ DataFrame(spark: self.spark, plan:
SparkConnectClient.getToDF(self.plan.root, cols))
}
return df
}
diff --git a/Sources/SparkConnect/SparkConnectClient.swift
b/Sources/SparkConnect/SparkConnectClient.swift
index 4bbbfb0..62d71cf 100644
--- a/Sources/SparkConnect/SparkConnectClient.swift
+++ b/Sources/SparkConnect/SparkConnectClient.swift
@@ -723,6 +723,13 @@ public actor SparkConnectClient {
return createPlan { $0.toSchema = toSchema }
}
+ static func getToDF(_ child: Relation, _ columnNames: [String]) -> Plan {
+ var toDF = Spark_Connect_ToDF()
+ toDF.input = child
+ toDF.columnNames = columnNames
+ return createPlan { $0.toDf = toDF }
+ }
+
static func getProjectExprs(_ child: Relation, _ exprs: [String]) -> Plan {
var project = Project()
project.input = child
diff --git a/Tests/SparkConnectTests/DataFrameTests.swift
b/Tests/SparkConnectTests/DataFrameTests.swift
index 7710d7a..71a119d 100644
--- a/Tests/SparkConnectTests/DataFrameTests.swift
+++ b/Tests/SparkConnectTests/DataFrameTests.swift
@@ -472,6 +472,15 @@ struct DataFrameTests {
let spark = try await SparkSession.builder.getOrCreate()
#expect(try await spark.range(1).toDF().columns == ["id"])
#expect(try await spark.range(1).toDF("id").columns == ["id"])
+ #expect(try await spark.range(1).toDF("x").columns == ["x"])
+
+ // Renames the columns by position instead of selecting them by name.
+ let df = try await spark.sql("SELECT 1 a, 2 b")
+ #expect(try await df.toDF("b", "a").columns == ["b", "a"])
+ #expect(try await df.toDF("b", "a").collect() == [Row(1, 2)])
+ try await #require(throws: Error.self) {
+ try await df.toDF("a").columns
+ }
await spark.stop()
}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]