dongjoon-hyun opened a new pull request, #458:
URL: https://github.com/apache/spark-connect-swift/pull/458
### What changes were proposed in this pull request?
This PR aims to support `Column`-based overloads for `sort`, `orderBy`,
`sortWithinPartitions`, `withColumn`, `drop`, and `join` in `DataFrame`.
Previously, only `select`/`agg`/`filter`/`where`/`observe` accepted `Column`
expressions. This PR extends the remaining common entry points to accept
`Column`, matching the Scala client `Dataset` signatures:
- `sort(_ col: Column, _ cols: Column...)`, `orderBy(_ col: Column, _ cols:
Column...)`, `sortWithinPartitions(_ col: Column, _ cols: Column...)` — a
`SortOrder` expression (e.g., `col("id").desc()`) is used as-is; a plain column
is wrapped in ascending order with nulls first, like the Scala client's
`Column.sortOrder`.
- `withColumn(_ colName: String, _ col: Column)` — builds an `Alias`
expression for the `WithColumns` relation.
- `drop(_ col: Column, _ cols: Column...)` — uses the `columns` (expression)
field of the `Drop` relation.
- `join(_ right: DataFrame, joinExprs: Column)` and `join(_ right:
DataFrame, joinExprs: Column, joinType: String)` — passes the column expression
as the `join_condition`.
All existing `String`-based overloads are kept intact.
### Why are the changes needed?
To improve the usability of the `Column` API (`col`, `lit`, operators,
`asc`/`desc`) by allowing it in the main `DataFrame` transformation entry
points, for feature parity with the Scala/PySpark clients.
```swift
df.sort(col("department"), col("salary").desc())
df.withColumn("doubled", col("id") * 2)
df.drop(col("a"), col("b"))
df1.alias("l").join(df2.alias("r"), joinExprs: col("l.id") == col("r.id"),
joinType: "left")
```
### Does this PR introduce _any_ user-facing change?
Yes, this adds new APIs. The existing string-based APIs are unchanged.
### How was this patch tested?
Pass the CIs with newly added test cases (`sortWithColumn`,
`orderByWithColumn`, `sortWithinPartitionsWithColumn`, `withColumnWithColumn`,
`dropWithColumn`, `joinWithColumn`), verified against a local Spark Connect
server (`apache/spark:4.2.0`).
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Fable 5
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
