dongjoon-hyun opened a new pull request, #548:
URL: https://github.com/apache/spark-connect-swift/pull/548
### What changes were proposed in this pull request?
This PR aims to support `col` and subscript column accessors in `DataFrame`.
```swift
public nonisolated func col(_ colName: String) -> Column
public nonisolated subscript(_ colName: String) -> Column
```
Unlike the global `col(_:)` function, the returned `Column` carries the plan
ID of the `DataFrame` (assigned since SPARK-59414), so the server can tell
apart the columns with the same name in multiple `DataFrame`s.
```swift
let df = try await spark.range(3)
let df1 = await df.filter("id > 0")
let df2 = await df.select("id")
let joined = await df1.join(df2, joinExprs: df1["id"] == df2["id"])
try await joined.select(df1["id"], df2.col("id")).show()
```
- `"*"` becomes `UnresolvedStar` with `plan_id`.
- A name ending with `".*"`, e.g. `"s.*"`, becomes `UnresolvedStar` with
`unparsed_target` only, like the global `col(_:)`. The server rejects a star
with both a target and a plan ID
(`CONNECT_INVALID_PLAN.UNRESOLVED_STAR_WITH_BOTH_TARGET_AND_PLAN_ID`).
- Other names become `UnresolvedAttribute` with `plan_id`.
The plan ID of the root relation is stored as an immutable property when a
`DataFrame` is created, so both accessors are `nonisolated` and don't require
`await`.
### Why are the changes needed?
To resolve ambiguous column references like `Dataset.col` and
`Dataset.apply` in Scala and `DataFrame.__getitem__` in PySpark. Spark Connect
has supported this since Apache Spark
[v3.4.0](https://github.com/apache/spark/releases/tag/v3.4.0) (2023-04-07) via
SPARK-41812 and SPARK-41823. No version gating is needed for the supported
Apache Spark 4.x servers.
### Does this PR introduce _any_ user-facing change?
No, this adds only new public APIs, `DataFrame.col` and
`DataFrame.subscript`.
### How was this patch tested?
Pass the CIs with the newly added test cases.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
