dongjoon-hyun opened a new pull request, #539:
URL: https://github.com/apache/spark-connect-swift/pull/539
### What changes were proposed in this pull request?
This PR adds support for lambda expressions and the higher-order array
functions that consume them, in a new `HigherOrderFunctions.swift` file. No
existing file is modified.
Lambda infrastructure (internal):
- `createLambda` in one-, two-, and three-argument forms, converting a Swift
closure into a `LambdaFunction` expression.
- Lambda variables get a unique name (`x_1`, `y_2`, ...) from an
`Atomic<Int>` counter, matching Scala's
`UnresolvedNamedLambdaVariable.apply` and PySpark's `fresh_var_name`.
New public functions:
| Function | Closure |
| -------- | ------- |
| `transform(col, f)` | `(Column) -> Column` and `(Column, Column) ->
Column` (element, index) |
| `filter(col, f)` | `(Column) -> Column` and `(Column, Column) -> Column`
(element, index) |
| `exists(col, f)` | `(Column) -> Column` |
| `forall(col, f)` | `(Column) -> Column` |
| `aggregate(col, initialValue, merge[, finish])` | `(Column, Column) ->
Column`, `(Column) -> Column` |
| `reduce(col, initialValue, merge[, finish])` | `(Column, Column) ->
Column`, `(Column) -> Column` |
| `zip_with(left, right, f)` | `(Column, Column) -> Column` |
| `array_sort(col, comparator)` | `(Column, Column) -> Column` |
The closures are non-escaping because they are invoked immediately while the
expression is built and never stored.
### Why are the changes needed?
For feature parity with Apache Spark. These functions exist in both the Scala
and Python clients but had no Swift equivalent, because there was no way to
build a Spark Connect lambda expression from a Swift closure. The generated
protobuf types (`Expression.LambdaFunction` and
`Expression.UnresolvedNamedLambdaVariable`) were already available, so no
proto
regeneration is required.
The unique variable naming is required for correctness, not cosmetics. The
server passes `name_parts` through unchanged
(`SparkConnectPlanner.transformUnresolvedNamedLambdaVariable`), so with fixed
names an inner lambda would shadow the variable of an enclosing one and
`transform(a) { x in transform(b) { y in x + y } }` would silently compute
`x + x`.
The functions live in their own file rather than in
`CollectionFunctions.swift`
because they share the lambda infrastructure, which is kept private to them,
and because the follow-up higher-order map functions (`map_filter`,
`transform_keys`, `transform_values`, `map_zip_with`) belong next to them.
This
also keeps `CollectionFunctions.swift` from growing past 900 lines.
### Does this PR introduce _any_ user-facing change?
Yes, this adds new public APIs. For example:
```swift
let df = try await spark.range(1)
let arr = array(lit(1), lit(2), lit(3))
try await df.select(transform(arr) { $0 * 2 }.cast("string")).show()
// [2, 4, 6]
try await df.select(aggregate(arr, lit(0)) { acc, x in acc + x }).show()
// 6
```
All of these functions were introduced in Spark 3.5 or earlier
(`array_sort` with a comparator in 3.4.0), so no version gate is needed.
### How was this patch tested?
Added a new `HigherOrderFunctionsTests` suite.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
