dongjoon-hyun opened a new pull request, #521:
URL: https://github.com/apache/spark-connect-swift/pull/521
### What changes were proposed in this pull request?
This PR aims to support 21 general aggregate functions in the Swift Spark
Connect
client by adding them to `Sources/SparkConnect/AggregateFunctions.swift`,
keeping
the file's alphabetical order:
| Function | Since | Notes |
| --- | --- | --- |
| `approx_percentile(col, percentage, accuracy = 10000)` | 3.5.0 |
`percentage` is a `Double` or `[Double]` |
| `array_agg(col)` | 3.5.0 | Alias of `collect_list` |
| `bit_and(col)` | 3.5.0 | Bitwise `AND` of all non-null input values |
| `bit_or(col)` | 3.5.0 | Bitwise `OR` of all non-null input values |
| `bit_xor(col)` | 3.5.0 | Bitwise `XOR` of all non-null input values |
| `collect_union(col)` | 4.3.0 | Distinct union of array elements across
rows |
| `count_min_sketch(col, eps, confidence, seed?)` | 3.5.0 | Returns a binary
sketch |
| `every(col)` | 3.5.0 | Alias of `bool_and` |
| `first_value(col, ignoreNulls?)` | 3.5.0 | |
| `histogram_numeric(col, nBins)` | 3.5.0 | |
| `last_value(col, ignoreNulls?)` | 3.5.0 | |
| `listagg(col, delimiter?)` | 4.0.0 | |
| `listagg_distinct(col, delimiter?)` | 4.0.0 | |
| `percentile(col, percentage, frequency = 1)` | 3.5.0 | `percentage` is a
`Double` or `[Double]` |
| `product(col)` | 3.2.0 | |
| `some(col)` | 3.5.0 | Alias of `bool_or` |
| `std(col)` | 3.5.0 | Alias of `stddev_samp` |
| `string_agg(col, delimiter?)` | 4.0.0 | Alias of `listagg` |
| `string_agg_distinct(col, delimiter?)` | 4.0.0 | Alias of
`listagg_distinct` |
| `try_avg(col)` | 3.5.0 | `null` on overflow |
| `try_sum(col)` | 3.5.0 | `null` on overflow |
All of these belong to the upstream "Aggregate Functions" documentation
group,
including `bit_and`/`bit_or`/`bit_xor` (which are aggregates, unlike the
scalar
bitwise functions in `BitwiseFunctions.swift`) and `try_avg`/`try_sum`.
Three details are worth calling out, since the naive mapping does not work:
1. `listagg_distinct` and `string_agg_distinct` are **not** separate function
names on the server. `FunctionRegistry` registers only `listagg` and
`string_agg`, and both `sql/api` `functions.scala`
(`Column.fn("listagg", isDistinct = true, e)`) and the Python Spark
Connect
client (`UnresolvedFunction("listagg", _exprs, is_distinct=True)`) send
the
base name with the `is_distinct` flag set. This PR follows the same
pattern
already used by `count_distinct` and `sum_distinct`. Sending
`listagg_distinct` as a function name fails with `UNRESOLVED_ROUTINE`.
2. `count_min_sketch` always requires four arguments:
`CountMinSketchAggExpressionBuilder` declares `seed` without a default,
so a
three-argument call fails with `REQUIRED_PARAMETER_NOT_FOUND`. The
overload
without a seed therefore generates a random `Int64` seed on the client,
which
is what `functions.count_min_sketch(e, eps, confidence)` does in Scala
with
`lit(SparkClassUtils.random.nextLong)`.
3. `approx_percentile` is a distinct registered function from the existing
`percentile_approx`, so it is added rather than aliased.
For the array form of `percentage`, the `[Double]` overloads build an
`array(...)`
of literals, which stays foldable and satisfies the analyzer's requirement
that
the percentage expression be a constant.
### Why are the changes needed?
To improve API coverage and feature parity with PySpark and Spark SQL. These
aggregate functions were entirely missing from the Swift client, including
the
SQL-standard aliases (`array_agg`, `std`, `every`, `some`, `first_value`,
`last_value`) that users coming from other SQL engines are most likely to
reach
for first.
### Does this PR introduce _any_ user-facing change?
No, this is an additive change that introduces new APIs.
```swift
let df = try await spark.sql("SELECT * FROM VALUES (1), (2), (3) T(v)")
try await df.select(bit_and(col("v")), bit_or(col("v")),
bit_xor(col("v"))).show()
try await df.select(percentile(col("v"), 0.5), approx_percentile(col("v"),
[0.0, 1.0])).show()
try await df.select(try_avg(col("v")), try_sum(col("v")),
product(col("v"))).show()
```
Non-`Column` arguments such as `accuracy`, `frequency`, `nBins`,
`delimiter`, and
`seed` are taken as Swift primitives and wrapped with `lit(...)`, following
the
existing convention (e.g. `approx_count_distinct(_:_:)` and
`regexp_extract(_:_:_:)`). Optional arguments are expressed as overloads,
except
`accuracy` and `frequency`, which use their upstream default values.
### How was this patch tested?
Pass the CIs with new test cases added to `AggregateFunctionsTests`.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
