dongjoon-hyun opened a new pull request, #466:
URL: https://github.com/apache/spark-connect-swift/pull/466
### What changes were proposed in this pull request?
This PR aims to support aggregate functions in a new
`AggregateFunctions.swift`, following the Scala client's `functions.scala`.
| Function | Description |
| --- | --- |
| `any_value` | Returns some value for a group of rows (with `ignoreNulls`
overload) |
| `approx_count_distinct` | Returns the approximate number of distinct items
(with `rsd` overload) |
| `bool_and` / `bool_or` | Returns true if all / at least one of the values
are true |
| `collect_list` / `collect_set` | Returns a list of objects with / without
duplicates |
| `corr` | Returns the Pearson Correlation Coefficient for two columns |
| `countDistinct` / `count_distinct` | Returns the number of distinct items
(via `isDistinct` flag) |
| `count_if` | Returns the number of `TRUE` values for the expression |
| `covar_pop` / `covar_samp` | Returns the population / sample covariance
for two columns |
| `first` / `last` | Returns the first / last value in a group (with
`ignoreNulls` overload) |
| `grouping` / `grouping_id` | Indicates whether a column is aggregated /
returns the level of grouping |
| `kurtosis` / `skewness` | Returns the kurtosis / skewness of the values in
a group |
| `max_by` / `min_by` | Returns the value associated with the maximum /
minimum of `ord` |
| `median` | Returns the median of the values in a group |
| `mode` | Returns the most frequent value (with `deterministic` overload) |
| `percentile_approx` | Returns the approximate percentile of the numeric
column |
| `stddev` / `stddev_samp` / `stddev_pop` | Returns the sample / population
standard deviation |
| `sumDistinct` / `sum_distinct` | Returns the sum of distinct values (via
`isDistinct` flag) |
| `variance` / `var_samp` / `var_pop` | Returns the unbiased / population
variance |
### Why are the changes needed?
Previously, only six basic aggregate functions (`count`, `sum`, `avg`,
`mean`, `min`, `max`) were available. This completes the aggregate function
category to improve feature parity with the Scala/PySpark clients.
```swift
df.groupBy("k").agg(countDistinct(col("v")), collect_list(col("v")),
median(col("v")))
```
### Does this PR introduce _any_ user-facing change?
No. This is a new addition to the unreleased functions API.
### How was this patch tested?
Pass the CIs with the newly added `AggregateFunctionsTests` test suite,
which verifies the generated expressions (function name, arguments,
`isDistinct` flag) and the query results of `groupBy().agg(...)`, global
aggregation via `select`, and `cube` with `grouping`/`grouping_id` against
Apache Spark 4.2.0.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Fable 5
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
