This is an automated email from the ASF dual-hosted git repository.
dongjoon-hyun pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/spark-connect-swift.git
The following commit(s) were added to refs/heads/main by this push:
new 14d1ddb [SPARK-58493] Support aggregate functions in
`AggregateFunctions`
14d1ddb is described below
commit 14d1ddb66d8116bfab4a0a19223984d134d8ddb6
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Fri Jul 31 20:49:58 2026 -0700
[SPARK-58493] Support aggregate functions in `AggregateFunctions`
### What changes were proposed in this pull request?
This PR aims to support aggregate functions in a new
`AggregateFunctions.swift`, following the Scala client's `functions.scala`.
| Function | Description |
| --- | --- |
| `any_value` | Returns some value for a group of rows (with `ignoreNulls`
overload) |
| `approx_count_distinct` | Returns the approximate number of distinct
items (with `rsd` overload) |
| `bool_and` / `bool_or` | Returns true if all / at least one of the values
are true |
| `collect_list` / `collect_set` | Returns a list of objects with / without
duplicates |
| `corr` | Returns the Pearson Correlation Coefficient for two columns |
| `countDistinct` / `count_distinct` | Returns the number of distinct items
(via `isDistinct` flag) |
| `count_if` | Returns the number of `TRUE` values for the expression |
| `covar_pop` / `covar_samp` | Returns the population / sample covariance
for two columns |
| `first` / `last` | Returns the first / last value in a group (with
`ignoreNulls` overload) |
| `grouping` / `grouping_id` | Indicates whether a column is aggregated /
returns the level of grouping |
| `kurtosis` / `skewness` | Returns the kurtosis / skewness of the values
in a group |
| `max_by` / `min_by` | Returns the value associated with the maximum /
minimum of `ord` |
| `median` | Returns the median of the values in a group |
| `mode` | Returns the most frequent value (with `deterministic` overload) |
| `percentile_approx` | Returns the approximate percentile of the numeric
column |
| `stddev` / `stddev_samp` / `stddev_pop` | Returns the sample / population
standard deviation |
| `sumDistinct` / `sum_distinct` | Returns the sum of distinct values (via
`isDistinct` flag) |
| `variance` / `var_samp` / `var_pop` | Returns the unbiased / population
variance |
### Why are the changes needed?
Previously, only six basic aggregate functions (`count`, `sum`, `avg`,
`mean`, `min`, `max`) were available. This completes the aggregate function
category to improve feature parity with the Scala/PySpark clients.
```swift
df.groupBy("k").agg(countDistinct(col("v")), collect_list(col("v")),
median(col("v")))
```
### Does this PR introduce _any_ user-facing change?
No. This is a new addition to the unreleased functions API.
### How was this patch tested?
Pass the CIs with the newly added `AggregateFunctionsTests` test suite,
which verifies the generated expressions (function name, arguments,
`isDistinct` flag) and the query results of `groupBy().agg(...)`, global
aggregation via `select`, and `cube` with `grouping`/`grouping_id` against
Apache Spark 4.2.0.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Fable 5
Closes #466 from dongjoon-hyun/SPARK-58493.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
---
Sources/SparkConnect/AggregateFunctions.swift | 319 +++++++++++++++++++++
.../AggregateFunctionsTests.swift | 209 ++++++++++++++
2 files changed, 528 insertions(+)
diff --git a/Sources/SparkConnect/AggregateFunctions.swift
b/Sources/SparkConnect/AggregateFunctions.swift
new file mode 100644
index 0000000..d065a48
--- /dev/null
+++ b/Sources/SparkConnect/AggregateFunctions.swift
@@ -0,0 +1,319 @@
+//
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+//
+
+/// Returns some value of the column for a group of rows.
+/// - Parameter col: A ``Column`` to aggregate.
+/// - Returns: A ``Column``.
+public func any_value(_ col: Column) -> Column {
+ return fn("any_value", col)
+}
+
+/// Returns some value of the column for a group of rows.
+/// If `ignoreNulls` is true, returns only non-null values.
+/// - Parameters:
+/// - col: A ``Column`` to aggregate.
+/// - ignoreNulls: A ``Column`` that evaluates to a boolean. Must be a
constant.
+/// - Returns: A ``Column``.
+public func any_value(_ col: Column, _ ignoreNulls: Column) -> Column {
+ return fn("any_value", col, ignoreNulls)
+}
+
+/// Returns the approximate number of distinct items in a group.
+/// - Parameter col: A ``Column`` to aggregate.
+/// - Returns: A ``Column``.
+public func approx_count_distinct(_ col: Column) -> Column {
+ return fn("approx_count_distinct", col)
+}
+
+/// Returns the approximate number of distinct items in a group.
+/// - Parameters:
+/// - col: A ``Column`` to aggregate.
+/// - rsd: The maximum relative standard deviation allowed (default = 0.05).
+/// - Returns: A ``Column``.
+public func approx_count_distinct(_ col: Column, _ rsd: Double) -> Column {
+ return fn("approx_count_distinct", col, lit(rsd))
+}
+
+/// Returns true if all values of the column are true.
+/// - Parameter col: A ``Column`` that evaluates to a boolean.
+/// - Returns: A ``Column``.
+public func bool_and(_ col: Column) -> Column {
+ return fn("bool_and", col)
+}
+
+/// Returns true if at least one value of the column is true.
+/// - Parameter col: A ``Column`` that evaluates to a boolean.
+/// - Returns: A ``Column``.
+public func bool_or(_ col: Column) -> Column {
+ return fn("bool_or", col)
+}
+
+/// Returns a list of objects with duplicates.
+/// - Parameter col: A ``Column`` to aggregate.
+/// - Returns: A ``Column``.
+public func collect_list(_ col: Column) -> Column {
+ return fn("collect_list", col)
+}
+
+/// Returns a set of objects with duplicate elements eliminated.
+/// - Parameter col: A ``Column`` to aggregate.
+/// - Returns: A ``Column``.
+public func collect_set(_ col: Column) -> Column {
+ return fn("collect_set", col)
+}
+
+/// Returns the Pearson Correlation Coefficient for two columns.
+/// - Parameters:
+/// - column1: A ``Column``.
+/// - column2: A ``Column``.
+/// - Returns: A ``Column``.
+public func corr(_ column1: Column, _ column2: Column) -> Column {
+ return fn("corr", column1, column2)
+}
+
+/// Returns the number of distinct items in a group.
+/// This is an alias of ``count_distinct(_:_:)``.
+/// - Parameters:
+/// - expr: A ``Column`` to count.
+/// - exprs: Additional ``Column``s to count.
+/// - Returns: A ``Column``.
+public func countDistinct(_ expr: Column, _ exprs: Column...) -> Column {
+ return fn("count", [expr] + exprs, isDistinct: true)
+}
+
+/// Returns the number of distinct items in a group.
+/// - Parameters:
+/// - expr: A ``Column`` to count.
+/// - exprs: Additional ``Column``s to count.
+/// - Returns: A ``Column``.
+public func count_distinct(_ expr: Column, _ exprs: Column...) -> Column {
+ return fn("count", [expr] + exprs, isDistinct: true)
+}
+
+/// Returns the number of `TRUE` values for the expression.
+/// - Parameter col: A ``Column`` that evaluates to a boolean.
+/// - Returns: A ``Column``.
+public func count_if(_ col: Column) -> Column {
+ return fn("count_if", col)
+}
+
+/// Returns the population covariance for two columns.
+/// - Parameters:
+/// - column1: A ``Column``.
+/// - column2: A ``Column``.
+/// - Returns: A ``Column``.
+public func covar_pop(_ column1: Column, _ column2: Column) -> Column {
+ return fn("covar_pop", column1, column2)
+}
+
+/// Returns the sample covariance for two columns.
+/// - Parameters:
+/// - column1: A ``Column``.
+/// - column2: A ``Column``.
+/// - Returns: A ``Column``.
+public func covar_samp(_ column1: Column, _ column2: Column) -> Column {
+ return fn("covar_samp", column1, column2)
+}
+
+/// Returns the first value in a group.
+///
+/// The function by default returns the first values it sees. It will return
the first non-null
+/// value it sees when `ignoreNulls` is set to true. If all values are null,
then null is returned.
+/// - Parameter col: A ``Column`` to aggregate.
+/// - Returns: A ``Column``.
+public func first(_ col: Column) -> Column {
+ return first(col, false)
+}
+
+/// Returns the first value in a group.
+///
+/// The function by default returns the first values it sees. It will return
the first non-null
+/// value it sees when `ignoreNulls` is set to true. If all values are null,
then null is returned.
+/// - Parameters:
+/// - col: A ``Column`` to aggregate.
+/// - ignoreNulls: Whether to skip null values.
+/// - Returns: A ``Column``.
+public func first(_ col: Column, _ ignoreNulls: Bool) -> Column {
+ return fn("first", col, lit(ignoreNulls))
+}
+
+/// Indicates whether a specified column in a GROUP BY list is aggregated or
not,
+/// returns 1 for aggregated or 0 for not aggregated in the result set.
+/// - Parameter col: A ``Column`` to check.
+/// - Returns: A ``Column``.
+public func grouping(_ col: Column) -> Column {
+ return fn("grouping", col)
+}
+
+/// Returns the level of grouping.
+/// - Parameter cols: ``Column``s to check.
+/// - Returns: A ``Column``.
+public func grouping_id(_ cols: Column...) -> Column {
+ return fn("grouping_id", cols)
+}
+
+/// Returns the kurtosis of the values in a group.
+/// - Parameter col: A ``Column`` to aggregate.
+/// - Returns: A ``Column``.
+public func kurtosis(_ col: Column) -> Column {
+ return fn("kurtosis", col)
+}
+
+/// Returns the last value in a group.
+///
+/// The function by default returns the last values it sees. It will return
the last non-null
+/// value it sees when `ignoreNulls` is set to true. If all values are null,
then null is returned.
+/// - Parameter col: A ``Column`` to aggregate.
+/// - Returns: A ``Column``.
+public func last(_ col: Column) -> Column {
+ return last(col, false)
+}
+
+/// Returns the last value in a group.
+///
+/// The function by default returns the last values it sees. It will return
the last non-null
+/// value it sees when `ignoreNulls` is set to true. If all values are null,
then null is returned.
+/// - Parameters:
+/// - col: A ``Column`` to aggregate.
+/// - ignoreNulls: Whether to skip null values.
+/// - Returns: A ``Column``.
+public func last(_ col: Column, _ ignoreNulls: Bool) -> Column {
+ return fn("last", col, lit(ignoreNulls))
+}
+
+/// Returns the value associated with the maximum value of `ord`.
+/// - Parameters:
+/// - col: A ``Column`` to return the value from.
+/// - ord: A ``Column`` to be maximized.
+/// - Returns: A ``Column``.
+public func max_by(_ col: Column, _ ord: Column) -> Column {
+ return fn("max_by", col, ord)
+}
+
+/// Returns the median of the values in a group.
+/// - Parameter col: A ``Column`` to aggregate.
+/// - Returns: A ``Column``.
+public func median(_ col: Column) -> Column {
+ return fn("median", col)
+}
+
+/// Returns the value associated with the minimum value of `ord`.
+/// - Parameters:
+/// - col: A ``Column`` to return the value from.
+/// - ord: A ``Column`` to be minimized.
+/// - Returns: A ``Column``.
+public func min_by(_ col: Column, _ ord: Column) -> Column {
+ return fn("min_by", col, ord)
+}
+
+/// Returns the most frequent value in a group.
+/// - Parameter col: A ``Column`` to aggregate.
+/// - Returns: A ``Column``.
+public func mode(_ col: Column) -> Column {
+ return fn("mode", col)
+}
+
+/// Returns the most frequent value in a group.
+/// - Parameters:
+/// - col: A ``Column`` to aggregate.
+/// - deterministic: If there are multiple equally-frequent results,
+/// whether to return the lowest (defined by min-hash) one.
+/// - Returns: A ``Column``.
+public func mode(_ col: Column, _ deterministic: Bool) -> Column {
+ return fn("mode", col, lit(deterministic))
+}
+
+/// Returns the approximate `percentile` of the numeric column `col` which is
the smallest value
+/// in the ordered `col` values (sorted from least to greatest) such that no
more than `percentage`
+/// of `col` values is less than the value or equal to that value.
+/// - Parameters:
+/// - col: A ``Column`` to aggregate.
+/// - percentage: A percentage ``Column``. Each value must be between 0.0
and 1.0.
+/// - accuracy: A positive numeric literal ``Column`` that controls
approximation accuracy
+/// at the cost of memory.
+/// - Returns: A ``Column``.
+public func percentile_approx(_ col: Column, _ percentage: Column, _ accuracy:
Column) -> Column {
+ return fn("percentile_approx", col, percentage, accuracy)
+}
+
+/// Returns the skewness of the values in a group.
+/// - Parameter col: A ``Column`` to aggregate.
+/// - Returns: A ``Column``.
+public func skewness(_ col: Column) -> Column {
+ return fn("skewness", col)
+}
+
+/// Returns the sample standard deviation of the expression in a group.
+/// This is an alias of ``stddev_samp(_:)``.
+/// - Parameter col: A ``Column`` to aggregate.
+/// - Returns: A ``Column``.
+public func stddev(_ col: Column) -> Column {
+ return fn("stddev", col)
+}
+
+/// Returns the population standard deviation of the expression in a group.
+/// - Parameter col: A ``Column`` to aggregate.
+/// - Returns: A ``Column``.
+public func stddev_pop(_ col: Column) -> Column {
+ return fn("stddev_pop", col)
+}
+
+/// Returns the sample standard deviation of the expression in a group.
+/// - Parameter col: A ``Column`` to aggregate.
+/// - Returns: A ``Column``.
+public func stddev_samp(_ col: Column) -> Column {
+ return fn("stddev_samp", col)
+}
+
+/// Returns the sum of distinct values in the expression.
+/// This is an alias of ``sum_distinct(_:)``.
+/// - Parameter col: A ``Column`` to aggregate.
+/// - Returns: A ``Column``.
+public func sumDistinct(_ col: Column) -> Column {
+ return sum_distinct(col)
+}
+
+/// Returns the sum of distinct values in the expression.
+/// - Parameter col: A ``Column`` to aggregate.
+/// - Returns: A ``Column``.
+public func sum_distinct(_ col: Column) -> Column {
+ return fn("sum", [col], isDistinct: true)
+}
+
+/// Returns the population variance of the values in a group.
+/// - Parameter col: A ``Column`` to aggregate.
+/// - Returns: A ``Column``.
+public func var_pop(_ col: Column) -> Column {
+ return fn("var_pop", col)
+}
+
+/// Returns the unbiased variance of the values in a group.
+/// - Parameter col: A ``Column`` to aggregate.
+/// - Returns: A ``Column``.
+public func var_samp(_ col: Column) -> Column {
+ return fn("var_samp", col)
+}
+
+/// Returns the unbiased variance of the values in a group.
+/// This is an alias of ``var_samp(_:)``.
+/// - Parameter col: A ``Column`` to aggregate.
+/// - Returns: A ``Column``.
+public func variance(_ col: Column) -> Column {
+ return fn("variance", col)
+}
diff --git a/Tests/SparkConnectTests/AggregateFunctionsTests.swift
b/Tests/SparkConnectTests/AggregateFunctionsTests.swift
new file mode 100644
index 0000000..3ac536a
--- /dev/null
+++ b/Tests/SparkConnectTests/AggregateFunctionsTests.swift
@@ -0,0 +1,209 @@
+//
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+//
+
+import Testing
+
+@testable import SparkConnect
+
+/// A test suite for `AggregateFunctions`
+@Suite(.serialized)
+struct AggregateFunctionsTests {
+
+ @Test
+ func aggregateFunctions() throws {
+ for (column, name) in [
+ (any_value(col("a")), "any_value"),
+ (approx_count_distinct(col("a")), "approx_count_distinct"),
+ (bool_and(col("a")), "bool_and"),
+ (bool_or(col("a")), "bool_or"),
+ (collect_list(col("a")), "collect_list"),
+ (collect_set(col("a")), "collect_set"),
+ (count_if(col("a")), "count_if"),
+ (grouping(col("a")), "grouping"),
+ (grouping_id(col("a")), "grouping_id"),
+ (kurtosis(col("a")), "kurtosis"),
+ (median(col("a")), "median"),
+ (mode(col("a")), "mode"),
+ (skewness(col("a")), "skewness"),
+ (stddev(col("a")), "stddev"),
+ (stddev_pop(col("a")), "stddev_pop"),
+ (stddev_samp(col("a")), "stddev_samp"),
+ (var_pop(col("a")), "var_pop"),
+ (var_samp(col("a")), "var_samp"),
+ (variance(col("a")), "variance"),
+ ] {
+ let expr = column.expr
+ #expect(expr.unresolvedFunction.functionName == name)
+ #expect(expr.unresolvedFunction.isDistinct == false)
+
#expect(expr.unresolvedFunction.arguments[0].unresolvedAttribute.unparsedIdentifier
== "a")
+ }
+ }
+
+ @Test
+ func aggregateFunctionArguments() throws {
+ for (column, name) in [
+ (any_value(col("a"), lit(true)), "any_value"),
+ (corr(col("a"), col("b")), "corr"),
+ (covar_pop(col("a"), col("b")), "covar_pop"),
+ (covar_samp(col("a"), col("b")), "covar_samp"),
+ (max_by(col("a"), col("b")), "max_by"),
+ (min_by(col("a"), col("b")), "min_by"),
+ ] {
+ let expr = column.expr
+ #expect(expr.unresolvedFunction.functionName == name)
+ #expect(expr.unresolvedFunction.arguments.count == 2)
+
#expect(expr.unresolvedFunction.arguments[0].unresolvedAttribute.unparsedIdentifier
== "a")
+ }
+
+ let rsd = approx_count_distinct(col("a"), 0.01).expr
+ #expect(rsd.unresolvedFunction.functionName == "approx_count_distinct")
+ #expect(rsd.unresolvedFunction.arguments[1].literal.double == 0.01)
+
+ let deterministic = mode(col("a"), true).expr
+ #expect(deterministic.unresolvedFunction.functionName == "mode")
+ #expect(deterministic.unresolvedFunction.arguments[1].literal.boolean ==
true)
+
+ let percentile = percentile_approx(col("a"), lit(0.5), lit(100)).expr
+ #expect(percentile.unresolvedFunction.functionName == "percentile_approx")
+ #expect(percentile.unresolvedFunction.arguments.count == 3)
+
+ #expect(grouping_id(col("a"),
col("b")).expr.unresolvedFunction.arguments.count == 2)
+ }
+
+ @Test
+ func firstAndLast() throws {
+ for (column, name, ignoreNulls) in [
+ (first(col("a")), "first", false),
+ (first(col("a"), true), "first", true),
+ (last(col("a")), "last", false),
+ (last(col("a"), true), "last", true),
+ ] {
+ let expr = column.expr
+ #expect(expr.unresolvedFunction.functionName == name)
+ #expect(expr.unresolvedFunction.arguments.count == 2)
+
#expect(expr.unresolvedFunction.arguments[0].unresolvedAttribute.unparsedIdentifier
== "a")
+ #expect(expr.unresolvedFunction.arguments[1].literal.boolean ==
ignoreNulls)
+ }
+ }
+
+ @Test
+ func distinctAggregateFunctions() throws {
+ for (column, name, count) in [
+ (countDistinct(col("a")), "count", 1),
+ (countDistinct(col("a"), col("b")), "count", 2),
+ (count_distinct(col("a")), "count", 1),
+ (count_distinct(col("a"), col("b")), "count", 2),
+ (sumDistinct(col("a")), "sum", 1),
+ (sum_distinct(col("a")), "sum", 1),
+ ] {
+ let expr = column.expr
+ #expect(expr.unresolvedFunction.functionName == name)
+ #expect(expr.unresolvedFunction.isDistinct)
+ #expect(expr.unresolvedFunction.arguments.count == count)
+
#expect(expr.unresolvedFunction.arguments[0].unresolvedAttribute.unparsedIdentifier
== "a")
+ }
+ }
+
+ @Test
+ func groupByAggregateFunctions() async throws {
+ let spark = try await SparkSession.builder.getOrCreate()
+ let df = try await spark.sql(
+ "SELECT * FROM VALUES ('a', 1), ('a', 1), ('a', 2), ('b', 3) T(k, v)")
+ let rows = try await df.groupBy("k")
+ .agg(
+ countDistinct(col("v")), sum_distinct(col("v")), median(col("v")),
mode(col("v")),
+ any_value(col("v"))
+ ).orderBy("k").collect()
+ #expect(rows == [Row("a", 2, 3, 1.0, 1, 1), Row("b", 1, 3, 3.0, 3, 3)])
+
+ let collected = try await df.groupBy("k")
+ .agg(
+ sort_array(collect_list(col("v"))).cast("string"),
+ sort_array(collect_set(col("v"))).cast("string")
+ ).orderBy("k").collect()
+ #expect(collected == [Row("a", "[1, 1, 2]", "[1, 2]"), Row("b", "[3]",
"[3]")])
+ await spark.stop()
+ }
+
+ @Test
+ func selectStatisticalFunctions() async throws {
+ let spark = try await SparkSession.builder.getOrCreate()
+ let df = try await spark.sql(
+ "SELECT * FROM VALUES (double(1.0), double(10.0)), (double(3.0),
double(20.0)) T(v, w)")
+ let variances = try await df.select(
+ variance(col("v")), var_samp(col("v")), var_pop(col("v")),
+ stddev(col("v")), stddev_samp(col("v")), stddev_pop(col("v"))
+ ).collect()
+ #expect(
+ variances == [Row(2.0, 2.0, 1.0, 1.4142135623730951, 1.4142135623730951,
1.0)])
+
+ let moments = try await df.select(skewness(col("v")),
kurtosis(col("v"))).collect()
+ #expect(moments == [Row(0.0, -2.0)])
+
+ let covariances = try await df.select(
+ corr(col("v"), col("w")), covar_samp(col("v"), col("w")),
covar_pop(col("v"), col("w"))
+ ).collect()
+ #expect(covariances == [Row(1.0, 10.0, 5.0)])
+
+ let percentiles = try await df.select(
+ median(col("v")), percentile_approx(col("v"), lit(0.5), lit(100))
+ ).collect()
+ #expect(percentiles == [Row(2.0, 1.0)])
+ await spark.stop()
+ }
+
+ @Test
+ func selectFirstLastFunctions() async throws {
+ let spark = try await SparkSession.builder.getOrCreate()
+ let df = try await spark.sql("SELECT * FROM VALUES (NULL), (1), (2),
(NULL) T(v)")
+ let rows = try await df.select(
+ first(col("v")), first(col("v"), true), last(col("v")), last(col("v"),
true)
+ ).collect()
+ #expect(rows == [Row(nil, 1, nil, 2)])
+ await spark.stop()
+ }
+
+ @Test
+ func selectBooleanAndCountFunctions() async throws {
+ let spark = try await SparkSession.builder.getOrCreate()
+ let df = try await spark.sql("SELECT * FROM VALUES (true, 1), (false,
NULL) T(b, v)")
+ let rows = try await df.select(
+ bool_and(col("b")), bool_or(col("b")), count_if(col("b")),
+ countDistinct(col("v")), approx_count_distinct(col("v")),
+ approx_count_distinct(col("v"), 0.01)
+ ).collect()
+ #expect(rows == [Row(false, true, 1, 1, 1, 1)])
+
+ let ordered = try await spark.sql("SELECT * FROM VALUES ('a', 1), ('b', 2)
T(k, v)")
+ .select(max_by(col("k"), col("v")), min_by(col("k"), col("v"))).collect()
+ #expect(ordered == [Row("b", "a")])
+ await spark.stop()
+ }
+
+ @Test
+ func selectGroupingFunctions() async throws {
+ let spark = try await SparkSession.builder.getOrCreate()
+ let df = try await spark.sql("SELECT * FROM VALUES ('a', 1), ('b', 2) T(k,
v)")
+ let rows = try await df.cube("k")
+ .agg(grouping(col("k")), grouping_id(col("k")).alias("gid"))
+ .orderBy("gid", "k").collect()
+ #expect(rows == [Row("a", 0, 0), Row("b", 0, 0), Row(nil, 1, 1)])
+ await spark.stop()
+ }
+}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]