This is an automated email from the ASF dual-hosted git repository.

dongjoon-hyun pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/spark-connect-swift.git


The following commit(s) were added to refs/heads/main by this push:
     new 14d1ddb  [SPARK-58493] Support aggregate functions in 
`AggregateFunctions`
14d1ddb is described below

commit 14d1ddb66d8116bfab4a0a19223984d134d8ddb6
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Fri Jul 31 20:49:58 2026 -0700

    [SPARK-58493] Support aggregate functions in `AggregateFunctions`
    
    ### What changes were proposed in this pull request?
    
    This PR aims to support aggregate functions in a new 
`AggregateFunctions.swift`, following the Scala client's `functions.scala`.
    
    | Function | Description |
    | --- | --- |
    | `any_value` | Returns some value for a group of rows (with `ignoreNulls` 
overload) |
    | `approx_count_distinct` | Returns the approximate number of distinct 
items (with `rsd` overload) |
    | `bool_and` / `bool_or` | Returns true if all / at least one of the values 
are true |
    | `collect_list` / `collect_set` | Returns a list of objects with / without 
duplicates |
    | `corr` | Returns the Pearson Correlation Coefficient for two columns |
    | `countDistinct` / `count_distinct` | Returns the number of distinct items 
(via `isDistinct` flag) |
    | `count_if` | Returns the number of `TRUE` values for the expression |
    | `covar_pop` / `covar_samp` | Returns the population / sample covariance 
for two columns |
    | `first` / `last` | Returns the first / last value in a group (with 
`ignoreNulls` overload) |
    | `grouping` / `grouping_id` | Indicates whether a column is aggregated / 
returns the level of grouping |
    | `kurtosis` / `skewness` | Returns the kurtosis / skewness of the values 
in a group |
    | `max_by` / `min_by` | Returns the value associated with the maximum / 
minimum of `ord` |
    | `median` | Returns the median of the values in a group |
    | `mode` | Returns the most frequent value (with `deterministic` overload) |
    | `percentile_approx` | Returns the approximate percentile of the numeric 
column |
    | `stddev` / `stddev_samp` / `stddev_pop` | Returns the sample / population 
standard deviation |
    | `sumDistinct` / `sum_distinct` | Returns the sum of distinct values (via 
`isDistinct` flag) |
    | `variance` / `var_samp` / `var_pop` | Returns the unbiased / population 
variance |
    
    ### Why are the changes needed?
    
    Previously, only six basic aggregate functions (`count`, `sum`, `avg`, 
`mean`, `min`, `max`) were available. This completes the aggregate function 
category to improve feature parity with the Scala/PySpark clients.
    
    ```swift
    df.groupBy("k").agg(countDistinct(col("v")), collect_list(col("v")), 
median(col("v")))
    ```
    
    ### Does this PR introduce _any_ user-facing change?
    
    No. This is a new addition to the unreleased functions API.
    
    ### How was this patch tested?
    
    Pass the CIs with the newly added `AggregateFunctionsTests` test suite, 
which verifies the generated expressions (function name, arguments, 
`isDistinct` flag) and the query results of `groupBy().agg(...)`, global 
aggregation via `select`, and `cube` with `grouping`/`grouping_id` against 
Apache Spark 4.2.0.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Fable 5
    
    Closes #466 from dongjoon-hyun/SPARK-58493.
    
    Authored-by: Dongjoon Hyun <[email protected]>
    Signed-off-by: Dongjoon Hyun <[email protected]>
---
 Sources/SparkConnect/AggregateFunctions.swift      | 319 +++++++++++++++++++++
 .../AggregateFunctionsTests.swift                  | 209 ++++++++++++++
 2 files changed, 528 insertions(+)

diff --git a/Sources/SparkConnect/AggregateFunctions.swift 
b/Sources/SparkConnect/AggregateFunctions.swift
new file mode 100644
index 0000000..d065a48
--- /dev/null
+++ b/Sources/SparkConnect/AggregateFunctions.swift
@@ -0,0 +1,319 @@
+//
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//  http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+//
+
+/// Returns some value of the column for a group of rows.
+/// - Parameter col: A ``Column`` to aggregate.
+/// - Returns: A ``Column``.
+public func any_value(_ col: Column) -> Column {
+  return fn("any_value", col)
+}
+
+/// Returns some value of the column for a group of rows.
+/// If `ignoreNulls` is true, returns only non-null values.
+/// - Parameters:
+///   - col: A ``Column`` to aggregate.
+///   - ignoreNulls: A ``Column`` that evaluates to a boolean. Must be a 
constant.
+/// - Returns: A ``Column``.
+public func any_value(_ col: Column, _ ignoreNulls: Column) -> Column {
+  return fn("any_value", col, ignoreNulls)
+}
+
+/// Returns the approximate number of distinct items in a group.
+/// - Parameter col: A ``Column`` to aggregate.
+/// - Returns: A ``Column``.
+public func approx_count_distinct(_ col: Column) -> Column {
+  return fn("approx_count_distinct", col)
+}
+
+/// Returns the approximate number of distinct items in a group.
+/// - Parameters:
+///   - col: A ``Column`` to aggregate.
+///   - rsd: The maximum relative standard deviation allowed (default = 0.05).
+/// - Returns: A ``Column``.
+public func approx_count_distinct(_ col: Column, _ rsd: Double) -> Column {
+  return fn("approx_count_distinct", col, lit(rsd))
+}
+
+/// Returns true if all values of the column are true.
+/// - Parameter col: A ``Column`` that evaluates to a boolean.
+/// - Returns: A ``Column``.
+public func bool_and(_ col: Column) -> Column {
+  return fn("bool_and", col)
+}
+
+/// Returns true if at least one value of the column is true.
+/// - Parameter col: A ``Column`` that evaluates to a boolean.
+/// - Returns: A ``Column``.
+public func bool_or(_ col: Column) -> Column {
+  return fn("bool_or", col)
+}
+
+/// Returns a list of objects with duplicates.
+/// - Parameter col: A ``Column`` to aggregate.
+/// - Returns: A ``Column``.
+public func collect_list(_ col: Column) -> Column {
+  return fn("collect_list", col)
+}
+
+/// Returns a set of objects with duplicate elements eliminated.
+/// - Parameter col: A ``Column`` to aggregate.
+/// - Returns: A ``Column``.
+public func collect_set(_ col: Column) -> Column {
+  return fn("collect_set", col)
+}
+
+/// Returns the Pearson Correlation Coefficient for two columns.
+/// - Parameters:
+///   - column1: A ``Column``.
+///   - column2: A ``Column``.
+/// - Returns: A ``Column``.
+public func corr(_ column1: Column, _ column2: Column) -> Column {
+  return fn("corr", column1, column2)
+}
+
+/// Returns the number of distinct items in a group.
+/// This is an alias of ``count_distinct(_:_:)``.
+/// - Parameters:
+///   - expr: A ``Column`` to count.
+///   - exprs: Additional ``Column``s to count.
+/// - Returns: A ``Column``.
+public func countDistinct(_ expr: Column, _ exprs: Column...) -> Column {
+  return fn("count", [expr] + exprs, isDistinct: true)
+}
+
+/// Returns the number of distinct items in a group.
+/// - Parameters:
+///   - expr: A ``Column`` to count.
+///   - exprs: Additional ``Column``s to count.
+/// - Returns: A ``Column``.
+public func count_distinct(_ expr: Column, _ exprs: Column...) -> Column {
+  return fn("count", [expr] + exprs, isDistinct: true)
+}
+
+/// Returns the number of `TRUE` values for the expression.
+/// - Parameter col: A ``Column`` that evaluates to a boolean.
+/// - Returns: A ``Column``.
+public func count_if(_ col: Column) -> Column {
+  return fn("count_if", col)
+}
+
+/// Returns the population covariance for two columns.
+/// - Parameters:
+///   - column1: A ``Column``.
+///   - column2: A ``Column``.
+/// - Returns: A ``Column``.
+public func covar_pop(_ column1: Column, _ column2: Column) -> Column {
+  return fn("covar_pop", column1, column2)
+}
+
+/// Returns the sample covariance for two columns.
+/// - Parameters:
+///   - column1: A ``Column``.
+///   - column2: A ``Column``.
+/// - Returns: A ``Column``.
+public func covar_samp(_ column1: Column, _ column2: Column) -> Column {
+  return fn("covar_samp", column1, column2)
+}
+
+/// Returns the first value in a group.
+///
+/// The function by default returns the first values it sees. It will return 
the first non-null
+/// value it sees when `ignoreNulls` is set to true. If all values are null, 
then null is returned.
+/// - Parameter col: A ``Column`` to aggregate.
+/// - Returns: A ``Column``.
+public func first(_ col: Column) -> Column {
+  return first(col, false)
+}
+
+/// Returns the first value in a group.
+///
+/// The function by default returns the first values it sees. It will return 
the first non-null
+/// value it sees when `ignoreNulls` is set to true. If all values are null, 
then null is returned.
+/// - Parameters:
+///   - col: A ``Column`` to aggregate.
+///   - ignoreNulls: Whether to skip null values.
+/// - Returns: A ``Column``.
+public func first(_ col: Column, _ ignoreNulls: Bool) -> Column {
+  return fn("first", col, lit(ignoreNulls))
+}
+
+/// Indicates whether a specified column in a GROUP BY list is aggregated or 
not,
+/// returns 1 for aggregated or 0 for not aggregated in the result set.
+/// - Parameter col: A ``Column`` to check.
+/// - Returns: A ``Column``.
+public func grouping(_ col: Column) -> Column {
+  return fn("grouping", col)
+}
+
+/// Returns the level of grouping.
+/// - Parameter cols: ``Column``s to check.
+/// - Returns: A ``Column``.
+public func grouping_id(_ cols: Column...) -> Column {
+  return fn("grouping_id", cols)
+}
+
+/// Returns the kurtosis of the values in a group.
+/// - Parameter col: A ``Column`` to aggregate.
+/// - Returns: A ``Column``.
+public func kurtosis(_ col: Column) -> Column {
+  return fn("kurtosis", col)
+}
+
+/// Returns the last value in a group.
+///
+/// The function by default returns the last values it sees. It will return 
the last non-null
+/// value it sees when `ignoreNulls` is set to true. If all values are null, 
then null is returned.
+/// - Parameter col: A ``Column`` to aggregate.
+/// - Returns: A ``Column``.
+public func last(_ col: Column) -> Column {
+  return last(col, false)
+}
+
+/// Returns the last value in a group.
+///
+/// The function by default returns the last values it sees. It will return 
the last non-null
+/// value it sees when `ignoreNulls` is set to true. If all values are null, 
then null is returned.
+/// - Parameters:
+///   - col: A ``Column`` to aggregate.
+///   - ignoreNulls: Whether to skip null values.
+/// - Returns: A ``Column``.
+public func last(_ col: Column, _ ignoreNulls: Bool) -> Column {
+  return fn("last", col, lit(ignoreNulls))
+}
+
+/// Returns the value associated with the maximum value of `ord`.
+/// - Parameters:
+///   - col: A ``Column`` to return the value from.
+///   - ord: A ``Column`` to be maximized.
+/// - Returns: A ``Column``.
+public func max_by(_ col: Column, _ ord: Column) -> Column {
+  return fn("max_by", col, ord)
+}
+
+/// Returns the median of the values in a group.
+/// - Parameter col: A ``Column`` to aggregate.
+/// - Returns: A ``Column``.
+public func median(_ col: Column) -> Column {
+  return fn("median", col)
+}
+
+/// Returns the value associated with the minimum value of `ord`.
+/// - Parameters:
+///   - col: A ``Column`` to return the value from.
+///   - ord: A ``Column`` to be minimized.
+/// - Returns: A ``Column``.
+public func min_by(_ col: Column, _ ord: Column) -> Column {
+  return fn("min_by", col, ord)
+}
+
+/// Returns the most frequent value in a group.
+/// - Parameter col: A ``Column`` to aggregate.
+/// - Returns: A ``Column``.
+public func mode(_ col: Column) -> Column {
+  return fn("mode", col)
+}
+
+/// Returns the most frequent value in a group.
+/// - Parameters:
+///   - col: A ``Column`` to aggregate.
+///   - deterministic: If there are multiple equally-frequent results,
+///     whether to return the lowest (defined by min-hash) one.
+/// - Returns: A ``Column``.
+public func mode(_ col: Column, _ deterministic: Bool) -> Column {
+  return fn("mode", col, lit(deterministic))
+}
+
+/// Returns the approximate `percentile` of the numeric column `col` which is 
the smallest value
+/// in the ordered `col` values (sorted from least to greatest) such that no 
more than `percentage`
+/// of `col` values is less than the value or equal to that value.
+/// - Parameters:
+///   - col: A ``Column`` to aggregate.
+///   - percentage: A percentage ``Column``. Each value must be between 0.0 
and 1.0.
+///   - accuracy: A positive numeric literal ``Column`` that controls 
approximation accuracy
+///     at the cost of memory.
+/// - Returns: A ``Column``.
+public func percentile_approx(_ col: Column, _ percentage: Column, _ accuracy: 
Column) -> Column {
+  return fn("percentile_approx", col, percentage, accuracy)
+}
+
+/// Returns the skewness of the values in a group.
+/// - Parameter col: A ``Column`` to aggregate.
+/// - Returns: A ``Column``.
+public func skewness(_ col: Column) -> Column {
+  return fn("skewness", col)
+}
+
+/// Returns the sample standard deviation of the expression in a group.
+/// This is an alias of ``stddev_samp(_:)``.
+/// - Parameter col: A ``Column`` to aggregate.
+/// - Returns: A ``Column``.
+public func stddev(_ col: Column) -> Column {
+  return fn("stddev", col)
+}
+
+/// Returns the population standard deviation of the expression in a group.
+/// - Parameter col: A ``Column`` to aggregate.
+/// - Returns: A ``Column``.
+public func stddev_pop(_ col: Column) -> Column {
+  return fn("stddev_pop", col)
+}
+
+/// Returns the sample standard deviation of the expression in a group.
+/// - Parameter col: A ``Column`` to aggregate.
+/// - Returns: A ``Column``.
+public func stddev_samp(_ col: Column) -> Column {
+  return fn("stddev_samp", col)
+}
+
+/// Returns the sum of distinct values in the expression.
+/// This is an alias of ``sum_distinct(_:)``.
+/// - Parameter col: A ``Column`` to aggregate.
+/// - Returns: A ``Column``.
+public func sumDistinct(_ col: Column) -> Column {
+  return sum_distinct(col)
+}
+
+/// Returns the sum of distinct values in the expression.
+/// - Parameter col: A ``Column`` to aggregate.
+/// - Returns: A ``Column``.
+public func sum_distinct(_ col: Column) -> Column {
+  return fn("sum", [col], isDistinct: true)
+}
+
+/// Returns the population variance of the values in a group.
+/// - Parameter col: A ``Column`` to aggregate.
+/// - Returns: A ``Column``.
+public func var_pop(_ col: Column) -> Column {
+  return fn("var_pop", col)
+}
+
+/// Returns the unbiased variance of the values in a group.
+/// - Parameter col: A ``Column`` to aggregate.
+/// - Returns: A ``Column``.
+public func var_samp(_ col: Column) -> Column {
+  return fn("var_samp", col)
+}
+
+/// Returns the unbiased variance of the values in a group.
+/// This is an alias of ``var_samp(_:)``.
+/// - Parameter col: A ``Column`` to aggregate.
+/// - Returns: A ``Column``.
+public func variance(_ col: Column) -> Column {
+  return fn("variance", col)
+}
diff --git a/Tests/SparkConnectTests/AggregateFunctionsTests.swift 
b/Tests/SparkConnectTests/AggregateFunctionsTests.swift
new file mode 100644
index 0000000..3ac536a
--- /dev/null
+++ b/Tests/SparkConnectTests/AggregateFunctionsTests.swift
@@ -0,0 +1,209 @@
+//
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//  http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+//
+
+import Testing
+
+@testable import SparkConnect
+
+/// A test suite for `AggregateFunctions`
+@Suite(.serialized)
+struct AggregateFunctionsTests {
+
+  @Test
+  func aggregateFunctions() throws {
+    for (column, name) in [
+      (any_value(col("a")), "any_value"),
+      (approx_count_distinct(col("a")), "approx_count_distinct"),
+      (bool_and(col("a")), "bool_and"),
+      (bool_or(col("a")), "bool_or"),
+      (collect_list(col("a")), "collect_list"),
+      (collect_set(col("a")), "collect_set"),
+      (count_if(col("a")), "count_if"),
+      (grouping(col("a")), "grouping"),
+      (grouping_id(col("a")), "grouping_id"),
+      (kurtosis(col("a")), "kurtosis"),
+      (median(col("a")), "median"),
+      (mode(col("a")), "mode"),
+      (skewness(col("a")), "skewness"),
+      (stddev(col("a")), "stddev"),
+      (stddev_pop(col("a")), "stddev_pop"),
+      (stddev_samp(col("a")), "stddev_samp"),
+      (var_pop(col("a")), "var_pop"),
+      (var_samp(col("a")), "var_samp"),
+      (variance(col("a")), "variance"),
+    ] {
+      let expr = column.expr
+      #expect(expr.unresolvedFunction.functionName == name)
+      #expect(expr.unresolvedFunction.isDistinct == false)
+      
#expect(expr.unresolvedFunction.arguments[0].unresolvedAttribute.unparsedIdentifier
 == "a")
+    }
+  }
+
+  @Test
+  func aggregateFunctionArguments() throws {
+    for (column, name) in [
+      (any_value(col("a"), lit(true)), "any_value"),
+      (corr(col("a"), col("b")), "corr"),
+      (covar_pop(col("a"), col("b")), "covar_pop"),
+      (covar_samp(col("a"), col("b")), "covar_samp"),
+      (max_by(col("a"), col("b")), "max_by"),
+      (min_by(col("a"), col("b")), "min_by"),
+    ] {
+      let expr = column.expr
+      #expect(expr.unresolvedFunction.functionName == name)
+      #expect(expr.unresolvedFunction.arguments.count == 2)
+      
#expect(expr.unresolvedFunction.arguments[0].unresolvedAttribute.unparsedIdentifier
 == "a")
+    }
+
+    let rsd = approx_count_distinct(col("a"), 0.01).expr
+    #expect(rsd.unresolvedFunction.functionName == "approx_count_distinct")
+    #expect(rsd.unresolvedFunction.arguments[1].literal.double == 0.01)
+
+    let deterministic = mode(col("a"), true).expr
+    #expect(deterministic.unresolvedFunction.functionName == "mode")
+    #expect(deterministic.unresolvedFunction.arguments[1].literal.boolean == 
true)
+
+    let percentile = percentile_approx(col("a"), lit(0.5), lit(100)).expr
+    #expect(percentile.unresolvedFunction.functionName == "percentile_approx")
+    #expect(percentile.unresolvedFunction.arguments.count == 3)
+
+    #expect(grouping_id(col("a"), 
col("b")).expr.unresolvedFunction.arguments.count == 2)
+  }
+
+  @Test
+  func firstAndLast() throws {
+    for (column, name, ignoreNulls) in [
+      (first(col("a")), "first", false),
+      (first(col("a"), true), "first", true),
+      (last(col("a")), "last", false),
+      (last(col("a"), true), "last", true),
+    ] {
+      let expr = column.expr
+      #expect(expr.unresolvedFunction.functionName == name)
+      #expect(expr.unresolvedFunction.arguments.count == 2)
+      
#expect(expr.unresolvedFunction.arguments[0].unresolvedAttribute.unparsedIdentifier
 == "a")
+      #expect(expr.unresolvedFunction.arguments[1].literal.boolean == 
ignoreNulls)
+    }
+  }
+
+  @Test
+  func distinctAggregateFunctions() throws {
+    for (column, name, count) in [
+      (countDistinct(col("a")), "count", 1),
+      (countDistinct(col("a"), col("b")), "count", 2),
+      (count_distinct(col("a")), "count", 1),
+      (count_distinct(col("a"), col("b")), "count", 2),
+      (sumDistinct(col("a")), "sum", 1),
+      (sum_distinct(col("a")), "sum", 1),
+    ] {
+      let expr = column.expr
+      #expect(expr.unresolvedFunction.functionName == name)
+      #expect(expr.unresolvedFunction.isDistinct)
+      #expect(expr.unresolvedFunction.arguments.count == count)
+      
#expect(expr.unresolvedFunction.arguments[0].unresolvedAttribute.unparsedIdentifier
 == "a")
+    }
+  }
+
+  @Test
+  func groupByAggregateFunctions() async throws {
+    let spark = try await SparkSession.builder.getOrCreate()
+    let df = try await spark.sql(
+      "SELECT * FROM VALUES ('a', 1), ('a', 1), ('a', 2), ('b', 3) T(k, v)")
+    let rows = try await df.groupBy("k")
+      .agg(
+        countDistinct(col("v")), sum_distinct(col("v")), median(col("v")), 
mode(col("v")),
+        any_value(col("v"))
+      ).orderBy("k").collect()
+    #expect(rows == [Row("a", 2, 3, 1.0, 1, 1), Row("b", 1, 3, 3.0, 3, 3)])
+
+    let collected = try await df.groupBy("k")
+      .agg(
+        sort_array(collect_list(col("v"))).cast("string"),
+        sort_array(collect_set(col("v"))).cast("string")
+      ).orderBy("k").collect()
+    #expect(collected == [Row("a", "[1, 1, 2]", "[1, 2]"), Row("b", "[3]", 
"[3]")])
+    await spark.stop()
+  }
+
+  @Test
+  func selectStatisticalFunctions() async throws {
+    let spark = try await SparkSession.builder.getOrCreate()
+    let df = try await spark.sql(
+      "SELECT * FROM VALUES (double(1.0), double(10.0)), (double(3.0), 
double(20.0)) T(v, w)")
+    let variances = try await df.select(
+      variance(col("v")), var_samp(col("v")), var_pop(col("v")),
+      stddev(col("v")), stddev_samp(col("v")), stddev_pop(col("v"))
+    ).collect()
+    #expect(
+      variances == [Row(2.0, 2.0, 1.0, 1.4142135623730951, 1.4142135623730951, 
1.0)])
+
+    let moments = try await df.select(skewness(col("v")), 
kurtosis(col("v"))).collect()
+    #expect(moments == [Row(0.0, -2.0)])
+
+    let covariances = try await df.select(
+      corr(col("v"), col("w")), covar_samp(col("v"), col("w")), 
covar_pop(col("v"), col("w"))
+    ).collect()
+    #expect(covariances == [Row(1.0, 10.0, 5.0)])
+
+    let percentiles = try await df.select(
+      median(col("v")), percentile_approx(col("v"), lit(0.5), lit(100))
+    ).collect()
+    #expect(percentiles == [Row(2.0, 1.0)])
+    await spark.stop()
+  }
+
+  @Test
+  func selectFirstLastFunctions() async throws {
+    let spark = try await SparkSession.builder.getOrCreate()
+    let df = try await spark.sql("SELECT * FROM VALUES (NULL), (1), (2), 
(NULL) T(v)")
+    let rows = try await df.select(
+      first(col("v")), first(col("v"), true), last(col("v")), last(col("v"), 
true)
+    ).collect()
+    #expect(rows == [Row(nil, 1, nil, 2)])
+    await spark.stop()
+  }
+
+  @Test
+  func selectBooleanAndCountFunctions() async throws {
+    let spark = try await SparkSession.builder.getOrCreate()
+    let df = try await spark.sql("SELECT * FROM VALUES (true, 1), (false, 
NULL) T(b, v)")
+    let rows = try await df.select(
+      bool_and(col("b")), bool_or(col("b")), count_if(col("b")),
+      countDistinct(col("v")), approx_count_distinct(col("v")),
+      approx_count_distinct(col("v"), 0.01)
+    ).collect()
+    #expect(rows == [Row(false, true, 1, 1, 1, 1)])
+
+    let ordered = try await spark.sql("SELECT * FROM VALUES ('a', 1), ('b', 2) 
T(k, v)")
+      .select(max_by(col("k"), col("v")), min_by(col("k"), col("v"))).collect()
+    #expect(ordered == [Row("b", "a")])
+    await spark.stop()
+  }
+
+  @Test
+  func selectGroupingFunctions() async throws {
+    let spark = try await SparkSession.builder.getOrCreate()
+    let df = try await spark.sql("SELECT * FROM VALUES ('a', 1), ('b', 2) T(k, 
v)")
+    let rows = try await df.cube("k")
+      .agg(grouping(col("k")), grouping_id(col("k")).alias("gid"))
+      .orderBy("gid", "k").collect()
+    #expect(rows == [Row("a", 0, 0), Row("b", 0, 0), Row(nil, 1, 1)])
+    await spark.stop()
+  }
+}


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to