andygrove opened a new issue, #247:
URL: https://github.com/apache/datafusion-spark/issues/247
Part of #176 and apache/datafusion-comet#7300.
### What the function does
`aggregate(expr, start, merge[, finish])` folds the array `expr` from left
to right. It starts with the accumulator
`start`, applies `merge(acc, x)` to each element, and then applies
`finish(acc)` to the final accumulator. If
`finish` is omitted it is the identity (`LambdaFunction.identity`). The
result type is `finish.dataType`, nullable
when `argument.nullable || finish.nullable`. A NULL array gives NULL.
`reduce` is an alias, added in Spark 3.4.0;
`aggregate` has existed since 2.4.0.
The accumulator type is the type of `start`. In `checkInputDataTypes`,
`merge.dataType` must equal `start.dataType`
structurally, ignoring nullability. Otherwise analysis fails with
`DATATYPE_MISMATCH.UNEXPECTED_INPUT_TYPE`
(`paramIndex = "third"`, `requiredType = start type`). There is no implicit
widening, so `start` must be written
with the type the merge produces. The accumulator variable is always treated
as nullable. `expr` must be an
`ARRAY`. `merge` must take exactly 2 parameters and `finish` exactly 1,
otherwise the call fails with
`INVALID_LAMBDA_FUNCTION_CALL.NUM_ARGS_MISMATCH`. Non-lambda `merge` or
`finish` arguments are wrapped in hidden
lambdas.
### Examples
```sql
SELECT aggregate(array(1, 2, 3), 0, (acc, x) -> acc + x);
-- 6
SELECT aggregate(array(1, 2, 3), 0, (acc, x) -> acc + x, acc -> acc * 10);
-- 60
SELECT reduce(array(1, 2, 3), 0, (acc, x) -> acc + x, acc -> acc * 10);
-- 60
SELECT aggregate(cast(null as array<int>), 0, (a, y) -> a + y + 1, a -> a +
2);
-- NULL
SELECT aggregate(ys, (0 as sum, 0 as n), (acc, x) -> (acc.sum + x, acc.n +
1), acc -> acc.sum / acc.n) FROM nested;
-- 64.5 / 0.5 / 12.0 (struct accumulator; DOUBLE result)
SELECT aggregate(ys, 0, (y, a) -> y + a + x) FROM nested; -- captured
outer column x
-- 131 / 5 / 15
SELECT aggregate(split('abcdefgh',''), array(array('')), (acc, x) ->
array(array(x)));
-- [["h"]]
SELECT aggregate(array(1, 2, 3), 0, 100);
-- 100
SELECT aggregate(array(1, 2, 3), struct('init', 0), struct('final', 999));
-- {"col1":"final","col2":999}
SELECT aggregate(array(1, 2, 3), map(), map('result', 999));
-- DATATYPE_MISMATCH.UNEXPECTED_INPUT_TYPE (requiredType "MAP<VOID, VOID>",
inputType "MAP<STRING, INT>")
```
The `nested` view is (x, ys) = (1, [32,97]), (2, [77,-76]), (3, [12]), with
results in view order. These come from the
`ExpressionDescription` and `higher-order-functions.sql.out`.
### Edge cases to test
- NULL array, empty array (gives `finish(start)`: an empty `ARRAY<INT>` with
`0, (acc, x) -> acc + x, acc -> acc * 10` gives `0`),
and NULL elements: `aggregate([1,null,3], 0, (acc, x) -> acc + coalesce(x,
0), acc -> acc * 10)` gives `40`
(`HigherOrderFunctionsSuite`). Without `coalesce`, a NULL element makes
the accumulator NULL from that point on.
- A NULL `start` value, and a merge that returns NULL partway through. The
accumulator is nullable and the NULL is
carried forward to `finish`.
- Accumulator type strictness. For example, by code reading,
`aggregate(array(1.5D), 0, (acc, x) -> acc + x)` fails
analysis because the merge type is `DOUBLE` and the start type is `INT`.
Callers must write `0D`, `0L`,
`cast(0 as decimal(p,s))`, and so on. Pay attention to DECIMAL precision
growth: under Spark's decimal rules
`acc + x` widens the type, so a DECIMAL accumulator usually needs an
explicit cast inside `merge`.
- Struct, array and map accumulators. The empty `map()` and `array()` starts
have `VOID` element types and are
rejected (see above).
- `finish` changing the type: `acc -> acc.sum / acc.n` returns DOUBLE.
- Nested folds: `aggregate(aai, 0, (acc, a) -> coalesce(aggregate(a, acc,
(acc, e) -> acc + e), acc))` on
`[[1,2,3],null,[4,5]]` gives `15`. Also nested lambdas that reuse the name
`acc`.
- Captured outer columns in `merge` and in `finish`.
- Overflow inside `merge` under ANSI mode: integer `acc + x` raises
`ARITHMETIC_OVERFLOW` with ANSI on and wraps with
ANSI off.
- Evaluation is strictly sequential. Each step sees the previous
accumulator, so the body cannot be evaluated once
over all flattened elements. Nondeterministic merge bodies are evaluated
once per element in order.
- Lambda arity errors: `aggregate(s, '', x -> x)` and `aggregate(s, '',
(acc, x) -> x, (acc, x) -> x)`, plus
`aggregate(i, 0, ...)` on a non-array argument.
### Differences across Spark versions
- The `reduce` alias was added in 3.4.0, so it is present in every version
in scope.
- The `paramIndex` in the type-mismatch error changed from `"3"` (3.x) to
`"third"` (4.0+).
- ANSI mode is on by default from 4.0, which changes overflow behaviour
inside `merge`.
### Implementation notes
- Spark: `ArrayAggregate` in
`sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/higherOrderFunctions.scala`.
The alias is registered in `FunctionRegistry.scala` with
`expression[ArrayAggregate]("reduce", setAlias = true,
Some("3.4.0"))`.
- Infrastructure. DataFusion's higher-order function framework
(`datafusion/expr/src/higher_order_function.rs`, on `branch-56`, which
this crate uses) supports several lambdas
whose parameter types depend on each other. `lambda_parameters` is called
repeatedly with
`LambdaParametersProgress::Partial` until the types are known. The trait
docs use an `array_reduce` with a merge
and a finish lambda as the worked example, and test-only `MockArrayReduce`
implementations exist in
`datafusion/expr/src/higher_order_function.rs` and
`datafusion/sql/tests/sql_integration.rs`.
- DataFusion core has no array fold or reduce function, so this needs to be
written from scratch. The evaluation can
still be vectorized across rows: step `k` evaluates `merge` on a batch
made of the accumulators and the `k`-th
element of every row that has at least `k + 1` elements, using
`spread_captures` to project captured columns onto
those rows. Then evaluate `finish` once.
- This crate's `crates/spark/src/function/lambda/mod.rs` is an empty
`ScalarUDF` placeholder. It needs a
`HigherOrderUDF` list and registration through
`register_higher_order_function` /
`SessionStateBuilder::higher_order_functions`. Register `reduce` as an
alias.
- Add `crates/sqllogictest/test_files/lambda/aggregate.slt` and document the
function with `#[user_doc]`.
---
_This issue was generated by an LLM (Claude) from the Apache Spark source
code and tests. Examples come from Spark's `ExpressionDescription` examples,
test suites and `.sql.out` golden files, but please verify them against Spark
before relying on them._
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]