andygrove opened a new issue, #246:
URL: https://github.com/apache/datafusion-spark/issues/246
Part of #176 and apache/datafusion-comet#7300.
### What the function does
`transform(expr, func)` applies `func` to each element of the array `expr`
and returns a new array of the results.
`func` is either `x -> ...` or `(x, i) -> ...`, where `i` is the 0-based
element index (`INT`, non-null). The result
type is `ARRAY<func.dataType>` with `containsNull = func.nullable`. The
result is nullable when `expr` is nullable,
and a NULL array gives NULL. `expr` must be an `ARRAY`
(`DATATYPE_MISMATCH.UNEXPECTED_INPUT_TYPE` otherwise). There
is no implicit cast. Available since Spark 2.4.0. No aliases. Spark's
`nodeName` is `transform`.
The element variable has the array's element type and `containsNull`
nullability. The body may reference outer
columns and contain nested lambdas. If `func` is not a lambda, Spark wraps
it in a hidden lambda, so it is evaluated
once per element: `transform(array(1, 2, 3), 42)` gives `[42,42,42]`.
### Examples
```sql
SELECT transform(array(1, 2, 3), x -> x + 1);
-- [2,3,4]
SELECT transform(array(1, 2, 3), (x, i) -> x + i);
-- [1,3,5]
SELECT transform(cast(null as array<int>), x -> x + 1);
-- NULL
SELECT transform(array(1, 2, 3), 42);
-- [42,42,42]
SELECT transform(array(1, 2, 3), array('a', 'b'));
-- [["a","b"],["a","b"],["a","b"]]
-- nested view (x, ys, zs) = (1, [32,97], [[12,99],[123,42],[1]]), (2,
[77,-76], [[6,96,65],[-1,-2]]), (3, [12], [[17]])
SELECT transform(ys, (y, i) -> y + i) FROM nested;
-- [32,98] / [77,-75] / [12]
SELECT transform(zs, z -> concat(ys, z)) FROM nested; -- captured
outer column
-- [[32,97,12,99],[32,97,123,42],[32,97,1]] /
[[77,-76,6,96,65],[77,-76,-1,-2]] / [[12,17]]
SELECT transform(zs, z -> filter(z, zz -> zz > 50)) FROM nested; -- nested
lambda
-- [[99],[123],[]] / [[96,65],[]] / [[]]
SELECT transform(ys, all -> all * all) FROM VALUES (array(32, 97)) AS t(ys);
-- [1024,9409]
```
These come from the `ExpressionDescription` and
`higher-order-functions.sql.out`. The `nested` rows are listed in
view order, while the `.sql.out` sorts them.
### Edge cases to test
- A NULL array gives NULL. An empty array gives `[]`. NULL elements are
passed to the lambda as NULL:
`transform(array(1, null, 3), x -> x + 1)` gives `[2,null,4]`, and `(x, i)
-> x + i` gives `[1,null,5]`
(`HigherOrderFunctionsSuite`).
- The index argument counts from 0 and counts NULL elements. A NULL array
inside an array of arrays gives NULL at
that position. `transform(aai, a -> cast(transform(a, (x, i) -> x + i) as
string))` on `[[1,2,3],null,[4,5]]` gives
`["[1, 3, 5]", null, "[4, 6]"]`.
- Lambda arity. Spark accepts 1 or 2 parameters. `(x, y, z) -> ...` fails
with
`INVALID_LAMBDA_FUNCTION_CALL.NUM_ARGS_MISMATCH`. Duplicate parameter
names fail with
`INVALID_LAMBDA_FUNCTION_CALL.DUPLICATE_ARG_NAMES`.
- A lambda variable shadows a column with the same name: `transform(arg, arg
-> arg)` is the identity, while
`transform(arg, arg)` repeats the whole column `arg` once per element
(`DataFrameFunctionsSuite`, "special cases").
- Captured outer columns must line up with the row each element came from,
including rows with empty arrays, rows
with NULL arrays, and sliced input.
- Errors inside the lambda under ANSI mode. Spark evaluates the body element
by element. By code reading,
`transform(array(1, 0), x -> 10 / x)` gives `[10.0,null]` with
`spark.sql.ansi.enabled=false` and raises
`DIVIDE_BY_ZERO` with ANSI on (the default from 4.0). Values hidden under
a NULL list entry, or outside a slice,
must not be evaluated, because they could raise errors Spark never raises.
- Nondeterministic bodies (`x -> rand()`, `x -> uuid()`) must be evaluated
once per element, not once per row, and
must not be constant-folded or common-subexpression-eliminated across
elements.
- Result nullability: `containsNull` follows the lambda body's nullability,
not the input's.
- Element types: primitives, strings, structs, arrays of arrays, maps, and
`NullType` arrays (`array()`).
### Differences across Spark versions
- None found in `ArrayTransform` between 3.4 and 4.2, apart from internal
refactors. Note that ANSI mode is on by
default from 4.0, which changes what arithmetic inside the lambda does.
### Implementation notes
- Spark: `ArrayTransform` in
`sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/higherOrderFunctions.scala`.
Lambda binding is in
`sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/higherOrderFunctions.scala`
(`ResolveLambdaVariables`).
- Infrastructure. DataFusion has higher-order functions:
- `HigherOrderUDF` / `HigherOrderUDFImpl` in
`datafusion/expr/src/higher_order_function.rs`;
- lambda planning in `datafusion/sql/src/expr/function.rs`;
- `LambdaExpr` / `LambdaVariable` in
`datafusion/physical-expr/src/expressions/`.
This is all on `branch-56`, which this crate uses.
`LambdaArgument::evaluate` takes one closure per parameter plus a
`spread_captures` callback that expands captured outer columns to the
flattened element rows. A lambda may declare
fewer parameters than `lambda_parameters` offers, but not more. Spark's
SQL dialect in sqlparser
(`SparkSqlDialect`) supports lambda syntax.
- DataFusion core has `array_transform` (alias `list_transform`) in
`datafusion/functions-nested/src/array_transform.rs`.
It differs from Spark in these ways:
- it offers only the element parameter, so `(x, i) -> ...` is a planning
error;
- it does not accept a non-lambda function argument;
- it evaluates the body once over the flattened values of the whole batch
rather than element by element.
For deterministic bodies the vectorized evaluation gives the same results.
It already avoids values under NULL
list entries and outside slices. A Spark version can follow the same
structure but must offer
`[element, Int32 index]` from `lambda_parameters` and build the index
array from the list offsets. The
`lambda_utils.rs` helpers are `pub(crate)`, so copy them or ask for them
to be made public upstream.
- This crate: `crates/spark/src/function/lambda/mod.rs` is an empty
placeholder typed as `Vec<Arc<ScalarUDF>>` and
chained into `all_default_scalar_functions`. It needs a
`Vec<Arc<HigherOrderUDF>>` list, and `register_all`
and `SessionStateBuilderSpark::with_spark_features` need to register it
(`register_higher_order_function` /
`SessionStateBuilder::higher_order_functions`).
- Spark's hidden-lambda form (a non-lambda second argument) should be
handled when planning, by wrapping the
expression in a zero-reference lambda.
- Add `crates/sqllogictest/test_files/lambda/transform.slt` and document the
function with `#[user_doc]`.
---
_This issue was generated by an LLM (Claude) from the Apache Spark source
code and tests. Examples come from Spark's `ExpressionDescription` examples,
test suites and `.sql.out` golden files, but please verify them against Spark
before relying on them._
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]