andygrove opened a new issue, #246:
URL: https://github.com/apache/datafusion-spark/issues/246

   Part of #176 and apache/datafusion-comet#7300.
   
   ### What the function does
   
   `transform(expr, func)` applies `func` to each element of the array `expr` 
and returns a new array of the results.
   `func` is either `x -> ...` or `(x, i) -> ...`, where `i` is the 0-based 
element index (`INT`, non-null). The result
   type is `ARRAY<func.dataType>` with `containsNull = func.nullable`. The 
result is nullable when `expr` is nullable,
   and a NULL array gives NULL. `expr` must be an `ARRAY` 
(`DATATYPE_MISMATCH.UNEXPECTED_INPUT_TYPE` otherwise). There
   is no implicit cast. Available since Spark 2.4.0. No aliases. Spark's 
`nodeName` is `transform`.
   
   The element variable has the array's element type and `containsNull` 
nullability. The body may reference outer
   columns and contain nested lambdas. If `func` is not a lambda, Spark wraps 
it in a hidden lambda, so it is evaluated
   once per element: `transform(array(1, 2, 3), 42)` gives `[42,42,42]`.
   
   ### Examples
   
   ```sql
   SELECT transform(array(1, 2, 3), x -> x + 1);
   -- [2,3,4]
   SELECT transform(array(1, 2, 3), (x, i) -> x + i);
   -- [1,3,5]
   SELECT transform(cast(null as array<int>), x -> x + 1);
   -- NULL
   SELECT transform(array(1, 2, 3), 42);
   -- [42,42,42]
   SELECT transform(array(1, 2, 3), array('a', 'b'));
   -- [["a","b"],["a","b"],["a","b"]]
   -- nested view (x, ys, zs) = (1, [32,97], [[12,99],[123,42],[1]]), (2, 
[77,-76], [[6,96,65],[-1,-2]]), (3, [12], [[17]])
   SELECT transform(ys, (y, i) -> y + i) FROM nested;
   -- [32,98] / [77,-75] / [12]
   SELECT transform(zs, z -> concat(ys, z)) FROM nested;            -- captured 
outer column
   -- [[32,97,12,99],[32,97,123,42],[32,97,1]] / 
[[77,-76,6,96,65],[77,-76,-1,-2]] / [[12,17]]
   SELECT transform(zs, z -> filter(z, zz -> zz > 50)) FROM nested;  -- nested 
lambda
   -- [[99],[123],[]] / [[96,65],[]] / [[]]
   SELECT transform(ys, all -> all * all) FROM VALUES (array(32, 97)) AS t(ys);
   -- [1024,9409]
   ```
   
   These come from the `ExpressionDescription` and 
`higher-order-functions.sql.out`. The `nested` rows are listed in
   view order, while the `.sql.out` sorts them.
   
   ### Edge cases to test
   
   - A NULL array gives NULL. An empty array gives `[]`. NULL elements are 
passed to the lambda as NULL:
     `transform(array(1, null, 3), x -> x + 1)` gives `[2,null,4]`, and `(x, i) 
-> x + i` gives `[1,null,5]`
     (`HigherOrderFunctionsSuite`).
   - The index argument counts from 0 and counts NULL elements. A NULL array 
inside an array of arrays gives NULL at
     that position. `transform(aai, a -> cast(transform(a, (x, i) -> x + i) as 
string))` on `[[1,2,3],null,[4,5]]` gives
     `["[1, 3, 5]", null, "[4, 6]"]`.
   - Lambda arity. Spark accepts 1 or 2 parameters. `(x, y, z) -> ...` fails 
with
     `INVALID_LAMBDA_FUNCTION_CALL.NUM_ARGS_MISMATCH`. Duplicate parameter 
names fail with
     `INVALID_LAMBDA_FUNCTION_CALL.DUPLICATE_ARG_NAMES`.
   - A lambda variable shadows a column with the same name: `transform(arg, arg 
-> arg)` is the identity, while
     `transform(arg, arg)` repeats the whole column `arg` once per element 
(`DataFrameFunctionsSuite`, "special cases").
   - Captured outer columns must line up with the row each element came from, 
including rows with empty arrays, rows
     with NULL arrays, and sliced input.
   - Errors inside the lambda under ANSI mode. Spark evaluates the body element 
by element. By code reading,
     `transform(array(1, 0), x -> 10 / x)` gives `[10.0,null]` with 
`spark.sql.ansi.enabled=false` and raises
     `DIVIDE_BY_ZERO` with ANSI on (the default from 4.0). Values hidden under 
a NULL list entry, or outside a slice,
     must not be evaluated, because they could raise errors Spark never raises.
   - Nondeterministic bodies (`x -> rand()`, `x -> uuid()`) must be evaluated 
once per element, not once per row, and
     must not be constant-folded or common-subexpression-eliminated across 
elements.
   - Result nullability: `containsNull` follows the lambda body's nullability, 
not the input's.
   - Element types: primitives, strings, structs, arrays of arrays, maps, and 
`NullType` arrays (`array()`).
   
   ### Differences across Spark versions
   
   - None found in `ArrayTransform` between 3.4 and 4.2, apart from internal 
refactors. Note that ANSI mode is on by
     default from 4.0, which changes what arithmetic inside the lambda does.
   
   ### Implementation notes
   
   - Spark: `ArrayTransform` in 
`sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/higherOrderFunctions.scala`.
     Lambda binding is in 
`sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/higherOrderFunctions.scala`
     (`ResolveLambdaVariables`).
   - Infrastructure. DataFusion has higher-order functions:
     - `HigherOrderUDF` / `HigherOrderUDFImpl` in 
`datafusion/expr/src/higher_order_function.rs`;
     - lambda planning in `datafusion/sql/src/expr/function.rs`;
     - `LambdaExpr` / `LambdaVariable` in 
`datafusion/physical-expr/src/expressions/`.
   
     This is all on `branch-56`, which this crate uses. 
`LambdaArgument::evaluate` takes one closure per parameter plus a
     `spread_captures` callback that expands captured outer columns to the 
flattened element rows. A lambda may declare
     fewer parameters than `lambda_parameters` offers, but not more. Spark's 
SQL dialect in sqlparser
     (`SparkSqlDialect`) supports lambda syntax.
   - DataFusion core has `array_transform` (alias `list_transform`) in 
`datafusion/functions-nested/src/array_transform.rs`.
     It differs from Spark in these ways:
     - it offers only the element parameter, so `(x, i) -> ...` is a planning 
error;
     - it does not accept a non-lambda function argument;
     - it evaluates the body once over the flattened values of the whole batch 
rather than element by element.
   
     For deterministic bodies the vectorized evaluation gives the same results. 
It already avoids values under NULL
     list entries and outside slices. A Spark version can follow the same 
structure but must offer
     `[element, Int32 index]` from `lambda_parameters` and build the index 
array from the list offsets. The
     `lambda_utils.rs` helpers are `pub(crate)`, so copy them or ask for them 
to be made public upstream.
   - This crate: `crates/spark/src/function/lambda/mod.rs` is an empty 
placeholder typed as `Vec<Arc<ScalarUDF>>` and
     chained into `all_default_scalar_functions`. It needs a 
`Vec<Arc<HigherOrderUDF>>` list, and `register_all`
     and `SessionStateBuilderSpark::with_spark_features` need to register it 
(`register_higher_order_function` /
     `SessionStateBuilder::higher_order_functions`).
   - Spark's hidden-lambda form (a non-lambda second argument) should be 
handled when planning, by wrapping the
     expression in a zero-reference lambda.
   - Add `crates/sqllogictest/test_files/lambda/transform.slt` and document the 
function with `#[user_doc]`.
   
   ---
   _This issue was generated by an LLM (Claude) from the Apache Spark source 
code and tests. Examples come from Spark's `ExpressionDescription` examples, 
test suites and `.sql.out` golden files, but please verify them against Spark 
before relying on them._


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to