andygrove opened a new issue, #250:
URL: https://github.com/apache/datafusion-spark/issues/250

   Part of #176 and apache/datafusion-comet#7300.
   
   ### What the function does
   
   `zip_with(left, right, func)` merges two arrays element by element with a 
two-argument lambda `(x, y) -> ...`. If
   one array is shorter, it is padded with NULLs at the end to the length of 
the longer one before `func` is applied.
   The result type is `ARRAY<func.dataType>` with `containsNull = 
func.nullable`. The result is NULL if either input
   array is NULL, and nullable if either input is nullable. Both lambda 
parameters are always declared nullable,
   because of the padding. Both arguments must be `ARRAY`s, and their element 
types can differ. There is no index
   argument. Non-lambda `func` arguments are wrapped in a hidden lambda. 
Available since Spark 2.4.0. No aliases.
   
   ### Examples
   
   ```sql
   SELECT zip_with(array(1, 2, 3), array('a', 'b', 'c'), (x, y) -> (y, x));
   -- [{"y":"a","x":1},{"y":"b","x":2},{"y":"c","x":3}]
   SELECT zip_with(array(1, 2), array(3, 4), (x, y) -> x + y);
   -- [4,6]
   SELECT zip_with(array('a', 'b', 'c'), array('d', 'e', 'f'), (x, y) -> 
concat(x, y));
   -- ["ad","be","cf"]
   SELECT zip_with(array('a'), array('d', null, 'f'), (x, y) -> coalesce(x, y));
   -- ["a",null,"f"]
   SELECT zip_with(array(1, 2, 3), array(4, 5, 6), 100);
   -- [100,100,100]
   -- nested view (x, ys, zs) = (1, [32,97], [[12,99],[123,42],[1]]), (2, 
[77,-76], [[6,96,65],[-1,-2]]), (3, [12], [[17]])
   SELECT zip_with(ys, zs, (a, b) -> a + size(b)) FROM nested;
   -- [34,99,null] / [80,-74] / [13]
   ```
   
   These come from the `ExpressionDescription` and 
`higher-order-functions.sql.out`. The last example is listed in view
   order, and it shows padding: row 1 has 2 `ys` and 3 `zs`, so the third `a` 
is NULL.
   
   ### Edge cases to test
   
   - Arrays of different lengths, in both directions: `zip_with([1,2,3], 
[1,2,3,4], (x, y) -> x + y)` gives
     `[2,4,6,null]`, and `zip_with([1,null], [1,null,3], add)` and 
`zip_with([1,null,3], [1,null], add)` both give
     `[2,null,null]` (`HigherOrderFunctionsSuite`). The padding must be NULL 
even when the input element type is
     non-nullable.
   - The lambda still runs on padded positions, so a body that ignores NULLs 
produces values there:
     `coalesce(x, y)` above gives `"f"`.
   - Either array NULL gives NULL. Spark evaluates `left` first and does not 
evaluate `right` if `left` is NULL.
   - Both arrays empty gives `[]`. One empty array gives a result as long as 
the other, computed with all-NULL values on
     one side.
   - A NULL inner array within an array of arrays:
     `zip_with(aai1, aai1, (a1, a2) -> cast(transform(a1, x -> x + 1) as 
string))` on `[[1,2,3],null,[4,5]]` gives
     `["[2, 3, 4]",null,"[5, 6]"]`.
   - Element types that differ between `left` and `right` (int and string), and 
struct results built with `(y, x)`.
     Spark names the struct fields after the lambda variables 
(`{"y":...,"x":...}`).
   - Captured outer columns, nested lambdas, and nondeterministic bodies 
evaluated once per output position.
   - Errors inside the lambda under ANSI mode, for example `x / y` with a zero 
divisor. They are raised only for
     positions that are actually evaluated, which includes padded positions.
   - Lambda arity: exactly 2 parameters, otherwise 
`INVALID_LAMBDA_FUNCTION_CALL.NUM_ARGS_MISMATCH`. A non-array
     argument fails with `DATATYPE_MISMATCH.UNEXPECTED_INPUT_TYPE`.
   
   ### Differences across Spark versions
   
   - None found in `ZipWith` between 3.4 and 4.2. ANSI mode is on by default 
from 4.0.
   
   ### Implementation notes
   
   - Spark: `ZipWith` in 
`sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/higherOrderFunctions.scala`.
   - Infrastructure. DataFusion's higher-order function framework 
(`HigherOrderUDFImpl` in
     `datafusion/expr/src/higher_order_function.rs`, and 
`LambdaArgument::evaluate` with `spread_captures` for captured
     columns) is on `branch-56`, which this crate uses. `lambda_parameters` 
must return two parameter fields, both
     nullable, with the two element types.
   - DataFusion core has no two-array higher-order function. `arrays_zip` is 
not a higher-order function. To evaluate
     vectorized, build per row the `max(len_l, len_r)` positions, then `take` 
both value arrays with NULL indices for the
     padding, which gives two aligned flattened columns. Evaluate the body once 
over them, and spread captures to the
     same positions. The result offsets are the per-row maximum lengths.
   - This crate's `crates/spark/src/function/lambda/mod.rs` is an empty 
`ScalarUDF` placeholder. It needs a
     `HigherOrderUDF` list and registration through 
`register_higher_order_function` /
     `SessionStateBuilder::higher_order_functions`.
   - Add `crates/sqllogictest/test_files/lambda/zip_with.slt` and document the 
function with `#[user_doc]`.
   
   ---
   _This issue was generated by an LLM (Claude) from the Apache Spark source 
code and tests. Examples come from Spark's `ExpressionDescription` examples, 
test suites and `.sql.out` golden files, but please verify them against Spark 
before relying on them._


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to