andygrove opened a new issue, #249:
URL: https://github.com/apache/datafusion-spark/issues/249

   Part of #176 and apache/datafusion-comet#7300.
   
   ### What the function does
   
   `forall(expr, pred)` returns whether `pred` holds for every element of the 
array `expr`. `pred` must be a
   one-argument lambda returning `BOOLEAN`. A non-boolean body, or a non-lambda 
literal such as
   `forall(array(1, 2, 3), 1)`, fails with 
`DATATYPE_MISMATCH.UNEXPECTED_INPUT_TYPE`. There is no index argument. The
   result type is `BOOLEAN`, nullable when `argument.nullable || 
function.nullable`. Available since Spark 3.0.0. No
   aliases.
   
   Semantics (`ArrayForAll`; unlike `exists`, there is no legacy switch, and 
three-valued logic always applies):
   
   - A NULL array gives NULL.
   - Elements are visited in order, and evaluation **stops at the first 
`false`**.
   - If any element gives `false`, the result is `false`.
   - Otherwise, if any element gave NULL, the result is NULL.
   - Otherwise the result is `true`. An empty array gives `true`.
   
   ### Examples
   
   ```sql
   SELECT forall(array(1, 2, 3), x -> x % 2 == 0);
   -- false
   SELECT forall(array(2, 4, 8), x -> x % 2 == 0);
   -- true
   SELECT forall(array(1, null, 3), x -> x % 2 == 0);
   -- false
   SELECT forall(array(2, null, 8), x -> x % 2 == 0);
   -- NULL
   ```
   
   These come from the `ExpressionDescription`. `HigherOrderFunctionsSuite` 
adds the following cases:
   
   | input | predicate | result |
   |---|---|---|
   | `[2,4,8]` | `x IS NULL OR x % 2 == 1` | false |
   | `[2,4,8]` | `NULL` | NULL |
   | `[1,null,3]` | `x IS NULL OR x % 2 == 1` | true |
   | `[1,null,3]` | `NULL` | NULL |
   | `[2,null,8]` | `false` | false |
   | `NULL` | any | NULL |
   | `['a0','a1','a2','a3']` | `startswith(x, 'a')` | true |
   | `[null,'b','c']` | `startswith(x, 'a')` | false |
   
   Nested: `transform(aai, a -> forall(a, x -> x IS NULL OR x % 2 == 1))` on 
`[[1,3,null],null,[4,5]]` gives
   `[true,null,false]`.
   
   ### Edge cases to test
   
   - A NULL array gives NULL. An empty array gives `true`. Also test NULL 
elements, and a predicate that is NULL for
     every element (NULL result).
   - The NULL-versus-`false` ordering. `[null, 'b', 'c']` with `startswith(x, 
'a')` gives `false`: the first element gives
     NULL, the second gives `false`, and `false` wins.
   - Short-circuiting. Evaluation stops at the first `false`. By code reading,
     `forall(array(-1, 0), x -> 1 / x > 0)` returns `false` even with ANSI on, 
because `1 / 0` is never evaluated, while
     `forall(array(1, 0), x -> 1 / x > 0)` raises `DIVIDE_BY_ZERO` with ANSI on.
   - Captured outer columns and nondeterministic predicates, evaluated per 
element in order.
   - Element types: primitives, strings, structs, and nested arrays.
   - Lambda arity: `(x, i) -> ...` fails with 
`INVALID_LAMBDA_FUNCTION_CALL.NUM_ARGS_MISMATCH`.
   
   ### Differences across Spark versions
   
   - None found between 3.4 and 4.2. ANSI mode is on by default from 4.0, which 
matters for errors inside the
     predicate.
   
   ### Implementation notes
   
   - Spark: `ArrayForAll` in 
`sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/higherOrderFunctions.scala`.
   - Infrastructure. DataFusion's higher-order function framework 
(`HigherOrderUDFImpl` in
     `datafusion/expr/src/higher_order_function.rs`, and 
`LambdaArgument::evaluate` with `spread_captures` for captured
     columns) is on `branch-56`, which this crate uses. This crate's 
`crates/spark/src/function/lambda/mod.rs` is an
     empty `ScalarUDF` placeholder. It needs a `HigherOrderUDF` list and 
registration through
     `register_higher_order_function` / 
`SessionStateBuilder::higher_order_functions`.
   - DataFusion core has no `forall` / `all_match`. `array_any_match` 
(`datafusion/functions-nested/src/array_any_match.rs`)
     is the mirror image and shows the framework pattern: evaluate the 
predicate over the flattened values, then reduce
     each row's range. A Spark `forall` can use the same reduction (any `false` 
gives `false`, else any NULL gives NULL,
     else `true`). Like `array_any_match`, though, vectorized evaluation does 
not short-circuit, so it raises errors and
     calls volatile functions on elements Spark never visits. Handle that the 
same way as the planned `exists`.
   - The `lambda_utils.rs` helpers in `functions-nested` are `pub(crate)`, so 
equivalents have to live in this crate
     (they could be shared with `exists`).
   - Add `crates/sqllogictest/test_files/lambda/forall.slt` and document the 
function with `#[user_doc]`.
   
   ---
   _This issue was generated by an LLM (Claude) from the Apache Spark source 
code and tests. Examples come from Spark's `ExpressionDescription` examples, 
test suites and `.sql.out` golden files, but please verify them against Spark 
before relying on them._


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to