andygrove opened a new issue, #249:
URL: https://github.com/apache/datafusion-spark/issues/249
Part of #176 and apache/datafusion-comet#7300.
### What the function does
`forall(expr, pred)` returns whether `pred` holds for every element of the
array `expr`. `pred` must be a
one-argument lambda returning `BOOLEAN`. A non-boolean body, or a non-lambda
literal such as
`forall(array(1, 2, 3), 1)`, fails with
`DATATYPE_MISMATCH.UNEXPECTED_INPUT_TYPE`. There is no index argument. The
result type is `BOOLEAN`, nullable when `argument.nullable ||
function.nullable`. Available since Spark 3.0.0. No
aliases.
Semantics (`ArrayForAll`; unlike `exists`, there is no legacy switch, and
three-valued logic always applies):
- A NULL array gives NULL.
- Elements are visited in order, and evaluation **stops at the first
`false`**.
- If any element gives `false`, the result is `false`.
- Otherwise, if any element gave NULL, the result is NULL.
- Otherwise the result is `true`. An empty array gives `true`.
### Examples
```sql
SELECT forall(array(1, 2, 3), x -> x % 2 == 0);
-- false
SELECT forall(array(2, 4, 8), x -> x % 2 == 0);
-- true
SELECT forall(array(1, null, 3), x -> x % 2 == 0);
-- false
SELECT forall(array(2, null, 8), x -> x % 2 == 0);
-- NULL
```
These come from the `ExpressionDescription`. `HigherOrderFunctionsSuite`
adds the following cases:
| input | predicate | result |
|---|---|---|
| `[2,4,8]` | `x IS NULL OR x % 2 == 1` | false |
| `[2,4,8]` | `NULL` | NULL |
| `[1,null,3]` | `x IS NULL OR x % 2 == 1` | true |
| `[1,null,3]` | `NULL` | NULL |
| `[2,null,8]` | `false` | false |
| `NULL` | any | NULL |
| `['a0','a1','a2','a3']` | `startswith(x, 'a')` | true |
| `[null,'b','c']` | `startswith(x, 'a')` | false |
Nested: `transform(aai, a -> forall(a, x -> x IS NULL OR x % 2 == 1))` on
`[[1,3,null],null,[4,5]]` gives
`[true,null,false]`.
### Edge cases to test
- A NULL array gives NULL. An empty array gives `true`. Also test NULL
elements, and a predicate that is NULL for
every element (NULL result).
- The NULL-versus-`false` ordering. `[null, 'b', 'c']` with `startswith(x,
'a')` gives `false`: the first element gives
NULL, the second gives `false`, and `false` wins.
- Short-circuiting. Evaluation stops at the first `false`. By code reading,
`forall(array(-1, 0), x -> 1 / x > 0)` returns `false` even with ANSI on,
because `1 / 0` is never evaluated, while
`forall(array(1, 0), x -> 1 / x > 0)` raises `DIVIDE_BY_ZERO` with ANSI on.
- Captured outer columns and nondeterministic predicates, evaluated per
element in order.
- Element types: primitives, strings, structs, and nested arrays.
- Lambda arity: `(x, i) -> ...` fails with
`INVALID_LAMBDA_FUNCTION_CALL.NUM_ARGS_MISMATCH`.
### Differences across Spark versions
- None found between 3.4 and 4.2. ANSI mode is on by default from 4.0, which
matters for errors inside the
predicate.
### Implementation notes
- Spark: `ArrayForAll` in
`sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/higherOrderFunctions.scala`.
- Infrastructure. DataFusion's higher-order function framework
(`HigherOrderUDFImpl` in
`datafusion/expr/src/higher_order_function.rs`, and
`LambdaArgument::evaluate` with `spread_captures` for captured
columns) is on `branch-56`, which this crate uses. This crate's
`crates/spark/src/function/lambda/mod.rs` is an
empty `ScalarUDF` placeholder. It needs a `HigherOrderUDF` list and
registration through
`register_higher_order_function` /
`SessionStateBuilder::higher_order_functions`.
- DataFusion core has no `forall` / `all_match`. `array_any_match`
(`datafusion/functions-nested/src/array_any_match.rs`)
is the mirror image and shows the framework pattern: evaluate the
predicate over the flattened values, then reduce
each row's range. A Spark `forall` can use the same reduction (any `false`
gives `false`, else any NULL gives NULL,
else `true`). Like `array_any_match`, though, vectorized evaluation does
not short-circuit, so it raises errors and
calls volatile functions on elements Spark never visits. Handle that the
same way as the planned `exists`.
- The `lambda_utils.rs` helpers in `functions-nested` are `pub(crate)`, so
equivalents have to live in this crate
(they could be shared with `exists`).
- Add `crates/sqllogictest/test_files/lambda/forall.slt` and document the
function with `#[user_doc]`.
---
_This issue was generated by an LLM (Claude) from the Apache Spark source
code and tests. Examples come from Spark's `ExpressionDescription` examples,
test suites and `.sql.out` golden files, but please verify them against Spark
before relying on them._
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]