andygrove opened a new issue, #248:
URL: https://github.com/apache/datafusion-spark/issues/248
Part of #176 and apache/datafusion-comet#7300.
### What the function does
`exists(expr, pred)` returns whether `pred` holds for at least one element
of the array `expr`. `pred` must be a
one-argument lambda returning `BOOLEAN`. A non-boolean body fails with
`DATATYPE_MISMATCH.UNEXPECTED_INPUT_TYPE`
(`requiredType "BOOLEAN"`), and so does a non-lambda literal such as
`exists(array(1, 2, 3), 1)`. There is no index
argument, so `(x, i) -> ...` fails with
`INVALID_LAMBDA_FUNCTION_CALL.NUM_ARGS_MISMATCH`. The result type is
`BOOLEAN`. Available since Spark 2.4.0. No aliases.
Semantics (`ArrayExists`, with `followThreeValuedLogic` captured from
`spark.sql.legacy.followThreeValuedLogicInArrayExists`, default `true`):
- A NULL array gives NULL.
- Elements are visited in order, and evaluation **stops at the first
`true`**.
- If any element gives `true`, the result is `true`.
- Otherwise, if any element gave NULL and three-valued logic is on, the
result is NULL.
- Otherwise the result is `false`. An empty array gives `false`.
With the legacy config set to `false`, NULL predicate results count as
`false`, and the result is NULL only for a
NULL array. Nullability is `argument.nullable || function.nullable` with
three-valued logic, and only
`argument.nullable` without it.
### Examples
```sql
SELECT exists(array(1, 2, 3), x -> x % 2 == 0);
-- true
SELECT exists(array(1, 2, 3), x -> x % 2 == 10);
-- false
SELECT exists(array(1, null, 3), x -> x % 2 == 0);
-- NULL
SELECT exists(array(0, null, 2, 3, null), x -> x IS NULL);
-- true
SELECT exists(array(1, 2, 3), x -> x IS NULL);
-- false
SELECT exists(cast(null as array<int>), y -> y > 30);
-- NULL
```
These come from the `ExpressionDescription` and
`higher-order-functions.sql.out`.
`HigherOrderFunctionsSuite` covers both settings of the legacy config. In
the table, "3VL" means
`followThreeValuedLogic`.
| input | predicate | 3VL = true | 3VL = false |
|---|---|---|---|
| `[1,2,3]` | `x % 2 == 0` | true | true |
| `[1,2,3]` | `NULL` | NULL | false |
| `[1,null,3]` | `x % 2 == 0` | NULL | false |
| `[1,null,3]` | `x IS NULL OR x % 2 == 1` | true | true |
| `[1,null,3]` | `false` | false | false |
| `NULL` | any | NULL | NULL |
### Edge cases to test
- A NULL array, an empty array, NULL elements, and a predicate that returns
NULL for every element.
- Both values of `spark.sql.legacy.followThreeValuedLogicInArrayExists`. The
value is captured when the expression
is constructed. Also check the declared nullability under each setting.
- Short-circuiting. After the first `true`, Spark never evaluates the
predicate on later elements. By code reading,
`exists(array(1, 0), x -> 1 / x > 0)` returns `true` even with ANSI mode
on, because `1 / 0` is never evaluated.
With ANSI on and the order reversed (`array(0, 1)`), it raises
`DIVIDE_BY_ZERO`.
- Nested use: `transform(aai, a -> exists(a, x -> x IS NULL OR x % 2 == 1))`
on `[[1,2,3],null,[4,5]]` gives
`[true,null,true]`.
- Captured outer columns in the predicate (`exists(ys, y -> y > x)`), and
nondeterministic predicates
(`x -> rand() > 0.5`), which must be evaluated per element and in element
order.
- Strings and non-primitive elements, for example `exists(array('a0','b1'),
x -> startswith(x, 'a'))`, and nested
arrays.
- SQL parsing. sqlparser 0.63 treats `exists(` as a function call only for
`DatabricksDialect`. Under
`SparkSqlDialect`, which this crate's sqllogictest runner uses, `exists(`
is parsed as an `EXISTS (subquery)`
expression, so the `.slt` tests need a parser fix or a different dialect.
### Differences across Spark versions
- None found between 3.4 and 4.2. The legacy config has existed since 3.0
with default `true`. ANSI mode is on by
default from 4.0, which matters for errors inside the predicate.
### Implementation notes
- Spark: `ArrayExists` in
`sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/higherOrderFunctions.scala`.
The config is `LEGACY_ARRAY_EXISTS_FOLLOWS_THREE_VALUED_LOGIC` in
`SQLConf.scala`.
- Infrastructure. DataFusion's higher-order function framework
(`HigherOrderUDFImpl` in `datafusion/expr/src/higher_order_function.rs`,
and `LambdaArgument::evaluate` with
`spread_captures` for captured columns) is on `branch-56`, which this
crate uses. This crate's
`crates/spark/src/function/lambda/mod.rs` is an empty `ScalarUDF`
placeholder. It needs a `HigherOrderUDF` list and
registration through `register_higher_order_function` /
`SessionStateBuilder::higher_order_functions`.
- DataFusion core has `array_any_match` (aliases `any_match`,
`list_any_match`) in
`datafusion/functions-nested/src/array_any_match.rs`. It already follows
Spark's default three-valued logic
(true / NULL / false, and NULL for a NULL array). The differences are:
- It evaluates the predicate once over all flattened elements of the
batch, with no short-circuit. Errors in later
elements (ANSI arithmetic, casts) are raised where Spark returns `true`.
Nondeterministic predicates are also
evaluated on elements Spark would skip.
- It does not support the legacy `false` mode.
A Spark version can reuse its result reduction, but it needs
short-circuit-aware error handling. One option is to
evaluate vectorized and fall back to element-by-element evaluation when
the body is fallible or volatile.
- Add `crates/sqllogictest/test_files/lambda/exists.slt` and document the
function with `#[user_doc]`.
---
_This issue was generated by an LLM (Claude) from the Apache Spark source
code and tests. Examples come from Spark's `ExpressionDescription` examples,
test suites and `.sql.out` golden files, but please verify them against Spark
before relying on them._
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]