ranflarion opened a new issue, #27411:
URL: https://github.com/apache/datafusion/issues/27411
### Describe the bug
`CaseExpr::nullable` decides whether a nullable THEN branch is reachable by
replacing the THEN expression with NULL in its WHEN condition and evaluating
the result against a one-row batch with no columns. Two problems follow.
A condition without a column reference is evaluated even when it is
volatile. For `CASE WHEN random() < 0.5 THEN NULL ELSE a END` the reported
nullability depends on one draw of `random()`, so the projection's schema can
declare the field non-nullable and the batch that holds NULLs is then rejected.
A condition that still references a column fails `Column::evaluate`'s bounds
check, and the error is discarded as inconclusive. With the `backtrace` feature
and `RUST_BACKTRACE=1`, `internal_err!` captures and formats a backtrace every
time. `ScalarFunctionExpr::evaluate` re-derives its argument fields with
`return_field(batch.schema_ref())` on every batch, so a CASE argument of a
scalar function pays this once per batch, and on an AND, OR or NOT the error
path evaluates each side again and builds one error per side.
### To Reproduce
On main at 4c7b30752, `datafusion-cli` built with `--profile release-nonlto
--features backtrace`, 14 cores:
```sql
CREATE TABLE t(a INT NOT NULL) AS VALUES (1), (2), (3), (4), (5), (6), (7),
(8);
SELECT CASE WHEN random() < 0.5 THEN NULL ELSE a END AS x FROM t;
```
fails in 8 of 12 runs with
```
Arrow error: Invalid argument error: Column 'x' is declared as non-nullable
but contains null values
```
```sql
SELECT sum(abs(CASE WHEN value % 7 = 0 THEN nullif(value, 3) END)) FROM
generate_series(1, 1000000000);
```
takes 0.98 s with `RUST_BACKTRACE=0` and 6.96 s with `RUST_BACKTRACE=1`.
### Expected behavior
The first query returns 8 rows every time, with `x` nullable as the logical
schema already reports it. The second takes the same time with and without
`RUST_BACKTRACE=1`.
### Additional context
The logical side (`expr_schema.rs`) answers the same question with
`predicate_bounds::evaluate_bounds`, which derives three-valued bounds from the
expression's structure and never evaluates it. A physical equivalent would fix
both problems and also #26284, where the physical side cannot tell that `NULL <
b` is never true.
A smaller fix, the one we are applying to our DataFusion 55.2 fork,
evaluates a condition only when it is deterministic with literal leaves and
otherwise goes straight to the existing AND/OR/NOT bounds:
```rust
// Only a deterministic expression over literals has a value without input
rows
let constant = !predicate.exists(|e| {
Ok(e.is_volatile_node() || (e.children().is_empty() &&
!e.is::<Literal>()))
})?;
let result = match constant.then(|| predicate.evaluate(&batch)) {
None | Some(Err(_)) => { /* existing AND/OR/NOT handling */ }
Some(Ok(ColumnarValue::Array(array))) =>
ScalarValue::try_from_array(array.as_ref(), 0)?,
Some(Ok(ColumnarValue::Scalar(scalar))) => scalar,
};
```
On main this returns 8 rows in 12 of 12 runs of the first query, brings the
second to 0.96 s with `RUST_BACKTRACE=1`, and passes the `expressions::case`
tests. It does not fix #26284.
### Willingness to contribute
I plan to contribute a fix for this bug independently
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
