andygrove opened a new issue, #5269:
URL: https://github.com/apache/datafusion-comet/issues/5269
## Describe the bug
`arrays_overlap` over `array<array<double>>` panics in native code when both
sides of the comparison are expression-constructed nested lists rather than
plain columns.
```
org.apache.comet.CometNativeException: native panic: index out of bounds:
the len is 2 but the index is 2
Comet native panic: panicked at
arrow-array-58.4.0/src/array/list_array.rs:352:19
```
`list_array.rs:352` is `GenericListArray::value`, so something in
`arrays_overlap_list_generic`
(`native/spark-expr/src/array_funcs/arrays_overlap.rs`) indexes past the end of
one of its two list arrays.
## Steps to reproduce
As a Comet SQL file test:
```sql
statement
CREATE TABLE t_panic(a array<array<double>>) USING parquet
statement
INSERT INTO t_panic VALUES
(array(array(-0.0))),
(array(array(0.0))),
(array(array(1.0, -0.0))),
(array(array(double('NaN')))),
(array(array(1.0))),
(array(array(-0.0), cast(NULL as array<double>))),
(array(cast(NULL as array<double>)))
query
SELECT arrays_overlap(array(array(a[0][0])), array(array(cast(0.0 as
double)))) FROM t_panic
```
Reproduced on `apache/main` at `f262b13d2`, Spark 4.1 profile, JDK 17, macOS.
## What I could and could not narrow down
The panic needs all seven rows. None of these reproduce it on their own:
- any single row from the table, including the two rows carrying a `NULL`
inner list
- a two-row table combining a multi-element inner list with a `NULL` inner
list
- the `int` equivalent of the same query
- `arrays_overlap(a, array(array(cast(0.0 as double))))`, a plain column
against a constant
- `arrays_overlap(array(a[0]), array(array(0)))`, only the outer list
constructed
The failing task index moves with the number of rows, so this looks like a
batch or partition boundary interacting with the per-row `left.value(i)` /
`right.value(i)` slicing in `arrays_overlap_list_generic`, rather than anything
about a specific row's contents. Worth checking whether the two sides can end
up with different `len()` when one of them arrives as a broadcast scalar and
the other is a per-row constructed array.
## Additional context
Found while reviewing #5235, which changes the same function. The panic
reproduces identically with and without that PR applied, so it is independent
of it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
