yashrb24 opened a new issue, #24978:
URL: https://github.com/apache/datafusion/issues/24978
### Describe the bug
`array_has_any` and `array_has_all` drop top-level input NULLs when every
row in the needle array is NULL or empty. In that batch shape, nullable rows
return `false` / `true` instead of NULL.
The result can depend on another row in the same batch: adding one non-empty
needle row disables the fast path and restores the expected NULL result.
This reproduces on DataFusion 52.1.0 and current `main` at
`722cbf2e7ad9a43bc6723f7795fe87e0552eb81f`.
### To Reproduce
Run with DataFusion CLI:
```sql
WITH t(id, haystack, needle) AS (
VALUES
(1, [1, 2], arrow_cast(NULL, 'List(Int64)')),
(2, arrow_cast(NULL, 'List(Int64)'), arrow_cast([], 'List(Int64)')),
(3, [3, 4], arrow_cast([], 'List(Int64)'))
)
SELECT array_has_any(haystack, needle),
array_has_all(haystack, needle)
FROM t
ORDER BY id;
```
Actual output:
```text
false true
false true
false true
```
As a control, put a non-empty needle in another row:
```sql
WITH t(haystack, needle) AS (
VALUES
([1, 2], arrow_cast(NULL, 'List(Int64)')),
([3, 4], [9])
)
SELECT array_has_any(haystack, needle) FROM t;
```
This returns `NULL`, `false`, showing that the nullable row changes with the
batch contents.
### Expected behavior
Rows where either input list is NULL should return NULL. Non-null rows with
an empty needle should keep the existing empty-set results: `false` for
`array_has_any` and `true` for `array_has_all`.
For the first query, the expected output is:
```text
NULL NULL
NULL NULL
false true
```
### Additional context
The empty-needle fast path constructs a `BooleanArray` without the input
null mask, while the general and string paths preserve the union of the input
null masks. The affected code is unchanged from upstream 52.1.0 in the
downstream fork.
This was found while wiring `ARRAYS_OVERLAP` in
e6data/e6-native-executor#818. A downstream fix is in e6data/e6-datafusion#223.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
