ologlogn opened a new pull request, #21895:
URL: https://github.com/apache/datafusion/pull/21895
## Which issue does this PR close?
- Closes #.
<!-- No existing issue — follow-on to the `array_transform` higher-order
function added in #21679. -->
## Rationale for this change
`array_transform` was recently added as the first `HigherOrderUDF`.
`array_filter` is the natural companion: filter elements of an array using a
boolean lambda, analogous to Spark's `filter` / `exists` functions and DuckDB's
`list_filter`.
## What changes are included in this PR?
- New `HigherOrderUDF` `ArrayFilter` in
`datafusion/functions-nested/src/array_filter.rs`
- `array_filter(array, x -> condition)` with alias `list_filter`
- Evaluates a boolean lambda per element; elements where the lambda
returns `true` (or non-null `true`) are kept
- Null predicate results are treated as `false` (element dropped),
matching Spark semantics
- Handles `List` and `LargeList`, sliced arrays, and null sublists
correctly
- Core logic: builds a per-element selection mask + recomputes per-sublist
offsets using `arrow::compute::filter`
- Registered in `all_default_higher_order_functions()` and `expr_fn`
## Are these changes tested?
Yes, unit tests in `array_filter.rs` cover:
- Basic filtering
- Multiple sublists with different filter counts
- Sliced list arrays (unreachable values not evaluated)
- Null sublists (predicate not evaluated; null preserved in output)
- All-filtered-out (empty output sublist)
## Are there any user-facing changes?
Yes — `array_filter(array, lambda)` and its alias `list_filter(array,
lambda)` are now available as SQL functions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
