freakyzoidberg opened a new issue, #23334: URL: https://github.com/apache/datafusion/issues/23334
## Is your feature request related to a problem or challenge? `array_has(array, element)` returns, for each row, whether `array` contains `element`. When `element` is a **scalar literal**, DataFusion has a fast path (added in #20374). But when `element` is an column, e.g. `array_has(t1.tags, t2.key)` used as a join filter, execution goes through `array_has_dispatch_for_array` (the `ColumnarValue::Array` needle branch in `datafusion/functions nested/src/array_has.rs`). That branch compares each row by invoking the Arrow `eq` comparison kernel **once per row**. Every invocation allocates a `BooleanArray` and pays downcast + dispatch overhead, so for an `N`-row batch it is `N` kernel calls and `N` allocations, the cost is dominated by fixed per-row overhead, not the element comparison itself. This is a common shape in real workloads: matching tags/labels/keys between two tables lowers to an array-needle `array_has` join filter (e.g. a `NestedLoopJoinExec` with `filter=array_has(tags, key)`). It was a visible fraction of one such profile even after the join's deep-copy was fixed in #18070 / #18161. ## Describe the solution you'd like Performance improvement for at least primitives, non-nulls, and/or more ## Proposition/draft of improvement PR with bench: xxx PR with perf: xxx ## Additional context Related: #20374, #18070 / #18161, #18727. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
