mbutrovich opened a new issue, #27598: URL: https://github.com/apache/datafusion/issues/27598
### Is your feature request related to a problem or challenge? Several `datafusion-functions-nested` kernels count set bits with one call per list row. Each call builds a new bit chunk, and all but `array_compact` also slice a buffer or array per row, which clones its `Arc`: - `array_remove` with a scalar needle ([remove.rs](https://github.com/apache/datafusion/blob/c6c1cc31d2f433930c9b4f3ca9fea023846ac276/datafusion/functions-nested/src/remove.rs#L575-L576)) - `inner_product` ([inner_product.rs](https://github.com/apache/datafusion/blob/c6c1cc31d2f433930c9b4f3ca9fea023846ac276/datafusion/functions-nested/src/inner_product.rs#L183-L185)) - `cosine_distance` ([cosine_distance.rs](https://github.com/apache/datafusion/blob/c6c1cc31d2f433930c9b4f3ca9fea023846ac276/datafusion/functions-nested/src/cosine_distance.rs#L177-L179)) - `array_normalize` ([array_normalize.rs](https://github.com/apache/datafusion/blob/c6c1cc31d2f433930c9b4f3ca9fea023846ac276/datafusion/functions-nested/src/array_normalize.rs#L151-L152)) - `array_compact` for non-primitive values ([array_compact.rs](https://github.com/apache/datafusion/blob/c6c1cc31d2f433930c9b4f3ca9fea023846ac276/datafusion/functions-nested/src/array_compact.rs#L219-L220)) `array_filter` already avoids this with a private [`SetBitCounter`](https://github.com/apache/datafusion/blob/c6c1cc31d2f433930c9b4f3ca9fea023846ac276/datafusion/functions-nested/src/utils.rs#L110-L147). apache/arrow-rs#12346 adds a `SetBitCounter` to `arrow-buffer`. It takes the bitmap instead of a word iterator, so callers can't pass an unpadded final word, and it returns correct counts when a position decreases instead of looping. @neilconway's apache/datafusion#26507 used a public DataFusion counter at these call sites, with benchmarks for each in its description. It was closed so the counter could move to arrow-rs. ### Describe the solution you'd like Once DataFusion depends on an arrow release with `SetBitCounter`, replace the private counter with it and use it at the call sites above. The commits of #26507 are a starting point: its benchmark commits ([c3d5ca8](https://github.com/apache/datafusion/pull/26507/commits/c3d5ca84b6382b65b89a0efa698e008fa4acf610)) and the call site changes ([18b2c43](https://github.com/apache/datafusion/pull/26507/commits/18b2c43a99232651929b36a71f7ece3d036f441a), [6225dfc](https://github.com/apache/datafusion/pull/26507/commits/6225dfc43b1d43c9c599ee994a611875a483f368)). The review of #26507 asked for these changes, which still apply: - Skip counting when the element validity has no nulls, because `nulls()` can return an all-valid buffer (https://github.com/apache/datafusion/pull/26507#discussion_r4318870152). Add an all-valid `NullBuffer` case to the `vector_math` benchmark, because its `no_nulls` cases pass `None` and never reach the counter (https://github.com/apache/datafusion/pull/26507#discussion_r4340399645). - In `inner_product` and `cosine_distance`, OR the two arguments' row bitmaps once before the loop instead of looking up both per row (https://github.com/apache/datafusion/pull/26507#discussion_r4318870156). - Name the new tests `test_sliced_capacity` like their siblings, and add multi-row `.slt` cases with a NULL element in only the first argument and in only the second (https://github.com/apache/datafusion/pull/26507#discussion_r4318870163). - Build the counter once above the branch in `prepare_list_filter` (https://github.com/apache/datafusion/pull/26507#pullrequestreview-5577773030). ### Describe alternatives you've considered Making the private counter public in `datafusion-common` would duplicate the arrow-rs type, and `datafusion-spark` would need it too. ### Additional context Routing `array_compact` through `prepare_list_filter` made it slower when NULL list rows still store elements: `parent_nulls/utf8/mostly` was 42% slower in #26507. A PR that includes that change has to measure and report it again. In the arrow-rs benchmark, rows of exactly 256 elements starting on a word boundary are 0.73x with the counter on x86 at the default target, and 1.10x with `-C target-cpu=x86-64-v3` (apache/arrow-rs#12347). The `vector_math` `width_256` cases cover that shape. ### Willingness to contribute I plan to contribute to this feature independently -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
