yinli-systems opened a new pull request, #25300: URL: https://github.com/apache/datafusion/pull/25300
## Which issue does this PR close? - Related to https://github.com/apache/datafusion-comet/issues/5708 (tracked under https://github.com/apache/datafusion-comet/issues/5709). ## Rationale for this change A sliced `ListArray` can retain a child values array that is much larger than its logical range. The floating-point set-operation paths currently call `normalize_float_zero()` on that full child array before slicing it to the visible offsets. When the backing array contains `-0.0`, this scans and allocates in proportion to hidden backing data rather than the values the query can observe. This is particularly expensive for small slices of large arrays. In the added Criterion benchmark, `array_distinct` over two visible values backed by 1,048,576 `Float64` values takes 184.46 us on the parent commit and 0.823 us with this change (about 224x faster). This is a component benchmark, not an end-to-end query speedup. ## What changes are included in this PR? - Slice each list's visible child range before floating-point zero normalization in `array_distinct`, `array_union`, `array_intersect`, and `array_except`. - Keep row and output indices relative to the normalized slice, including the right-hand concatenation offset used by union and intersection. - Preserve the existing full-child path when the entire child array is visible. - Add a benchmark that independently varies backing-buffer size and visible-range size, plus an unsliced control. - Add regression coverage for non-zero offsets, `List` and `LargeList`, signed zero, repeated NaNs, null elements, null list rows, and an empty visible list. Benchmark results on the same machine and parent commit (`c149764`), using Criterion's 100-sample estimates: | Input | Parent | This PR | |---|---:|---:| | 2 visible values, 1,024 backing values | 1.024 us | 0.858 us | | 2 visible values, 1,048,576 backing values | 184.46 us | 0.823 us | | 2,048 visible values, 1,048,576 backing values | 167.27 us | 12.50 us | With two visible values, the new implementation remains approximately flat as the backing array grows from 1,024 to 1,048,576 values. Increasing the visible range still increases runtime as expected. ## What is the testing strategy for this PR? The new unit tests exercise all affected set operations and make the pre-slice and in-slice values intentionally different so incorrect absolute/local index conversion is observable. Validation completed locally: - `cargo test -p datafusion-functions-nested` (127 tests and 2 doctests) - Extended workspace test command from `AGENTS.md`, including all 519 sqllogictest files - `cargo clippy --all-targets --all-features -- -D warnings` - `cargo fmt --all -- --check` - `cargo bench -p datafusion-functions-nested --bench array_set_ops -- array_distinct_sliced_float` ## Are there any user-facing changes? There are no API or result-semantics changes. Sliced floating-point list set operations avoid work and allocation for child values outside their logical range while retaining the existing signed-zero, NaN, and null behavior. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
