sunchao opened a new pull request, #6430: URL: https://github.com/apache/datafusion-comet/pull/6430
## Which issue does this PR close? No linked issue. ## Rationale for this change Native shuffle gathers each reducer's selected rows into an Arrow batch. Boolean columns, including aggregate-state flags, currently use Arrow's generic `MutableArrayData` interleave path. Rebuilding that state for scattered bit selections costs substantially more than gathering the values and validity bits directly. ## What changes are included in this PR? Gather scattered Boolean values and nulls directly, while using Arrow interleave for the other columns. Contiguous and clustered selections retain Arrow's range-copy path. Cache Boolean eligibility once per immutable producer flush so non-Boolean output chunks do not repeatedly inspect the schema. Partition assignments, row order, output schemas and the run-based iterator retain their existing behavior. ## How are these changes tested? - A differential regression exercises the actual producer across 40 combinations of mixed decimal/Boolean state, all-null and nullable columns, non-byte-aligned input slices, repeated/reordered rows, contiguous/clustered selections and output chunk boundaries. Each emitted batch is compared with Arrow interleave. - Native shuffle library tests: **172 passed**, zero failures or ignored tests, after cleaning all local Comet packages and recompiling this head. - `cargo clippy --locked -p datafusion-comet-shuffle --lib --tests -- -D warnings` passes. - Changed-file Rust formatting and `git diff --check` pass. A standalone component harness compares the exact base (`bd1d4d113cd4524b44927d59f10adc5791997a47`) Arrow interleave call with this head (`1280df420e66058b774882162bf16c4f8e27fabe`) helper and cached-eligibility dispatch in the same executable. Public Arrow **59.3.0** and all shared dependency versions match the repository lockfile. Rust 1.98.1 release build, thin LTO, one codegen unit, x86_64 AMD EPYC-Milan shared host, pinned to CPU 31. All **420 fixture cases** matched Arrow before timing. Each arm received 250 warmup calls, followed by seven paired rounds with alternating arm order and at least 25 ms per arm. The table gives median time per gather and median paired time change; these are descriptive component measurements. | Fixture, four input batches | Base | Head | Paired time change | | --- | ---: | ---: | ---: | | 64 scattered rows, one Boolean column | 1.777 us | 0.275 us | -84.15% | | 1,024 scattered rows, eight nullable Boolean columns | 194.480 us | 30.137 us | -84.48% | | 1,024 scattered rows, one nullable Boolean plus three Int64 columns | 30.332 us | 10.046 us | -67.55% | | 1,024 rows in eight-row clusters, eight nullable Boolean columns | 59.593 us | 60.102 us | +0.54% | | 1,024 scattered rows, four Int64 columns | 8.051 us | 8.104 us | +0.57% | The matrix includes 1/7/64/1,024/8,192 selected rows, one/four source batches, scattered/reversed/contiguous selections and eight-row clusters with shifted prefixes. Every case above +3% was repeated for nine 100 ms rounds. The twelve repeated controls ranged from -6.53% to +2.38%; the initial largest increase of +6.00% repeated at -3.94%. All adverse measurements are retained in the evidence bundle. Small controls vary on this shared host, and these measurements do not establish zero overhead. Timing includes gather dispatch, output construction and output destruction. Fixture construction, the once-per-flush eligibility check, index widening, partitioning, encoding, Spark/JNI and network are excluded. These measurements cover the gather kernel. Whole-shuffle and query performance remain unmeasured. Reproduction: the harness uses `interleave_record_batch` for the base and the unchanged `interleave_shuffle_batches` function plus `RowIterator::next`'s cached Boolean branch for the head. Helper SHA-256: `ecc8f74f6afb1856d272d7adb4c5e707ffb69f7fb72c1f791e2e71dc5f637f5f`. Build with `cargo build --release --locked`, then run `taskset -c 31 target/release/comet-boolean-interleave-evidence 7 25`. [Reproduction source, dependency lockfile, and complete current-base/head raw results](https://gist.github.com/sunchao/0a58a8e4c9a42f510a6bcdb15216542b). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
