This is an automated email from the ASF dual-hosted git repository.
github-merge-queue[bot] pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/datafusion.git
The following commit(s) were added to refs/heads/main by this push:
new 47ca49043a perf: skip re-slicing window partition batches with nothing
to prune (#24047)
47ca49043a is described below
commit 47ca49043aeb2da3cc236940d408983517344ba8
Author: Neil Conway <[email protected]>
AuthorDate: Mon Aug 3 19:08:11 2026 -0400
perf: skip re-slicing window partition batches with nothing to prune
(#24047)
## Which issue does this PR close?
- Part of #23982
## Rationale for this change
After each emission, BoundedWindowAggStream prunes the rows of each
partition's buffered batch that no longer contribute to any window
frame. The prune loop re-sliced every live partition's RecordBatch even
when there was nothing to prune, allocating fresh array metadata for an
identical batch and then dropping the replaced one.
Profiling the many-partitions benchmark showed prune_state at ~39% of
the sparse case, where nearly all live partitions are quiet and almost
every slice is a no-op, and ~21% of the dense case, where most slices
prune real rows and only the partitions missed by a batch hit the no-op
path.
One simple improvement is to leave a partition's buffered batch
untouched when the prune count is zero. This avoids the array metadata
allocation churn mentioned above.
Benchmarks:
```
- linear, range, single, 100 part, dense: 44.1 ms -> 43.8 ms (~noise)
- linear, range, single, 10000 part, dense: 171.1 ms -> 164.8 ms (-3.6%)
- linear, range, single, 32768 part, sparse: 209.4 ms -> 166.1 ms (-20.6%)
- linear, rows, single, 10000 part, dense: 141.5 ms -> 136.4 ms (-3.7%)
- linear, range, multi, 10000 part, dense: 268.0 ms -> 261.0 ms (-2.6%)
- sorted, range, single, 10000 part: 34.0 ms -> 34.3 ms (+0.9%)
```
## What changes are included in this PR?
See above.
## Are these changes tested?
Covered by existing tests.
## Are there any user-facing changes?
No.
---
datafusion/physical-plan/src/windows/bounded_window_agg_exec.rs | 7 ++++++-
1 file changed, 6 insertions(+), 1 deletion(-)
diff --git a/datafusion/physical-plan/src/windows/bounded_window_agg_exec.rs
b/datafusion/physical-plan/src/windows/bounded_window_agg_exec.rs
index 07751a70ec..680d5b657e 100644
--- a/datafusion/physical-plan/src/windows/bounded_window_agg_exec.rs
+++ b/datafusion/physical-plan/src/windows/bounded_window_agg_exec.rs
@@ -1188,10 +1188,15 @@ impl BoundedWindowAggStream {
// Retract no longer needed parts during window calculations from
partition batch:
for (partition_row, n_prune) in n_prune_each_partition.iter() {
let pb_state = &mut self.partition_buffers[partition_row];
+ pb_state.n_out_row = 0;
+
+ // If there is nothing to prune, leave the batch as-is
+ if *n_prune == 0 {
+ continue;
+ }
let batch = &pb_state.record_batch;
pb_state.record_batch = batch.slice(*n_prune, batch.num_rows() -
n_prune);
- pb_state.n_out_row = 0;
// Update state indices since we have pruned some rows from the
beginning:
for window_agg_state in self.window_agg_states.iter_mut() {
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
