comphead opened a new pull request, #27466: URL: https://github.com/apache/datafusion/pull/27466
## Which issue does this PR close? - Closes #27463. This builds on #27462 and includes its two commits. Only the last commit is new. I will rebase it once #27462 is merged. ## Rationale for this change With about 2 rows per partition, `last_value(x) IGNORE NULLS OVER (PARTITION BY ... ORDER BY ...)` spends most of its time in `BoundedWindowAggExec` on per-partition work: a key, a buffered batch and a state per partition, and a `take` of the argument per partition. #27462 skips this work for aggregate windows only. ## What changes are included in this PR? `first_value`, `last_value` and `nth_value` set a new `PartitionEvaluator::evaluates_frames_independently` flag, because their `evaluate_ranges` result for a frame depends only on that frame. For such functions the new `WindowExpr::evaluate_partitions` computes the frames of every row of a run of complete partitions and evaluates them with one `evaluate_ranges` call, so there is one `take` per batch instead of one per partition. `BoundedWindowAggExec` evaluates the complete partitions of these functions this way, and `WindowAggExec` uses the same call for all partitions of its input. | Reproducer of #27463, 5M rows, about 2 rows per partition, `elapsed_compute` | `main` | PR | |---|---:|---:| | `last_value(s) IGNORE NULLS` on a struct, 10 carried columns | 5.5 s | 0.68 s | | `last_value(i) IGNORE NULLS` on an int, 10 carried columns | 3.0 s | 0.57 s | | the struct query over `ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING` (`WindowAggExec`) | 3.8 s | 0.27 s | Complete partitions are no longer sliced, so carried columns now cost little: the struct query takes 0.66 s without them. In the `window_query_sql` benchmark, `window partition and order by, u64_wide, built-in functions` goes from 418 ms to 122 ms and `window partition by, u64_wide, built-in functions` from 224 ms to 94 ms. The other cases stay within noise. ## What is the testing strategy for this PR? A `window.slt` case at `batch_size = 8` has several complete partitions in each batch, ORDER BY values repeated across partitions, NULLs, and ROWS, RANGE and GROUPS frames next to an aggregate. A frame that leaves its partition, a frame state carried over from the previous partition, or a result in the wrong column changes its output. ## Are there any user-facing changes? No. `PartitionEvaluator::evaluates_frames_independently` and `WindowExpr::evaluate_partitions` are new provided methods. A user-defined window function keeps the current path unless it sets the flag. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
