comphead opened a new pull request, #27466:
URL: https://github.com/apache/datafusion/pull/27466

   ## Which issue does this PR close?
   
   - Closes #27463.
   
   This builds on #27462 and includes its two commits. Only the last commit is 
new. I will rebase it once #27462 is merged.
   
   ## Rationale for this change
   
   With about 2 rows per partition, `last_value(x) IGNORE NULLS OVER (PARTITION 
BY ... ORDER BY ...)` spends most of its time in `BoundedWindowAggExec` on 
per-partition work: a key, a buffered batch and a state per partition, and a 
`take` of the argument per partition. #27462 skips this work for aggregate 
windows only.
   
   ## What changes are included in this PR?
   
   `first_value`, `last_value` and `nth_value` set a new 
`PartitionEvaluator::evaluates_frames_independently` flag, because their 
`evaluate_ranges` result for a frame depends only on that frame. For such 
functions the new `WindowExpr::evaluate_partitions` computes the frames of 
every row of a run of complete partitions and evaluates them with one 
`evaluate_ranges` call, so there is one `take` per batch instead of one per 
partition. `BoundedWindowAggExec` evaluates the complete partitions of these 
functions this way, and `WindowAggExec` uses the same call for all partitions 
of its input.
   
   | Reproducer of #27463, 5M rows, about 2 rows per partition, 
`elapsed_compute` | `main` | PR |
   |---|---:|---:|
   | `last_value(s) IGNORE NULLS` on a struct, 10 carried columns | 5.5 s | 
0.68 s |
   | `last_value(i) IGNORE NULLS` on an int, 10 carried columns | 3.0 s | 0.57 
s |
   | the struct query over `ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED 
FOLLOWING` (`WindowAggExec`) | 3.8 s | 0.27 s |
   
   Complete partitions are no longer sliced, so carried columns now cost 
little: the struct query takes 0.66 s without them. In the `window_query_sql` 
benchmark, `window partition and order by, u64_wide, built-in functions` goes 
from 418 ms to 122 ms and `window partition by, u64_wide, built-in functions` 
from 224 ms to 94 ms. The other cases stay within noise.
   
   ## What is the testing strategy for this PR?
   
   A `window.slt` case at `batch_size = 8` has several complete partitions in 
each batch, ORDER BY values repeated across partitions, NULLs, and ROWS, RANGE 
and GROUPS frames next to an aggregate. A frame that leaves its partition, a 
frame state carried over from the previous partition, or a result in the wrong 
column changes its output.
   
   ## Are there any user-facing changes?
   
   No. `PartitionEvaluator::evaluates_frames_independently` and 
`WindowExpr::evaluate_partitions` are new provided methods. A user-defined 
window function keeps the current path unless it sets the flag.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to