comphead opened a new pull request, #27462:
URL: https://github.com/apache/datafusion/pull/27462

   ## Which issue does this PR close?
   
   - Part of #26541.
   
   ## Rationale for this change
   
   In `Sorted` mode, `BoundedWindowAggExec` builds a key, a buffered batch and 
a state per window expression for every partition, and prunes them again. When 
partitions hold one or a few rows, this bookkeeping dominates. The issue has a 
reproducer.
   
   ## What changes are included in this PR?
   
   When every window expression is an aggregate, partitions that start and end 
inside an input batch are evaluated with `WindowExpr::evaluate` on their slice 
of the batch. For an aggregate this is the calculation `evaluate_stateful` does 
for a whole partition, without the key, buffer or state. Only the first and 
last partition of a batch are buffered, and the results of the others are 
spliced in between them. A stream with a `WindowStateObserver` keeps the 
current path, since it needs the state of every partition.
   
   | 1M rows, `target_partitions = 1`, `datafusion-cli` (`release-nonlto`) | 
`main` | PR |
   |---|---:|---:|
   | `max(v) OVER (PARTITION BY k ORDER BY i)`, one row per partition | 0.61 s 
| 0.33 s |
   | the same with 10 carried string columns | 0.93 s | 0.56 s |
   | `sum(v)` over `ROWS BETWEEN 1 PRECEDING AND 1 FOLLOWING`, one row per 
partition | 0.56 s | 0.28 s |
   
   In the `window_query_sql` benchmark, `window partition and order by, 
u64_wide, aggregate functions` goes from 443 ms to 318 ms, and the other cases 
stay within noise.
   
   ## What is the testing strategy for this PR?
   
   - A `window.slt` case at `batch_size = 4`, where a batch starts inside a 
buffered partition, holds a complete one and ends inside another.
   - The observer exactly-once test gets a partition inside a batch, which must 
still be observed.
   - A UDWF test pins the evaluator calls for a partition inside a batch.
   
   ## Are there any user-facing changes?
   
   No. Known limitation: non-aggregate window functions keep the current path. 
Their `evaluate` calls other `PartitionEvaluator` methods than the stream does, 
and evaluating each partition with `evaluate_stateful` made `row_number` and 
`lag` slower.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to