neilconway opened a new pull request, #24532: URL: https://github.com/apache/datafusion/pull/24532
## Which issue does this PR close? - Related to #23982 ## Rationale for this change In Linear mode, `BoundedWindowAggStream` does an evaluation sweep that visits every live partition for every window expression on every input batch. For standard (non-aggregate) window functions (e.g., row_number, rank, lag/lead, and nth_value), a partition that received no rows since the previous evaluation pass can be skipped: evaluating a function only depends on the contents of the partition itself. New rows are detected with a received-row counter on `PartitionBatchState` that each expression's `WindowAggState` snapshots at the end of every evaluation pass. This test is deliberately different from the "every buffered row has a result" test used to skip aggregate window expressions (#24127). Aggregate evaluation also consults the ORDER BY values of the most recent input row across all partitions, which can produce new results for a partition whose own input is unchanged. Benchmarks: - linear dense count 100: 44.4 ms -> 44.0 ms (-0.8%) - linear dense count 10000: 161.9 ms -> 160.7 ms (-0.9%) - linear sparse count 32768: 89.3 ms -> 88.4 ms (-0.9%) - linear dense count rows-frame: 136.4 ms -> 136.1 ms (~noise) - linear dense count+sum 10000: 253.1 ms -> 235.3 ms (-7.0%) - linear dense row_number 10000: 114.7 ms -> 110.9 ms (-3.2%) - linear sparse row_number 32768: 73.5 ms -> 42.2 ms (-42.5%) - linear sparse lead 32768: 108.5 ms -> 46.6 ms (-57.0%) - linear dense rank 10000: 125.8 ms -> 121.5 ms (-3.4%) - sorted count 10000: 34.5 ms -> 33.4 ms (-3.6%) ## What changes are included in this PR? * Add counter to track # of received rows per partition * Use received-row-count to avoid redundant evaluation of standard window functions * Add tests ## Are these changes tested? Yes; new tests added. ## Are there any user-facing changes? No. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
