Sunninsky commented on PR #27410: URL: https://github.com/apache/datafusion/pull/27410#issuecomment-6923095214
Thanks @alamb, that is fair: 4.7k lines is too much to review, and the description starts with the implementation instead of the problem. I will split this up and rewrite the descriptions. The problem, from a user's side: window queries with an `ORDER BY` take far more time per row than the arithmetic needs. On `main`, with 60,000-row partitions, `SUM(v) OVER (PARTITION BY k ORDER BY ts)` costs 286 ns of CPU per input row (3.8 ns of that is the scan and the final aggregate), and with `RANGE BETWEEN 5 PRECEDING AND 5 FOLLOWING` it costs 779 ns. With 10-row partitions even `row_number()` costs 92 ns. The prototype here brings these to 15, 21 and 6 ns. The reproducer is in #27400. The design adds no new operator and changes no plans or SQL semantics: the frames and results of a run of rows are computed in one call instead of one row at a time, through optional methods whose default is today's per-row code, so a function that does not implement them runs exactly as it does today. Plan: a first PR that only adds these shapes to the `window_query_sql` benchmark (about 130 lines), so the cost is visible on `main`; then one optimization per PR, a few hundred lines each, with its own tests, a before/after number, and a description that starts from the query. Whatever overlaps #27466 will wait for it. I will move this PR back to draft and keep it as the reference for the whole picture. If you would prefer a different first step, tell me and I will change the order. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
