andygrove commented on PR #6279: URL: https://github.com/apache/datafusion-comet/pull/6279#issuecomment-6957364583
We checked whether this is still needed once the DataFusion 56 upgrade (#7249) merges, and it is. #7249 pins DataFusion `branch-56` at `964f925`. Nothing under `datafusion/physical-plan/src/windows/` there reserves memory or spills (no `MemoryReservation`, `MemoryConsumer`, `try_grow` or `DiskManager` in `window_agg_exec.rs`, `bounded_window_agg_exec.rs` or the rest of the directory). So on 56, `WindowAggExec` still buffers the task's whole input without telling the pool, which is the gap this PR and #6253 describe. Spilling for `WindowAggExec` is still apache/datafusion#22946, which is open. Even once that lands, this PR's operator is likely to stay useful. The memory savings come from emitting each window partition as soon as it completes, and the upstream issue keeps the existing model of buffering up to the whole window partition. We haven't read an upstream design that streams completed window partitions out, so that is worth rechecking when #22946 gets a PR. We expect this to rebase cleanly onto #7249, since that PR doesn't touch the window planner code or `window_agg.rs`. We haven't tried the rebase. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
