andygrove commented on PR #6279:
URL: 
https://github.com/apache/datafusion-comet/pull/6279#issuecomment-6957364583

   We checked whether this is still needed once the DataFusion 56 upgrade 
(#7249) merges, and it is.
   
   #7249 pins DataFusion `branch-56` at `964f925`. Nothing under 
`datafusion/physical-plan/src/windows/` there reserves memory or spills (no 
`MemoryReservation`, `MemoryConsumer`, `try_grow` or `DiskManager` in 
`window_agg_exec.rs`, `bounded_window_agg_exec.rs` or the rest of the 
directory). So on 56, `WindowAggExec` still buffers the task's whole input 
without telling the pool, which is the gap this PR and #6253 describe. Spilling 
for `WindowAggExec` is still apache/datafusion#22946, which is open.
   
   Even once that lands, this PR's operator is likely to stay useful. The 
memory savings come from emitting each window partition as soon as it 
completes, and the upstream issue keeps the existing model of buffering up to 
the whole window partition. We haven't read an upstream design that streams 
completed window partitions out, so that is worth rechecking when #22946 gets a 
PR.
   
   We expect this to rebase cleanly onto #7249, since that PR doesn't touch the 
window planner code or `window_agg.rs`. We haven't tried the rebase.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to