wirybeaver commented on issue #22946: URL: https://github.com/apache/datafusion/issues/22946#issuecomment-4713593972
Calling out an important clarification from the discussion: the current `WindowAggExec` memory behavior is worse than only "large window partition may OOM". Current upstream behavior is: ```text buffer all input -> concat all input -> compute all partitions -> emit once ``` That means memory usage can scale with the full child input, even if every logical window partition is small. The proposed direction changes the execution model to: ```text buffer one active partition -> spill it if needed -> compute completed partition -> emit partition output ``` So the feature is not just adding a spill file fallback to the existing whole-input buffer. It also makes `WindowAggExec` process completed partitions incrementally, with spill used when the active partition cannot grow its memory reservation. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
