alamb opened a new issue, #26711: URL: https://github.com/apache/datafusion/issues/26711
## Describe the problem Collects the existing issues so the special cases can be revisited together rather than one at a time. See the discussion in https://github.com/apache/datafusion/pull/25853#discussion_r4178384382. Memory-limited execution has grown by accretion. Each spilling operator ([sort][sort-spill], [aggregate][aggregate-spill], [sort-merge join][smj-spill], [nested loop join][nlj-spill], [repartition][repartition-spill]) has fixed its own out-of-memory failures with a local special case, and the shared pieces such as [`StreamingMergeBuilder`][streaming-merge-builder], [`MultiLevelMergeBuilder`][multi-level-merge-builder] and [`SpillManager`][spill-manager] have picked up knobs that only one caller sets. For example, the spill merge now has three options that only aggregate spilling uses ([`with_replay_headroom`][with-replay-headroom], [`with_intermediate_merge_sizing`][with-intermediate-merge-sizing] and [`with_max_batch_bytes`][with-max-batch-bytes], the last added in #25853) The result is hard to reason about locally ## Issues ### Shared spill and merge infrastructure - [ ] Revisit the aggregate-only options on the spill merge (`with_replay_headroom`, `with_intermediate_merge_sizing`, `with_max_batch_bytes`) -- *to be filed*, follow-up to https://github.com/apache/datafusion/pull/25853#discussion_r4178384382 - [ ] #23760 - [ ] #17595 - [ ] #18261 - [ ] #13430 - [ ] #16908 ### Aggregate - [ ] #25537 - [ ] #25902 - [ ] #25851 - [ ] #26701 - [ ] #26628 ### Sort - [ ] #25804 - [ ] #25183 - [ ] #21543 - [ ] #26411 ### Testing - [ ] #13431 - [ ] #23658 - [ ] #26700 ## Related discussions and epics - #16841 [DISCUSSION] Memory accounting model discussion - #26225 [EPIC] Refactor Operators for Maintainability - #26309 [EPIC] Reduce complexity and duplication hotspots in DataFusion - #25637 [EPIC] Spilling coverage gaps vs Spark and DuckDB: window, cross join, and pool-driven reclaim - #24768 [EPIC] Spilling Hash Join - #13123 [EPIC] Improved Externalized / Spilling / Large than Memory Hash Aggregation - #20715 Fix Memory Backpressure/Spilling Coordination [sort-spill]: https://github.com/apache/datafusion/blob/c8c046491ed44f0f969695f4aeade4e70af3317e/datafusion/physical-plan/src/sorts/sort.rs#L302 [aggregate-spill]: https://github.com/apache/datafusion/blob/c8c046491ed44f0f969695f4aeade4e70af3317e/datafusion/physical-plan/src/aggregates/hash_stream.rs#L269 [smj-spill]: https://github.com/apache/datafusion/blob/c8c046491ed44f0f969695f4aeade4e70af3317e/datafusion/physical-plan/src/joins/sort_merge_join/exec.rs#L964 [nlj-spill]: https://github.com/apache/datafusion/blob/c8c046491ed44f0f969695f4aeade4e70af3317e/datafusion/physical-plan/src/joins/nested_loop_join.rs#L1786 [repartition-spill]: https://github.com/apache/datafusion/blob/c8c046491ed44f0f969695f4aeade4e70af3317e/datafusion/physical-plan/src/repartition/mod.rs#L509 [streaming-merge-builder]: https://github.com/apache/datafusion/blob/c8c046491ed44f0f969695f4aeade4e70af3317e/datafusion/physical-plan/src/sorts/streaming_merge.rs#L88 [multi-level-merge-builder]: https://github.com/apache/datafusion/blob/c8c046491ed44f0f969695f4aeade4e70af3317e/datafusion/physical-plan/src/sorts/multi_level_merge.rs#L144 [spill-manager]: https://github.com/apache/datafusion/blob/c8c046491ed44f0f969695f4aeade4e70af3317e/datafusion/physical-plan/src/spill/spill_manager.rs#L40 [with-replay-headroom]: https://github.com/apache/datafusion/blob/c8c046491ed44f0f969695f4aeade4e70af3317e/datafusion/physical-plan/src/sorts/streaming_merge.rs#L169 [with-intermediate-merge-sizing]: https://github.com/apache/datafusion/blob/c8c046491ed44f0f969695f4aeade4e70af3317e/datafusion/physical-plan/src/sorts/streaming_merge.rs#L179 [with-max-batch-bytes]: https://github.com/apache/datafusion/blob/3289c00a6bfdf14b9ad444d7ff47b2b9364b1be2/datafusion/physical-plan/src/sorts/streaming_merge.rs#L154 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
