jayzhan211 commented on issue #25804: URL: https://github.com/apache/datafusion/issues/25804#issuecomment-6704063478
@alamb to answer part of your question: finding 9 is ticked in the plan, but #26264 is still open, and it leaves one gap that I'd track here rather than grow that PR. I don't think this gap should hold 56; does that work for you? On the #26264 reproducer (62.5 MB fair pool, 10 partitions, 3 MB `sort_spill_reservation_bytes`), 33-35 of 40 runs succeed with that PR, against 0 of 40 with its headroom switched off in the same build. At 1 MB and 1.6 MB it reaches 39-40 of 40. The remaining failures come from a partition that can't get its own memory: ``` Failed to allocate additional 3.0 MB for ExternalSorterMerge[6] with 0.0 B already allocated for this reservation - 1997.3 KB remain available for the total memory pool ``` Each merge leaves as much free as it takes, but merges that start one after another still add up, so a guarantee needs each merge to know how many partitions share the pool, for example by capping it at a per-partition share of the limit. That is a larger policy change and nobody has prototyped it. It would affect sorts with many partitions whose workspaces take a large part of the pool; here 10 × 3 MB is about half. @andygrove could you add this to the plan as an open item, once #26264 merges? Anyone is welcome to pick it up. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
