ding-young commented on PR #15700: URL: https://github.com/apache/datafusion/pull/15700#issuecomment-3157119280
@adriangb There are still some cases that external sorting gives up with memory allocation failure, especially when (1) internal memory accounting is wrong https://github.com/apache/datafusion/issues/14748 (which is also the cause of https://github.com/apache/datafusion/issues/16979) and (2) when memory pressure is so high that multi level merge step can't grow the reservation https://github.com/apache/datafusion/blob/6043be448fe19333b0f1469026a62c580f7bbb31/datafusion/physical-plan/src/sorts/multi_level_merge.rs#L381 We need to estimate the memory required for row-formatted batch correctly for (1) and further limit the number of spills to merge if multi level merge fails (https://github.com/apache/datafusion/issues/16908) for (2). I'm working on these follow up issues recently but it will take time. Anyway, until these fixes are done, I'd recommend trying to run above query with a smaller `partitions` setting. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
