alamb commented on issue #11390: URL: https://github.com/apache/datafusion/issues/11390#issuecomment-2275595828
Hi @Ablu -- it could be. In general I think our support of ensuring a sort can spill might need some tweaking / help -- once the system is under memory pressure (aka the pool is almost exhausted) several operations to actually spill require (more) memory to actually do so (such as merging the in memory buffers) So in other words, the actual process of trying to spill can itself trigger the "resources exhausted" error One approach I can think of is to turn off resource checking while spilling -- this would be simple to implement, but would result in a memory overshoot. An alternate is to pre-reserve memory for the spilling before memory pressure is too high, but this would likely result in memory that is almost never used (only held in reserve in case it was needed) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
