comphead commented on issue #2452: URL: https://github.com/apache/datafusion-comet/issues/2452#issuecomment-3362011811
Yes, you are right @andygrove the way spilling done in sorter and SMJ is: - try reserve memory - if failed and spilling supported then spill However we need to keep in mind that it makes sense for the specific memory pool. So the large memory is already in the memory and spilling works later and for pool only and if the pool == available memory the spilling doesn't make sense IMO. One idea is to configure the pool individually so spilling would start sooner -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
