SongYadong commented on issue #23794: [SPARK-26884][CORE] Let task acquire memory accurately when using spilled memory URL: https://github.com/apache/spark/pull/23794#issuecomment-465389247 Thanks for reminding me. @jiangxb1987 @liupc Before spill, we have called `acquireExecutionMemory `in which task already got at least `1 / (2 * numActiveTasks)` of total memory pool. But `numActiveTasks `may have changed when calling `acquireExecutionMemory `again after spill. That's my mistake. If task havn't got `1 / (2 * numActiveTasks)` of total memory pool, the change in this PR will in some cases result in more spill instead of waiting for other tasks to free up more memory. This result may lead to direction we don't want. And spill is more expensive generally. I will close this PR later.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
