timarmstrong opened a new pull request #34186: URL: https://github.com/apache/spark/pull/34186
### What changes were proposed in this pull request? * Factor out a method `trySpillAndAcquire()` from `acquireExecutionMemory()` that handles the details of how to spill a `MemoryConsumer` and acquire the spilled memory. This logic was duplicated twice. * Combine the two loops (spill other consumers and self-spill) into a single loop that implements equivalent logic. I made self-spill the lowest priority consumer and this is exactly equivalent. * Consolidate comments a little to explain what the policy is trying to achieve and how at a high level ### Why are the changes needed? Reduce code duplication and better separate the policy decision of which MemoryConsumer to spill from the mechanism of requesting it to spill. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Added some unit tests to verify the details of the spilling decisions in some scenarios that are not covered by current unit tests. Ran these on Spark master without the TaskMemoryManager changes to confirm that the behaviour is the same before and after my refactoring. The SPARK-35486 test also provides some coverage for the retry loop. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
