dongjoon-hyun commented on PR #60191: URL: https://github.com/apache/spark/pull/60191#issuecomment-6699964987
Thank you, @viirya. Good catch on `SPILL_OUT_OF_MEMORY`. When another consumer's spill fails with an `IOException`, `TaskMemoryManager.acquireExecutionMemory` may have already acquired part of the request, and that part stays with the task until it ends. The task would also keep using the consumer whose spill failed. To keep the 4.1 behavior exactly, I narrowed the catch to `UNABLE_TO_ACQUIRE_MEMORY` and extended the test to check that `SPILL_OUT_OF_MEMORY` still propagates (e0a654bdca2). I'll backport this to branch-4.3 and branch-4.2 too. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
