sunchao commented on PR #56293: URL: https://github.com/apache/spark/pull/56293#issuecomment-4625254187
@shrirangmhalgi Thanks for calling this out. Yes, progress here is intentionally measured conservatively as the net reduction in the consumer's tracked memory. `MemoryConsumer.spill()` callbacks must not reacquire execution memory; calling `TaskMemoryManager.acquireExecutionMemory()` directly from a spill callback would violate the same contract. If a custom consumer does violate it, this path treats the consumer as making no progress rather than over-crediting released memory. I added a short clarification next to the calculation in `7dc2c94627d`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
