shrirangmhalgi commented on PR #56293: URL: https://github.com/apache/spark/pull/56293#issuecomment-4619258196
Question (non-blocking): If a consumer's `spill()` callback internally re-acquires execution memory (e.g., via `acquireExecutionMemory` for a non-page allocation, which bypasses `inPageAllocationRecovery`), then `consumerToSpill.getUsed()` after the spill could be ≥ `usedBeforeSpill`. The `Math.max(0L, ...)` clamp treats this as "no progress," and the consumer gets removed from the iteration list. In that scenario the consumer did free physical memory (which is what we need for the allocator retry), but also re-acquired tracked memory in the same callback. The method would undercount released bytes and potentially give up before the allocator has enough headroom. Is this intentional best-effort behavior? If so, a brief comment here noting that progress tracking is conservative (may undercount when spill callbacks re-acquire) would help future readers understand why the bounded fallback paths exist even when spillable consumers remain. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
