r7raul1984 opened a new pull request, #13045:
URL: https://github.com/apache/gluten/pull/13045
Fixes #13044
DynamicOffHeapSizingMemoryTarget.exceedsMaxMemoryUsage() previously used
Runtime.getRuntime().totalMemory() (committed heap) as the on-heap memory
pressure indicator. Committed heap is the memory the JVM has reserved from the
OS, which includes a potentially large
amount of free pages that have not yet been returned. As a result, the
check overstates memory pressure between Spark stages and triggers spurious
OOMs.
Example of the failure:
- Stage 0 runs with 2 concurrent tasks (~2 GB peak each); JVM commits ~8.9
GB of a 9 GB heap
- Stage 0 finishes; actual heap usage drops to ~2 GB, but totalMemory()
remains ~8.9 GB because the JVM has not yet returned the committed pages to the
OS
- Stage 1 starts and requests 8 MB: 8 MB + 0 + 8.9 GB >= 9 GB → OOM
triggered
- Actual free heap was ~6.5 GB (freeMemory()); the allocation should have
succeeded
Fix:
Replace totalMemory() with totalMemory() - freeMemory() (actual used heap)
at all four call sites of exceedsMaxMemoryUsage() — three in borrow() and one
in shouldTriggerAsyncOnHeapMemoryShrink(). The parameter is renamed from
totalOnHeapMemory to usedOnHeapMemory for
clarity.
// Before
exceedsMaxMemoryUsage(totalHeapMemory, usedOffHeapMemory, size, 1.0)
// After
exceedsMaxMemoryUsage(totalHeapMemory - freeHeapMemory, usedOffHeapMemory,
size, 1.0)
Note: freeHeapMemory is already read from
Runtime.getRuntime().freeMemory() at the top of borrow(), so no additional JVM
calls are introduced.
How was this patch tested?
Existing unit tests in DynamicOffHeapSizingMemoryTargetTest (added in
#12425) continue to pass. New tests covering the spurious-OOM scenario between
stages will be added as a follow-up.
Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Sonnet 4.6, Anthropic
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
