r7raul1984 opened a new issue, #13044:
URL: https://github.com/apache/gluten/issues/13044
### Backend
VL (Velox)
### Bug description
Bug description:
## Summary
`DynamicOffHeapSizingMemoryTarget.exceedsMaxMemoryUsage()` uses
`Runtime.getRuntime().totalMemory()` (committed heap) instead of actual
used heap
(`totalMemory() - freeMemory()`) to decide whether an allocation would
exceed the
memory budget. This causes spurious OOMs between Spark stages when the JVM
has
committed but not yet released heap pages back to the OS.
## Root Cause
When `spark.gluten.memory.dynamic.offHeap.sizing.enabled=true`, the OOM
gate is:
```java
requestedSize + usedOffHeap + Runtime.totalMemory() >= Runtime.maxMemory()
Runtime.totalMemory() is the JVM committed heap — memory the JVM has
reserved
from the OS, including a large amount that is free. After a
memory-intensive stage
completes, the JVM does not immediately return committed pages, so
totalMemory()
remains high even though actual heap usage has dropped significantly.
Impact
A short-lived application with back-to-back stages can OOM at the start of
Stage N
even though actual heap usage is well within budget:
- Stage 0 runs, JVM commits ~8.9 GB of a 9 GB heap
- Stage 0 finishes, actual heap usage drops to ~2 GB, but totalMemory()
stays ~8.9 GB
- Stage 1 starts, requests 8 MB: 8 MB + 0 + 8.9 GB >= 9 GB → OOM triggered
- The allocation would have succeeded if the check used totalMemory() -
freeMemory()
Fix
Replace totalMemory() with totalMemory() - freeMemory() in all call sites
of
exceedsMaxMemoryUsage(), and rename the parameter from totalOnHeapMemory to
usedOnHeapMemory for clarity:
// Before
exceedsMaxMemoryUsage(totalHeapMemory, usedOffHeapMemory, size, 1.0)
// After
exceedsMaxMemoryUsage(totalHeapMemory - freeHeapMemory, usedOffHeapMemory,
size, 1.0)
This applies to the three call sites in borrow() and the one call site in
shouldTriggerAsyncOnHeapMemoryShrink().
Environment
- Gluten version: 1.7
- Spark version: 3.5
- spark.gluten.memory.dynamic.offHeap.sizing.enabled=true
- spark.executor.memory=9216m, no explicit -Xms
- GC: -XX:+UseParallelGC
### Gluten version
main branch
### Spark version
Spark-3.5.x
### Spark configurations
spark.gluten.memory.dynamic.offHeap.sizing.enabled=true
spark.memory.offHeap.enabled=false
spark.memory.offHeap.size=0
spark.executor.memory=9216m
spark.executor.cores=2
spark.executor.extraJavaOptions=-XX:+UseParallelGC -XX:ParallelGCThreads=4
### System information
_No response_
### Relevant logs
```bash
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
