wForget commented on issue #10932:
URL:
https://github.com/apache/incubator-gluten/issues/10932#issuecomment-3440418390
From this log, the offheap memory usage has exceeded the max memory of
TaskMemoryManager `(3 * 1024 MB -300 MB) * 0.6 = 1663.2 MB`, so it should be
enough to trigger spill.
```
25/10/24 09:35:25 INFO DynamicOffHeapSizingMemoryTarget: Updated VM flags:
MaxHeapFreeRatio from 30 to 5.
25/10/24 09:35:25 WARN DynamicOffHeapSizingMemoryTarget: Starting full gc to
shrink JVM memory: Total On-heap: 519110656, Free On-heap: 337648400, Total
Off-heap: 2524971008, Used On-Heap: 181462256, Executor memory: 3113877504.
25/10/24 09:35:26 WARN DynamicOffHeapSizingMemoryTarget: Finished full gc to
shrink JVM memory: Total On-heap: 519110656, Free On-heap: 414046152, Total
Off-heap: 2524971008, Used On-Heap: 105064504, Executor memory: 3113877504, [GC
Retry times: 3].
25/10/24 09:35:26 INFO DynamicOffHeapSizingMemoryTarget: Reverted VM flags
back.
25/10/24 09:35:26 WARN DynamicOffHeapSizingMemoryTarget: Failing allocation
as unified memory is OOM. Used Off-heap: 2524971008, Used On-Heap: 107549216,
Free On-heap: 411561440, Total On-heap: 519110656, Max On-heap: 3113877504,
Allocation: 486539264.
25/10/24 09:35:26 INFO TaskMemoryManager: Memory used in task 9695
25/10/24 09:35:26 INFO TaskMemoryManager: 0 bytes of memory were used by
task 9695 but are not associated with specific consumers
25/10/24 09:35:26 INFO TaskMemoryManager: 0 bytes of memory are used for
execution and 15127767 bytes of memory are used for storage
25/10/24 09:35:26 WARN ManagedReservationListener: Error reserving memory
from target
org.apache.gluten.memory.memtarget.ThrowOnOomMemoryTarget$OutOfMemoryException:
Not enough spark off-heap execution memory. Acquired: 464.0 MiB, granted: 0.0
B. Try tweaking config option spark.memory.offHeap.size to get larger space to
run this application (if spark.gluten.memory.dynamic.offHeap.sizing.enabled is
not enabled).
Current config settings:
spark.gluten.memory.offHeap.size.in.bytes=1663.2 MiB
spark.gluten.memory.task.offHeap.size.in.bytes=1663.2 MiB
spark.gluten.memory.conservative.task.offHeap.size.in.bytes=831.6 MiB
spark.memory.offHeap.enabled=true
spark.gluten.memory.dynamic.offHeap.sizing.enabled=true
Dynamic off-heap sizing memory target stats:
DynamicOffHeapSizing.96: Current used bytes: 2.3 GiB, peak bytes: 2.3
GiB
```
spark conf:
```
spark.gluten.memory.dynamic.offHeap.sizing.enabled=true
spark.executor.memory=3g
spark.executor.extraJavaOptions=-XX:MinHeapFreeRatio=10
-XX:MaxHeapFreeRatio=30 -Xms512m -XX:+UseSerialGC -Dfile.encoding=utf-8
-verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc:<LOG_DIR>/gc.log
-XX:ErrorFile=<LOG_DIR>/hs_err_pid%p.log
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
