liuzqt commented on code in PR #47192:
URL: https://github.com/apache/spark/pull/47192#discussion_r1678325966
##########
core/src/main/java/org/apache/spark/memory/TaskMemoryManager.java:
##########
@@ -202,6 +212,14 @@ public long acquireExecutionMemory(long required,
MemoryConsumer requestingConsu
logger.debug("Task {} acquired {} for {}", taskAttemptId,
Utils.bytesToString(got),
requestingConsumer);
}
+
+ if (mode == MemoryMode.OFF_HEAP) {
+ peakOffHeapMemory = Math.max(peakOffHeapMemory,
+ memoryManager.getOffHeapExecutionMemoryUsageForTask(taskAttemptId));
Review Comment:
Theoretically we can maintain both currentMem and peakMem within
`TaskMemoryManager` so that we don’t need to ask `memoryManager`, but on the
other hand `memoryManager` is designed to maintain per-task mem usage so by
doing this we kinda maintain this in two places. @JoshRosen WDYT
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
