FMX commented on code in PR #3018:
URL: https://github.com/apache/celeborn/pull/3018#discussion_r1923004789
##########
worker/src/main/java/org/apache/celeborn/service/deploy/worker/memory/MemoryManager.java:
##########
@@ -282,7 +292,7 @@ private MemoryManager(CelebornConf conf, StorageManager
storageManager, Abstract
Utils.bytesToString(readBufferThreshold),
Utils.bytesToString(readBufferTarget),
Utils.bytesToString(memoryFileStorageThreshold),
- resumeRatio);
+ directMemoryResumeRatio);
Review Comment:
You can add pinned memory resume ratio here. It is an important parameter
for memory manager.
##########
worker/src/main/java/org/apache/celeborn/service/deploy/worker/memory/MemoryManager.java:
##########
@@ -93,6 +96,9 @@ public class MemoryManager {
private long memoryFileStorageThreshold;
private final LongAdder memoryFileStorageCounter = new LongAdder();
private final StorageManager storageManager;
+ private boolean pinnedMemoryCheckEnabled;
+ private long pinnedMemoryCheckInterval;
+ private long pinnedMemoryLastCheckTime = 0L;
Review Comment:
The default value for this is 0.
##########
worker/src/main/java/org/apache/celeborn/service/deploy/worker/memory/MemoryManager.java:
##########
@@ -93,6 +96,9 @@ public class MemoryManager {
private long memoryFileStorageThreshold;
private final LongAdder memoryFileStorageCounter = new LongAdder();
private final StorageManager storageManager;
+ private boolean pinnedMemoryCheckEnabled;
+ private long pinnedMemoryCheckInterval;
Review Comment:
To avoid frequently calling a pinned memory counter, I think you can cache
the last pinned memory value and refresh it periodically. And exporting the
pinned memory value to the metrics.
##########
worker/src/main/java/org/apache/celeborn/service/deploy/worker/memory/MemoryManager.java:
##########
@@ -436,6 +445,16 @@ public long getMemoryUsage() {
return getNettyUsedDirectMemory() + sortMemoryCounter.get();
}
+ public long getAllocatedMemory() {
Review Comment:
This method should be renamed to getPinnedMemory. The allocated memory is
the netty memory counter.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
