azagrebin commented on a change in pull request #10034: [FLINK-14400] Shrink
scope of MemoryManager from TaskExecutor to slot
URL: https://github.com/apache/flink/pull/10034#discussion_r347275162
##########
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/taskexecutor/TaskManagerServices.java
##########
@@ -490,14 +497,17 @@ private static void checkTempDirs(String[] tmpDirs)
throws IOException {
}
public static ResourceProfile computeSlotResourceProfile(int
numOfSlots, long managedMemorySize) {
- int managedMemoryPerSlotMB = (int)
bytesToMegabytes(managedMemorySize / numOfSlots);
+ // TODO: before operators separate on-heap/off-heap managed
memory, we use on-heap managed memory to denote total managed memory
+ return computeSlotResourceProfile(numOfSlots,
Collections.singletonMap(MemoryType.HEAP, managedMemorySize));
+ }
+
+ private static ResourceProfile computeSlotResourceProfile(int
numOfSlots, Map<MemoryType, Long> memorySizeByType) {
return new ResourceProfile(
Double.MAX_VALUE,
MemorySize.MAX_VALUE,
MemorySize.MAX_VALUE,
- // TODO: before operators separate on-heap/off-heap
managed memory, we use on-heap managed memory to denote total managed memory
- MemorySize.parse(managedMemoryPerSlotMB + "m"),
- MemorySize.MAX_VALUE,
+ new
MemorySize(memorySizeByType.getOrDefault(MemoryType.HEAP, 0L) / numOfSlots),
+ new
MemorySize(memorySizeByType.getOrDefault(MemoryType.OFF_HEAP, 0L) / numOfSlots),
Review comment:
At the moment, our configuration contains only one size of managed memory
and it can be only of one configured type but internal spec/profile data
structures are already prepared to have two sizes for both types of memory.
When the main PR #10161 of FLIP-49 is merged, we will switch everything into
having two types of memory.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
