xintongsong commented on a change in pull request #9693: [FLINK-13984] Separate
on-heap and off-heap managed memory pools
URL: https://github.com/apache/flink/pull/9693#discussion_r328001404
##########
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/memory/MemoryManager.java
##########
@@ -257,133 +216,96 @@ public boolean verifyEmpty() {
* of memory pages any more.
*/
public List<MemorySegment> allocatePages(Object owner, int numPages)
throws MemoryAllocationException {
- final ArrayList<MemorySegment> segs = new
ArrayList<MemorySegment>(numPages);
- allocatePages(owner, segs, numPages);
- return segs;
+ List<MemorySegment> segments = new ArrayList<>(numPages);
+ allocatePages(owner, segments, numPages);
+ return segments;
}
/**
- * Allocates a set of memory segments from this memory manager. If the
memory manager pre-allocated the
- * segments, they will be taken from the pool of memory segments.
Otherwise, they will be allocated
- * as part of this call.
+ * Allocates a set of memory segments from this memory manager.
+ *
+ * <p>The returned segments can have any memory type. The total
allocated memory for each type will not exceed its
+ * size limit, announced in the constructor.
*
* @param owner The owner to associate with the memory segment, for the
fallback release.
* @param target The list into which to put the allocated memory pages.
* @param numPages The number of pages to allocate.
* @throws MemoryAllocationException Thrown, if this memory manager
does not have the requested amount
* of memory pages any more.
*/
- public void allocatePages(Object owner, List<MemorySegment> target, int
numPages)
- throws MemoryAllocationException {
+ public void allocatePages(
Review comment:
In addition to this, I think we also need 'allocatePages' that support
specifying memory types.
The current 'allocatePages', which do not specify which type (on-heap /
off-heap) of memory should be allocated, may work well with DataSet jobs
without fine grained resource requirement. But for blink sql jobs (bounded
streaming) with fine grained resource profiles (flip-53), operators have
specific and separate memory quota for on-heap / off-heap managed memory, and
will be scheduled accordingly. Thus even they can use pages of either type,
they need to allocate pages respecting their quota for each pool. Otherwise,
other operators that can only use one of the type (e.g., flip-58 python udx
that uses off-heap managed memory) might be affected.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
