[
https://issues.apache.org/jira/browse/FLINK-18188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17129158#comment-17129158
]
Andrey Zagrebin commented on FLINK-18188:
-----------------------------------------
Here is the offline discussion conclusion between [~chesnay] [~trohrmann]
[~xintongsong] [~azagrebin].
Problem:
jvm.heap = 128m (set)
*default* off-heap = 128m (not set)
total.flink.mem = 512m (set)
(Normally: total.flink.mem = heap + off-heap)
currently we fail if `jvm.heap + off-heap <> total.flink.mem`
The agreed solution is to implicitly extend off-heap (if not set) to
(total.flink.mem - jvm.heap). This should be logged as info message and
documented for JM off-heap.
Alternatively, if there is another sub-component of total.flink.mem later, we
can introduce additional fractions or decide to how break the tie based on
importance. If this new component cannot be zero by default, it would require
migration anyways.
> Document asymmetric allocation of Flink memory
> ----------------------------------------------
>
> Key: FLINK-18188
> URL: https://issues.apache.org/jira/browse/FLINK-18188
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Coordination
> Reporter: Chesnay Schepler
> Priority: Major
> Fix For: 1.11.0
>
>
> I stumbled on some non-documented and surprising behavior while configuring
> the heap/off-heap memory sizes.
> Let's start with a base line, where we configure the total process memory to
> 1024mb:
> {code}
> Final Master Memory configuration:
> Total Process Memory: 1024.000mb (1073741824 bytes)
> Total Flink Memory: 576.000mb (603979776 bytes)
> Flink Heap Memory: 448.000mb (469762048 bytes)
> Flink Off-heap Memory: 128.000mb (134217728 bytes)
> JVM Metaspace Memory: 256.000mb (268435456 bytes)
> JVM Overhead Memory: 192.000mb (201326592 bytes)
> {code}
> If we reduce the off-heap memory to 16mb, then the gained memory is allocated
> to the heap memory:
> {code}
> Final Master Memory configuration:
> Total Process Memory: 1024.000mb (1073741824 bytes)
> Total Flink Memory: 576.000mb (603979776 bytes)
> Flink Heap Memory: 560.000mb (587202560 bytes)
> Flink Off-heap Memory: 16.000mb (16777216 bytes)
> JVM Metaspace Memory: 256.000mb (268435456 bytes)
> JVM Overhead Memory: 192.000mb (201326592 bytes)
> {code}
> Conversely however, if we reduce the heap memory to 128mb, then the gained
> memory is allocated to the JVM overhead.
> {code}
> Final Master Memory configuration:
> Total Process Memory: 1024.000mb (1073741824 bytes)
> Total Flink Memory: 256.000mb (268435456 bytes)
> Flink Heap Memory: 128.000mb (134217728 bytes)
> Flink Off-heap Memory: 128.000mb (134217728 bytes)
> JVM Metaspace Memory: 256.000mb (268435456 bytes)
> JVM Overhead Memory: 512.000mb (536870912 bytes)
> {code}
> I'm not sure whether this behavior is correct.
> The documentation for {{Capped Fractionated Components}} does that "_It can
> also happen that the fraction is ignored if the sizes of the total memory and
> its other components are defined. In this case, the JVM Overhead is the rest
> of the total memory._"
> This however only explains one of the above cases (the last one).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
