[
https://issues.apache.org/jira/browse/FLINK-2641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14738986#comment-14738986
]
Maximilian Michels commented on FLINK-2641:
-------------------------------------------
The new configuration paramters are
- taskmanager.memory.off-heap: Boolean
- taskmanager.memory.off-heap-ratio: Float
These have to interplay with these existing config entries:
- taskmanager.memory.size: Int
- taskmanager.memory.fraction: Float
- taskmanager.heap.mb: Int
This works as follows:
* If {{taskmanager.memory.size}} is set, the task manager allocates the number
of memory either on the JVM heap or as direct memory depending on whether
{{taskmanager.memory.off-heap}} is set to true.
* If {{taskmanager.memory.size}} is not set, the memory allocation is slightly
more involved:
** In the standard heap mode ({{taskmanager.memory.off-heap}} set to false),
the task manager gets the free heap memory during startup, multiplies it by
{{taskmanager.memory.fraction}}, and allocates the resulting memory size as
heap memory.
** In off-heap mode ({{taskmanager.memory.off-heap}} set to true), the task
manager gets the maximum available allocatable memory (set by the -Xmx
parameter), multiplies it by the {{taskmanager.memory.off-heap-ratio}}, and
allocates the resulting memory size as direct (off-heap) memory size.
In order to be able to switch from heap to off-heap memory usage and not having
to adjust any memory settings, the {{taskmanager.heap.mb}} setting has to be
dynamically adjusted:
* If {{taskmanager.memory.off-heap}} is set and
** {{taskmanager.memory.size}} is set, then the maximum JVM memory size needs
to be adjusted to {{taskmanager.heap.mb - taskmanager.memory.size}}.
** {{taskmanager.memory.size}} is not set, the maximum JVM memory size needs to
be adjusted to {{taskmanager.heap.mb * (1 -
(1/taskmanager.memory.off-heap-ratio))}}. Note that
{{taskmanager.memory.off-heap-ratio > 1.0}}.
-----
Up for discussion:
It might confuse users that {{taskmanager.heap.mb}} specifies the JVM heap size
but we use it to calculate the total memory (on and off-heap combined).
Therefore, I'd propose to add another config entry which let's you specify the
JVM process memory: {{taskmanager.jvm.mb}} would set the maximum memory which
is then split up between JVM heap and off-heap memory. The old parameter can
still be used to explicitly set the heap memory. Since we introduce new config
parameters anyways with the off-heap memory, I think this is a fair compromise.
{{taskmanager.heap.mb}} will be deprecated in favor of {{taskmanager.jvm.mb}}.
Then we should also rename {{jobmanager.heap.mb}} to {{jobmanager.jvm.mb}}.
Since I'm not entirely sure about the name and whether we should introduce
additional config parameters, I will leave that up for discussion now and
implement the config parameter handling with the default
{{taskmanager.heap.mb}} parameter and its dynamic adjustment.
> Integrate the off-heap memory configuration with the TaskManager start script
> -----------------------------------------------------------------------------
>
> Key: FLINK-2641
> URL: https://issues.apache.org/jira/browse/FLINK-2641
> Project: Flink
> Issue Type: New Feature
> Components: Start-Stop Scripts
> Affects Versions: 0.10
> Reporter: Stephan Ewen
> Assignee: Maximilian Michels
> Fix For: 0.10
>
>
> The TaskManager start script needs to adjust the {{-Xmx}}, {{-Xms}}, and
> {{-XX:MaxDirectMemorySize}} parameters according to the off-heap memory
> settings.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
