[ 
https://issues.apache.org/jira/browse/FLINK-17061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17091460#comment-17091460
 ] 

Xintong Song commented on FLINK-17061:
--------------------------------------

As mentioned in the description, this is not a problem for now. It is only 
needed when later we have a slot manager implementation that dynamically decide 
TM resources according to the job resource requirements.

There could be problem not only the requested worker resource is larger than 
the flink/process memory, but also when it is much smaller than that.
With FLIP-49, we have the following equation.
{code}
processMemorySize = workerResourceSpec.totalMemorySize + frameworkMemorySize + 
jvmMetaspace + jvmOverhead
{code}

{{frameworkMemorySize}} and {{jvmMetaspace}} always have absolute values 
(explicitly configured or default), and {{jvmOverhead}} always have absolute 
min-max range.

If {{processMemorySize}} is also configured, we might have conflict when:
{code}
processMemorySize > workerResourceSpec.totalMemorySize + frameworkMemorySize + 
jvmMetaspace + jvmOverhead.max
{code}

> Unset process/flink memory size from configuration once dynamic worker 
> resource is activated.
> ---------------------------------------------------------------------------------------------
>
>                 Key: FLINK-17061
>                 URL: https://issues.apache.org/jira/browse/FLINK-17061
>             Project: Flink
>          Issue Type: Task
>          Components: Runtime / Configuration, Runtime / Coordination
>    Affects Versions: 1.11.0
>            Reporter: Xintong Song
>            Priority: Major
>
> With FLINK-14106, memory of a TaskExecutor is decided in two steps on active 
> resource managers.
> - {{SlotManager}} decides {{WorkerResourceSpec}}, including memory used by 
> Flink tasks: task heap, task off-heap, network and managed memory.
> - {{ResourceManager}} derives {{TaskExecutorProcessSpec}} from 
> {{WorkerResourceSpec}} and the configuration, deciding sizes of memory used 
> by Flink framework and JVM: framework heap, framework off-heap, jvm metaspace 
> and jvm overhead.
> This works fine for now, because both {{WorkerResourceSpec}} and 
> {{TaskExecutorProcessSpec}} are derived from the same configurations. 
> However, it might cause problem if later we have new {{SlotManager}} 
> implementations that decides {{WorkerResourceSpec}} dynamically. In such 
> cases, the process/flink sizes in configuration should be ignored, or it may 
> easily lead to configuration conflicts.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Reply via email to