[
https://issues.apache.org/jira/browse/FLINK-14566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16983126#comment-16983126
]
Zhu Zhu edited comment on FLINK-14566 at 11/27/19 3:13 AM:
-----------------------------------------------------------
I'm relaunching this discussion because in the latest syncs we found it is
necessary for batch jobs to run as good as possible in 1.10 when using UNKNOWN
resources. Then there are two questions to answer:
1. Should a weight get introduced for unknown managed memory, so that those
operators that heavily depend managed memory would be able to acquire more
managed memory than others? Or if a simple boolean flag is better to not mess
up the concepts of UNKNOWN resources with real numbers?
2. Where should we put this weight/flag, in {{ResourceSpec}} or in operator
internally({{StreamNode}})?
The benefit to put it in {{ResourceSpec}} is that we can have an overview of
all the resources requirements. The downside is that we are having different
types of UNKNOWN resources, which can be confusing.
The benefit to put it in {{StreamNode}} is that the {{ResourceSpec}} will not
be affected by the weight/flag. But then we are adding some resources
requirements out of the {{ResourceSpec}}.
I'd prefer the option to add a {{weight}} for managed memory and put it in the
{{StreamNode}}. It can unlock better performance for batch jobs running with
UNKNOWN resources, and will not affect current resource structures and usages.
Having the {{weight}} out of {{ResourceSpec}} is not problematic at the moment,
since it is used in compilation stage only and is not needed in runtime.
[~azagrebin] [~sewen] [~xintongsong] [~ykt836] [~lzljs3620320]
WDYT? If it is OK, we will take it this way.
was (Author: zhuzh):
I'm relaunching this discussion because in the latest syncs we found it is
necessary for batch jobs to run as good as possible in 1.10 when using UNKNOWN
resources. Then there are two questions to answer:
1. Should a weight get introduced for unknown managed memory, so that those
operators that heavily depend managed memory would be able to acquire more
managed memory than others? Or if a simple boolean flag is better to not mess
up the concepts of UNKNOWN resources with real numbers?
2. Where should we put this weight/flag, in {{ResourceSpec}} or in operator
internally({{StreamNode}})?
The benefit to put it in {{ResourceSpec}} is that we can have an overview of
all the resources requirements. The downside is that we are having different
types of UNKNOWN resources, which can be confusing.
The benefit to put it in {{StreamNode}} is that the {{ResourceSpec}} will not
be affected by the weight/flag. But then we are adding some resources
requirements out of the {{ResourceSpec}}.
I'd prefer the option to add a {{weight}} for managed memory and put it in the
{{StreamNode}}. It can unlock better performance for batch jobs running with
UNKNOWN resources, and will not affect current resource structures and usages.
Having the {{weight}} out of {{ResourceSpec}} is not problematic at the moment,
since it is used in compilation stage only and is not needed in runtime.
[~azagrebin][~sewen][~xintongsong][~ykt836][~lzljs3620320]
WDYT? If it is OK, we will take it this way.
> Enable ResourceSpec to get/set whether managed memory is used
> -------------------------------------------------------------
>
> Key: FLINK-14566
> URL: https://issues.apache.org/jira/browse/FLINK-14566
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / Coordination
> Affects Versions: 1.10.0
> Reporter: Zhu Zhu
> Priority: Minor
> Fix For: 1.10.0
>
>
> To calculate managed memory fraction for an operator with UNKNOWN resources,
> we need to know whether the operator will use managed memory, according to
> FLINK-14062.
> I'd propose to define special values of
> onHeapManagedMemory/offHeapManagedMemory
> in ResourceSpec(introduced in FLINK-14405) to distinguish whether managed
> memory is used.
> - For UNKNOWN resource spec, onHeapManagedMemory==MemorySize.ZERO means on
> heap managed memory is used, while onHeapManagedMemory==null means not. For
> non-UNKNOWN resource spec, managed memory size larger than zero indicates it
> is used.
> - similar for off heap managed memory.
> getters: isUsingManagedMemoryOnHeap()/isUsingManagedMemoryOffHeap()
> setters:
> setUsingManagedMemoryOnHeap(boolean)/setUsingManagedMemoryOffHeap(boolean)
> (setters are only allowed to invoke on UNKNOWN, should be ignored for
> non-UNKNOWN for compatibility)
> {{ResourceSpec#isKnown()}} should be added to replace existing UNKNOWN usages
> for equality checking.
> cc [~ykt836] [~trohrmann] [~azagrebin] [~xintongsong]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
