[jira] [Commented] (FLINK-14566) Enable ResourceSpec to get/set whether managed memory is used
[
https://issues.apache.org/jira/browse/FLINK-14566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16986810#comment-16986810
]
Zhu Zhu commented on FLINK-14566:
-
Thanks for the feedback [~azagrebin]
Then let's take that simple way and refactor it later when we are ready to
introduce WeightedResourceSpec.
> Enable ResourceSpec to get/set whether managed memory is used
> -
>
> Key: FLINK-14566
> URL: https://issues.apache.org/jira/browse/FLINK-14566
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / Coordination
>Affects Versions: 1.10.0
>Reporter: Zhu Zhu
>Assignee: Zhu Zhu
>Priority: Critical
> Fix For: 1.10.0
>
>
> To calculate managed memory fraction for an operator with UNKNOWN resources,
> we need to know whether the operator will use managed memory, according to
> FLINK-14062.
> To achieve this, we need an interface to set/get whether an operator uses
> managed memory. It can be part of {{ResourceSpec}} or directly locates in the
> operator description classes ({{Transformation}}/{{StreamNode}}).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-14566) Enable ResourceSpec to get/set whether managed memory is used
[
https://issues.apache.org/jira/browse/FLINK-14566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16986766#comment-16986766
]
Andrey Zagrebin commented on FLINK-14566:
-
[~zhuzh]
I think you are right it raises a lot of questions and would need more time to
understand.
Now I tend to agree with the previous suggestion to expose method for
{{Transformation}}/{{StreamNode}} to set weights for managed memory. We should
just carefully handle the case of already set known spec and then trying to set
the weight for managed memory. I think we should not allow mixing weight and
known spec.
If we decide to add WeightedResourceSpec later we can adjust the weight setting
methods accordingly.
> Enable ResourceSpec to get/set whether managed memory is used
> -
>
> Key: FLINK-14566
> URL: https://issues.apache.org/jira/browse/FLINK-14566
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / Coordination
>Affects Versions: 1.10.0
>Reporter: Zhu Zhu
>Priority: Critical
> Fix For: 1.10.0
>
>
> To calculate managed memory fraction for an operator with UNKNOWN resources,
> we need to know whether the operator will use managed memory, according to
> FLINK-14062.
> To achieve this, we need an interface to set/get whether an operator uses
> managed memory. It can be part of {{ResourceSpec}} or directly locates in the
> operator description classes ({{Transformation}}/{{StreamNode}}).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-14566) Enable ResourceSpec to get/set whether managed memory is used
[
https://issues.apache.org/jira/browse/FLINK-14566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16985556#comment-16985556
]
Zhu Zhu commented on FLINK-14566:
-
Thanks for the feedbacks [~xintongsong] [~azagrebin].
Agreed that a {{WeightedResourceSpec}} can supersede current
{{ResourceSpec.UNKNOWN}} and is more powerful.
It can be used in the cases that a user
(1) has no idea/expectation of available resources
(2) has no idea/expectation of exact resource required resources for operators
(3) wants operators to acquire resources adaptively to available resources, in
a customizable relative amount rather than a fixed absolute value
However, in my mind, there are still a few open questions before we can use the
new resource spec type to replace UNKNOWN resources:
1. What would a ResourceProfile from a weighted resource spec look like? A
resource profile is now used for resource requirements as well as for available
resources representation. Looks to me the weighted concept is not suitable to
represent available/allocated resources. Maybe we need also to reconsider how
to properly represent the resource requirements and real resources, like make
ResourceSpec for requirements only and ResourceProfile for available/allocated
resources only, so that the requirements can come in different forms(absolute
values or weights) and the available/allocated resources would have one only
format.
2. What would the unified interfaces look like, for both the existing
{{ResourceSpec}} and the wanted {{WeightedResourceSpec}}, so that they can be
different implementation options rather than different fields of an operator?
3. How to do comparison, merge and subtract with one weighted resource spec and
one resource profile (related to #1)?
We may need some time to figure them out, but the overall idea looks good to
me.
Given that 1.10 code freeze is coming and we need to achieve the original goal
of this ticket(better batch performance with UNKNOWN resources), I'd like to
make the first step simple enough to not affect current resource mechanism:
1. Introduce a {{WeightedResourceSpec}} which contains weights for all
resources of {{ResourceSpec}} and their getters. Each weight is 1 by default.
2. Adding {{WeightedResourceSpec}} to {{Transformation}}/{{StreamNode}}, by the
side of existing {{ResourceSpec}}.
3. The {{WeightedResourceSpec}} can then be used to improve the fraction
calculations, and for weights setting in FLINK-14063
WDYT? [~azagrebin][~xintongsong]
cc [~trohrmann] [~chesnay]
> Enable ResourceSpec to get/set whether managed memory is used
> -
>
> Key: FLINK-14566
> URL: https://issues.apache.org/jira/browse/FLINK-14566
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / Coordination
>Affects Versions: 1.10.0
>Reporter: Zhu Zhu
>Priority: Minor
> Fix For: 1.10.0
>
>
> To calculate managed memory fraction for an operator with UNKNOWN resources,
> we need to know whether the operator will use managed memory, according to
> FLINK-14062.
> To achieve this, we need an interface to set/get whether an operator uses
> managed memory. It can be part of {{ResourceSpec}} or directly locates in the
> operator description classes ({{Transformation}}/{{StreamNode}}).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-14566) Enable ResourceSpec to get/set whether managed memory is used
[
https://issues.apache.org/jira/browse/FLINK-14566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16984419#comment-16984419
]
Andrey Zagrebin commented on FLINK-14566:
-
It sounds to me like 2 different ways to set resource spec in general:
# absolute requirements: this is known ResourceSpec which we have now
# weighted requirements: another type of ResourceSpec where each dimension has
a relative weight then current UNKNOWN is just everything with weight 1 for all
operators similar to what [~xintongsong] suggests and can be implicitly the
default spec. Not sure how we can implement this by adding it to
{{Transformation}}/{{StreamNode}}. I would prefer just setting another type of
spec.
I would not again allow to mix those types of specs in the job atm to avoid
complexity.
But true, the described approach requires a bit bigger effort than just
adjusting current ResourceSpec with boolean flags, I would still try to go in
this direction.
> Enable ResourceSpec to get/set whether managed memory is used
> -
>
> Key: FLINK-14566
> URL: https://issues.apache.org/jira/browse/FLINK-14566
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / Coordination
>Affects Versions: 1.10.0
>Reporter: Zhu Zhu
>Priority: Minor
> Fix For: 1.10.0
>
>
> To calculate managed memory fraction for an operator with UNKNOWN resources,
> we need to know whether the operator will use managed memory, according to
> FLINK-14062.
> I'd propose to define special values of
> onHeapManagedMemory/offHeapManagedMemory
> in ResourceSpec(introduced in FLINK-14405) to distinguish whether managed
> memory is used.
> - For UNKNOWN resource spec, onHeapManagedMemory==MemorySize.ZERO means on
> heap managed memory is used, while onHeapManagedMemory==null means not. For
> non-UNKNOWN resource spec, managed memory size larger than zero indicates it
> is used.
> - similar for off heap managed memory.
> getters: isUsingManagedMemoryOnHeap()/isUsingManagedMemoryOffHeap()
> setters:
> setUsingManagedMemoryOnHeap(boolean)/setUsingManagedMemoryOffHeap(boolean)
> (setters are only allowed to invoke on UNKNOWN, should be ignored for
> non-UNKNOWN for compatibility)
> {{ResourceSpec#isKnown()}} should be added to replace existing UNKNOWN usages
> for equality checking.
> cc [~ykt836] [~trohrmann] [~azagrebin] [~xintongsong]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-14566) Enable ResourceSpec to get/set whether managed memory is used
[
https://issues.apache.org/jira/browse/FLINK-14566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16983269#comment-16983269
]
Xintong Song commented on FLINK-14566:
--
Thanks [~zhuzh], the proposal looks good for me.
- For weights vs. boolean flag, I'm in favor of weight. IMO, weights can cover
the use cases of boolean flags, by simply setting it to 0 / 1. And enabling
weighted calculation do not add much efforts in runtime. It should be fine that
the operator only provide information about whether it uses managed memory or
not, by setting the weight to 0 / 1 only. If later the operators provide
weights other than 0 / 1, there should be no need for the runtime to rework on
it.
- I'm in favor of not putting the weight / flag in ResourceSpec. Adding it into
ResourceSpec could make the operations on ResourceSpec (merging, subtracting,
determining equality) complicated, and I would try to avoid such completeness
if possible.
> Enable ResourceSpec to get/set whether managed memory is used
> -
>
> Key: FLINK-14566
> URL: https://issues.apache.org/jira/browse/FLINK-14566
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / Coordination
>Affects Versions: 1.10.0
>Reporter: Zhu Zhu
>Priority: Minor
> Fix For: 1.10.0
>
>
> To calculate managed memory fraction for an operator with UNKNOWN resources,
> we need to know whether the operator will use managed memory, according to
> FLINK-14062.
> I'd propose to define special values of
> onHeapManagedMemory/offHeapManagedMemory
> in ResourceSpec(introduced in FLINK-14405) to distinguish whether managed
> memory is used.
> - For UNKNOWN resource spec, onHeapManagedMemory==MemorySize.ZERO means on
> heap managed memory is used, while onHeapManagedMemory==null means not. For
> non-UNKNOWN resource spec, managed memory size larger than zero indicates it
> is used.
> - similar for off heap managed memory.
> getters: isUsingManagedMemoryOnHeap()/isUsingManagedMemoryOffHeap()
> setters:
> setUsingManagedMemoryOnHeap(boolean)/setUsingManagedMemoryOffHeap(boolean)
> (setters are only allowed to invoke on UNKNOWN, should be ignored for
> non-UNKNOWN for compatibility)
> {{ResourceSpec#isKnown()}} should be added to replace existing UNKNOWN usages
> for equality checking.
> cc [~ykt836] [~trohrmann] [~azagrebin] [~xintongsong]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-14566) Enable ResourceSpec to get/set whether managed memory is used
[
https://issues.apache.org/jira/browse/FLINK-14566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16983164#comment-16983164
]
Zhu Zhu commented on FLINK-14566:
-
Thanks [~ykt836] for the feedback.
Agreed that we can have a boolean flag in the first step since we haven't
decided how to properly set the weight ATM.
> Enable ResourceSpec to get/set whether managed memory is used
> -
>
> Key: FLINK-14566
> URL: https://issues.apache.org/jira/browse/FLINK-14566
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / Coordination
>Affects Versions: 1.10.0
>Reporter: Zhu Zhu
>Priority: Minor
> Fix For: 1.10.0
>
>
> To calculate managed memory fraction for an operator with UNKNOWN resources,
> we need to know whether the operator will use managed memory, according to
> FLINK-14062.
> I'd propose to define special values of
> onHeapManagedMemory/offHeapManagedMemory
> in ResourceSpec(introduced in FLINK-14405) to distinguish whether managed
> memory is used.
> - For UNKNOWN resource spec, onHeapManagedMemory==MemorySize.ZERO means on
> heap managed memory is used, while onHeapManagedMemory==null means not. For
> non-UNKNOWN resource spec, managed memory size larger than zero indicates it
> is used.
> - similar for off heap managed memory.
> getters: isUsingManagedMemoryOnHeap()/isUsingManagedMemoryOffHeap()
> setters:
> setUsingManagedMemoryOnHeap(boolean)/setUsingManagedMemoryOffHeap(boolean)
> (setters are only allowed to invoke on UNKNOWN, should be ignored for
> non-UNKNOWN for compatibility)
> {{ResourceSpec#isKnown()}} should be added to replace existing UNKNOWN usages
> for equality checking.
> cc [~ykt836] [~trohrmann] [~azagrebin] [~xintongsong]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-14566) Enable ResourceSpec to get/set whether managed memory is used
[
https://issues.apache.org/jira/browse/FLINK-14566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16983161#comment-16983161
]
Kurt Young commented on FLINK-14566:
I think memory weight is not required for now, we can revisit this later. But
the flag of whether using managed memory is very important right now.
> Enable ResourceSpec to get/set whether managed memory is used
> -
>
> Key: FLINK-14566
> URL: https://issues.apache.org/jira/browse/FLINK-14566
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / Coordination
>Affects Versions: 1.10.0
>Reporter: Zhu Zhu
>Priority: Minor
> Fix For: 1.10.0
>
>
> To calculate managed memory fraction for an operator with UNKNOWN resources,
> we need to know whether the operator will use managed memory, according to
> FLINK-14062.
> I'd propose to define special values of
> onHeapManagedMemory/offHeapManagedMemory
> in ResourceSpec(introduced in FLINK-14405) to distinguish whether managed
> memory is used.
> - For UNKNOWN resource spec, onHeapManagedMemory==MemorySize.ZERO means on
> heap managed memory is used, while onHeapManagedMemory==null means not. For
> non-UNKNOWN resource spec, managed memory size larger than zero indicates it
> is used.
> - similar for off heap managed memory.
> getters: isUsingManagedMemoryOnHeap()/isUsingManagedMemoryOffHeap()
> setters:
> setUsingManagedMemoryOnHeap(boolean)/setUsingManagedMemoryOffHeap(boolean)
> (setters are only allowed to invoke on UNKNOWN, should be ignored for
> non-UNKNOWN for compatibility)
> {{ResourceSpec#isKnown()}} should be added to replace existing UNKNOWN usages
> for equality checking.
> cc [~ykt836] [~trohrmann] [~azagrebin] [~xintongsong]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-14566) Enable ResourceSpec to get/set whether managed memory is used
[
https://issues.apache.org/jira/browse/FLINK-14566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16983126#comment-16983126
]
Zhu Zhu commented on FLINK-14566:
-
I'm relaunching this discussion because in the latest syncs we found it is
necessary for batch jobs to run as good as possible in 1.10 when using UNKNOWN
resources. Then there are two questions to answer:
1. Should a weight get introduced for unknown managed memory, so that those
operators that heavily depend managed memory would be able to acquire more
managed memory than others? Or if a simple boolean flag is better to not mess
up the concepts of UNKNOWN resources with real numbers?
2. Where should we put this weight/flag, in {{ResourceSpec}} or in operator
internally({{StreamNode}})?
The benefit to put it in {{ResourceSpec}} is that we can have an overview of
all the resources requirements. The downside is that we are having different
types of UNKNOWN resources, which can be confusing.
The benefit to put it in {{StreamNode}} is that the {{ResourceSpec}} will not
be affected by the weight/flag. But then we are adding some resources
requirements out of the {{ResourceSpec}}.
I'd prefer the option to add a {{weight}} for managed memory and put it in the
{{StreamNode}}. It can unlock better performance for batch jobs running with
UNKNOWN resources, and will not affect current resource structures and usages.
Having the {{weight}} out of {{ResourceSpec}} is not problematic at the moment,
since it is used in compilation stage only and is not needed in runtime.
[~azagrebin][~sewen][~wuzang][~ykt836][~lzljs3620320]
WDYT? If it is OK, we will take it this way.
> Enable ResourceSpec to get/set whether managed memory is used
> -
>
> Key: FLINK-14566
> URL: https://issues.apache.org/jira/browse/FLINK-14566
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / Coordination
>Affects Versions: 1.10.0
>Reporter: Zhu Zhu
>Priority: Minor
> Fix For: 1.10.0
>
>
> To calculate managed memory fraction for an operator with UNKNOWN resources,
> we need to know whether the operator will use managed memory, according to
> FLINK-14062.
> I'd propose to define special values of
> onHeapManagedMemory/offHeapManagedMemory
> in ResourceSpec(introduced in FLINK-14405) to distinguish whether managed
> memory is used.
> - For UNKNOWN resource spec, onHeapManagedMemory==MemorySize.ZERO means on
> heap managed memory is used, while onHeapManagedMemory==null means not. For
> non-UNKNOWN resource spec, managed memory size larger than zero indicates it
> is used.
> - similar for off heap managed memory.
> getters: isUsingManagedMemoryOnHeap()/isUsingManagedMemoryOffHeap()
> setters:
> setUsingManagedMemoryOnHeap(boolean)/setUsingManagedMemoryOffHeap(boolean)
> (setters are only allowed to invoke on UNKNOWN, should be ignored for
> non-UNKNOWN for compatibility)
> {{ResourceSpec#isKnown()}} should be added to replace existing UNKNOWN usages
> for equality checking.
> cc [~ykt836] [~trohrmann] [~azagrebin] [~xintongsong]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-14566) Enable ResourceSpec to get/set whether managed memory is used
[
https://issues.apache.org/jira/browse/FLINK-14566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16964903#comment-16964903
]
Till Rohrmann commented on FLINK-14566:
---
Let's make this then a low priority improvement which do once the rest of the
feature has been completed.
> Enable ResourceSpec to get/set whether managed memory is used
> -
>
> Key: FLINK-14566
> URL: https://issues.apache.org/jira/browse/FLINK-14566
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / Coordination
>Affects Versions: 1.10.0
>Reporter: Zhu Zhu
>Priority: Minor
> Fix For: 1.10.0
>
>
> To calculate managed memory fraction for an operator with UNKNOWN resources,
> we need to know whether the operator will use managed memory, according to
> FLINK-14062.
> I'd propose to define special values of
> onHeapManagedMemory/offHeapManagedMemory
> in ResourceSpec(introduced in FLINK-14405) to distinguish whether managed
> memory is used.
> - For UNKNOWN resource spec, onHeapManagedMemory==MemorySize.ZERO means on
> heap managed memory is used, while onHeapManagedMemory==null means not. For
> non-UNKNOWN resource spec, managed memory size larger than zero indicates it
> is used.
> - similar for off heap managed memory.
> getters: isUsingManagedMemoryOnHeap()/isUsingManagedMemoryOffHeap()
> setters:
> setUsingManagedMemoryOnHeap(boolean)/setUsingManagedMemoryOffHeap(boolean)
> (setters are only allowed to invoke on UNKNOWN, should be ignored for
> non-UNKNOWN for compatibility)
> {{ResourceSpec#isKnown()}} should be added to replace existing UNKNOWN usages
> for equality checking.
> cc [~ykt836] [~trohrmann] [~azagrebin] [~xintongsong]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-14566) Enable ResourceSpec to get/set whether managed memory is used
[
https://issues.apache.org/jira/browse/FLINK-14566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16963844#comment-16963844
]
Zhu Zhu commented on FLINK-14566:
-
[~ykt836] When saying "little", I mean even "no" in the worst case. The
operator can throw error in this case.
But it might be acceptable, given that the resources is set to be UNKNOWN. Just
like OOM is acceptable if the heap resource is not specified(UNKNOWN) and the
TM heap memory is not properly configured.
To solve it, users need to specify the required resource.
Only counting operators really using managed memory is an optimization to
reduce managed memory waste, but it still cannot guarantee an operator will
always acquire enough managed memory to work. So technically it's not necessary.
> Enable ResourceSpec to get/set whether managed memory is used
> -
>
> Key: FLINK-14566
> URL: https://issues.apache.org/jira/browse/FLINK-14566
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / Coordination
>Affects Versions: 1.10.0
>Reporter: Zhu Zhu
>Priority: Major
> Fix For: 1.10.0
>
>
> To calculate managed memory fraction for an operator with UNKNOWN resources,
> we need to know whether the operator will use managed memory, according to
> FLINK-14062.
> I'd propose to define special values of
> onHeapManagedMemory/offHeapManagedMemory
> in ResourceSpec(introduced in FLINK-14405) to distinguish whether managed
> memory is used.
> - For UNKNOWN resource spec, onHeapManagedMemory==MemorySize.ZERO means on
> heap managed memory is used, while onHeapManagedMemory==null means not. For
> non-UNKNOWN resource spec, managed memory size larger than zero indicates it
> is used.
> - similar for off heap managed memory.
> getters: isUsingManagedMemoryOnHeap()/isUsingManagedMemoryOffHeap()
> setters:
> setUsingManagedMemoryOnHeap(boolean)/setUsingManagedMemoryOffHeap(boolean)
> (setters are only allowed to invoke on UNKNOWN, should be ignored for
> non-UNKNOWN for compatibility)
> {{ResourceSpec#isKnown()}} should be added to replace existing UNKNOWN usages
> for equality checking.
> cc [~ykt836] [~trohrmann] [~azagrebin] [~xintongsong]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-14566) Enable ResourceSpec to get/set whether managed memory is used
[
https://issues.apache.org/jira/browse/FLINK-14566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16963833#comment-16963833
]
Kurt Young commented on FLINK-14566:
First we should define what's "little managed memory". Currently managed memory
are allocate with the unit of MemorySegment which has a default 32KB size. I
don't think any operator will work if user specified less than 32KB managed
memory. Furthermore, some operators will have a minimum runnable memory, for
the simplicity of operators. But the minimum requirement won't be too high, and
normally around couple of MBs.
> Enable ResourceSpec to get/set whether managed memory is used
> -
>
> Key: FLINK-14566
> URL: https://issues.apache.org/jira/browse/FLINK-14566
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / Coordination
>Affects Versions: 1.10.0
>Reporter: Zhu Zhu
>Priority: Major
> Fix For: 1.10.0
>
>
> To calculate managed memory fraction for an operator with UNKNOWN resources,
> we need to know whether the operator will use managed memory, according to
> FLINK-14062.
> I'd propose to define special values of
> onHeapManagedMemory/offHeapManagedMemory
> in ResourceSpec(introduced in FLINK-14405) to distinguish whether managed
> memory is used.
> - For UNKNOWN resource spec, onHeapManagedMemory==MemorySize.ZERO means on
> heap managed memory is used, while onHeapManagedMemory==null means not. For
> non-UNKNOWN resource spec, managed memory size larger than zero indicates it
> is used.
> - similar for off heap managed memory.
> getters: isUsingManagedMemoryOnHeap()/isUsingManagedMemoryOffHeap()
> setters:
> setUsingManagedMemoryOnHeap(boolean)/setUsingManagedMemoryOffHeap(boolean)
> (setters are only allowed to invoke on UNKNOWN, should be ignored for
> non-UNKNOWN for compatibility)
> {{ResourceSpec#isKnown()}} should be added to replace existing UNKNOWN usages
> for equality checking.
> cc [~ykt836] [~trohrmann] [~azagrebin] [~xintongsong]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-14566) Enable ResourceSpec to get/set whether managed memory is used
[
https://issues.apache.org/jira/browse/FLINK-14566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16963824#comment-16963824
]
Zhu Zhu commented on FLINK-14566:
-
I think we can do it if there is assumption that an operator with UNKOWN
resource should always work with little managed memory.
Or if we can, in the first version, accept the limitation that an operator with
UNKOWN resource might break in the case of little managed memory available,
which may happen if there are too many operators in one slot sharing
group(logical pipelined region).
[~ykt836] what do you think?
> Enable ResourceSpec to get/set whether managed memory is used
> -
>
> Key: FLINK-14566
> URL: https://issues.apache.org/jira/browse/FLINK-14566
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / Coordination
>Affects Versions: 1.10.0
>Reporter: Zhu Zhu
>Priority: Major
> Fix For: 1.10.0
>
>
> To calculate managed memory fraction for an operator with UNKNOWN resources,
> we need to know whether the operator will use managed memory, according to
> FLINK-14062.
> I'd propose to define special values of
> onHeapManagedMemory/offHeapManagedMemory
> in ResourceSpec(introduced in FLINK-14405) to distinguish whether managed
> memory is used.
> - For UNKNOWN resource spec, onHeapManagedMemory==MemorySize.ZERO means on
> heap managed memory is used, while onHeapManagedMemory==null means not. For
> non-UNKNOWN resource spec, managed memory size larger than zero indicates it
> is used.
> - similar for off heap managed memory.
> getters: isUsingManagedMemoryOnHeap()/isUsingManagedMemoryOffHeap()
> setters:
> setUsingManagedMemoryOnHeap(boolean)/setUsingManagedMemoryOffHeap(boolean)
> (setters are only allowed to invoke on UNKNOWN, should be ignored for
> non-UNKNOWN for compatibility)
> {{ResourceSpec#isKnown()}} should be added to replace existing UNKNOWN usages
> for equality checking.
> cc [~ykt836] [~trohrmann] [~azagrebin] [~xintongsong]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-14566) Enable ResourceSpec to get/set whether managed memory is used
[
https://issues.apache.org/jira/browse/FLINK-14566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16963806#comment-16963806
]
Till Rohrmann commented on FLINK-14566:
---
Would it be possible to say in the first version that {{UNKNOWN}} always uses
on and off heap? It might not be perfect but it simplifies the logic.
> Enable ResourceSpec to get/set whether managed memory is used
> -
>
> Key: FLINK-14566
> URL: https://issues.apache.org/jira/browse/FLINK-14566
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / Coordination
>Affects Versions: 1.10.0
>Reporter: Zhu Zhu
>Priority: Major
> Fix For: 1.10.0
>
>
> To calculate managed memory fraction for an operator with UNKNOWN resources,
> we need to know whether the operator will use managed memory, according to
> FLINK-14062.
> I'd propose to define special values of
> onHeapManagedMemory/offHeapManagedMemory
> in ResourceSpec(introduced in FLINK-14405) to distinguish whether managed
> memory is used.
> - For UNKNOWN resource spec, onHeapManagedMemory==MemorySize.ZERO means on
> heap managed memory is used, while onHeapManagedMemory==null means not. For
> non-UNKNOWN resource spec, managed memory size larger than zero indicates it
> is used.
> - similar for off heap managed memory.
> getters: isUsingManagedMemoryOnHeap()/isUsingManagedMemoryOffHeap()
> setters:
> setUsingManagedMemoryOnHeap(boolean)/setUsingManagedMemoryOffHeap(boolean)
> (setters are only allowed to invoke on UNKNOWN, should be ignored for
> non-UNKNOWN for compatibility)
> {{ResourceSpec#isKnown()}} should be added to replace existing UNKNOWN usages
> for equality checking.
> cc [~ykt836] [~trohrmann] [~azagrebin] [~xintongsong]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
