[
https://issues.apache.org/jira/browse/FLINK-14566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16985556#comment-16985556
]
Zhu Zhu commented on FLINK-14566:
---------------------------------
Thanks for the feedbacks [~xintongsong] [~azagrebin].
Agreed that a {{WeightedResourceSpec}} can supersede current
{{ResourceSpec.UNKNOWN}} and is more powerful.
It can be used in the cases that a user
(1) has no idea/expectation of available resources
(2) has no idea/expectation of exact resource required resources for operators
(3) wants operators to acquire resources adaptively to available resources, in
a customizable relative amount rather than a fixed absolute value
However, in my mind, there are still a few open questions before we can use the
new resource spec type to replace UNKNOWN resources:
1. What would a ResourceProfile from a weighted resource spec look like? A
resource profile is now used for resource requirements as well as for available
resources representation. Looks to me the weighted concept is not suitable to
represent available/allocated resources. Maybe we need also to reconsider how
to properly represent the resource requirements and real resources, like make
ResourceSpec for requirements only and ResourceProfile for available/allocated
resources only, so that the requirements can come in different forms(absolute
values or weights) and the available/allocated resources would have one only
format.
2. What would the unified interfaces look like, for both the existing
{{ResourceSpec}} and the wanted {{WeightedResourceSpec}}, so that they can be
different implementation options rather than different fields of an operator?
3. How to do comparison, merge and subtract with one weighted resource spec and
one resource profile (related to #1)?
We may need some time to figure them out, but the overall idea looks good to
me.
Given that 1.10 code freeze is coming and we need to achieve the original goal
of this ticket(better batch performance with UNKNOWN resources), I'd like to
make the first step simple enough to not affect current resource mechanism:
1. Introduce a {{WeightedResourceSpec}} which contains weights for all
resources of {{ResourceSpec}} and their getters. Each weight is 1 by default.
2. Adding {{WeightedResourceSpec}} to {{Transformation}}/{{StreamNode}}, by the
side of existing {{ResourceSpec}}.
3. The {{WeightedResourceSpec}} can then be used to improve the fraction
calculations, and for weights setting in FLINK-14063
WDYT? [~azagrebin][~xintongsong]
cc [~trohrmann] [~chesnay]
> Enable ResourceSpec to get/set whether managed memory is used
> -------------------------------------------------------------
>
> Key: FLINK-14566
> URL: https://issues.apache.org/jira/browse/FLINK-14566
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / Coordination
> Affects Versions: 1.10.0
> Reporter: Zhu Zhu
> Priority: Minor
> Fix For: 1.10.0
>
>
> To calculate managed memory fraction for an operator with UNKNOWN resources,
> we need to know whether the operator will use managed memory, according to
> FLINK-14062.
> To achieve this, we need an interface to set/get whether an operator uses
> managed memory. It can be part of {{ResourceSpec}} or directly locates in the
> operator description classes ({{Transformation}}/{{StreamNode}}).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
