[
https://issues.apache.org/jira/browse/FLINK-14536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16963630#comment-16963630
]
zhijiang commented on FLINK-14536:
----------------------------------
For the chained operators, the final aggregated cpu resource chose max() and
the memory resource chose sum().
It is based on the consideration that all the chained operators must execute in
a serial way. That means only one single operator can execute to use cpu at any
time, so the max cpu resource can be reused to satisfy any chained operator
requirement.
But for the memory, it might not be released immediately by the former
operator, so the next chained operator can not reuse this portion of memory,
then we chose the sum() way to satisfy all the chained operators.
For the case of shared slot, all the internal tasks might run in parallel, so
the cpu resource should be the sum() to satisfy all the task running.
>From the respective of users, it is also reasonable to use sum() for the cpu
>of chained operators, because the users should not understand the transparent
>chained improvement to save cpu resources to some extent.
Or from another dimension if we regard the cpu resource as execution time for a
single operator, then it also make sense to use sum() for the chained operators
because the whole would occupy more time to execute.
> Make clear the way to aggregate specified cpuCores resources
> ------------------------------------------------------------
>
> Key: FLINK-14536
> URL: https://issues.apache.org/jira/browse/FLINK-14536
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / Coordination
> Affects Versions: 1.10.0
> Reporter: Zhu Zhu
> Priority: Major
> Fix For: 1.10.0
>
>
> I'm raising this question because I find {{cpuCores}} in
> {{ResourceSpec#merge}} are aggregated with {{max()}}, while in
> {{ResourceProfile#merge}} it is {{sum()}}.
> This means that when calculating resources of a vertex from within operators,
> the {{cpuCores}} is the max value. While it is a sum(or subtraction in
> {{ResourceProfile#subtract}}) when dealing with shared slot bookkeeping and
> related checks.
> This is confusing to me, especially when I'm considering how to generate a
> shared slot resource spec merged from all vertices in it(see FLINK-14314).
> I'm not pretty sure if we already have a concise definition for it?
> If there is, we need to respect it and change {{ResourceSpec}} or
> {{ResourceProfile}} accordingly.
> If not, we need to decide it first before we can move on with fine grained
> resources.
> Need to mention that if we take the {{max()}} way, we need to re-consider how
> we can conduct a correct {{ResourceProfile#subtract}} regarding {{cpuCores}},
> since there is not a clear way to reverse a {{max()}} computation.
> cc [~trohrmann] [~xintongsong] [~azagrebin]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
