[
https://issues.apache.org/jira/browse/FLINK-31827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gyula Fora closed FLINK-31827.
------------------------------
Resolution: Fixed
> Incorrect estimation of the target data rate of a vertex when only a subset
> of its upstream vertex's output is consumed
> -----------------------------------------------------------------------------------------------------------------------
>
> Key: FLINK-31827
> URL: https://issues.apache.org/jira/browse/FLINK-31827
> Project: Flink
> Issue Type: Bug
> Components: Autoscaler
> Reporter: Zhanghao Chen
> Assignee: Gyula Fora
> Priority: Major
> Labels: pull-request-available, stale-assigned
> Attachments: image-2023-04-17-23-37-35-280.png
>
>
> Currently, the target data rate of a vertex = SUM(target data rate *
> input/output ratio) for all of its upstream vertices. This assumes that all
> output records of an upstream vertex is consumed by the downstream vertex.
> However, it does not always hold. Consider the following job plan generated
> by a Flink SQL job. The middle vertex contains multiple chained Calc(select
> xx) operators, each connecting to a separate downstream sink tasks. As a
> result, each sink task only consumes a sub-portion of the middle vertex's
> output.
> To fix it, we need operator level edge info to infer the upstream-downstream
> relationship as well as operator level output metrics. The metrics part is
> easy but AFAIK, there's no way to get the operator level edge info from the
> Flink REST API yet.
> !image-2023-04-17-23-37-35-280.png!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
