[
https://issues.apache.org/jira/browse/FLINK-25046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lijie Wang updated FLINK-25046:
-------------------------------
Description:
Currently, unspecified edges(partitioner == null) will be setted to {{FORWARD}}
when the parallelism of both upstream and downstream is -1. This may cause many
job vertices whose parallelism is not calculated based on data volume, but is
aligned with their upstream vertices' parallelism because of forward edges,
which is contrary to the original intention of our adaptive batch scheduler.
For this problem, we will convert the unspecfied edges between OperatorChains
to {{RESCALE}} edges(instead of {{FORWARD}} edges) after OperatorChains'
creation.
> Convert unspecified edge to rescale when using adapive batch scheduler
> ----------------------------------------------------------------------
>
> Key: FLINK-25046
> URL: https://issues.apache.org/jira/browse/FLINK-25046
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / Coordination
> Reporter: Lijie Wang
> Priority: Major
>
> Currently, unspecified edges(partitioner == null) will be setted to
> {{FORWARD}} when the parallelism of both upstream and downstream is -1. This
> may cause many job vertices whose parallelism is not calculated based on data
> volume, but is aligned with their upstream vertices' parallelism because of
> forward edges, which is contrary to the original intention of our adaptive
> batch scheduler.
> For this problem, we will convert the unspecfied edges between OperatorChains
> to {{RESCALE}} edges(instead of {{FORWARD}} edges) after OperatorChains'
> creation.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
