[
https://issues.apache.org/jira/browse/FLINK-25036?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lijie Wang updated FLINK-25036:
-------------------------------
Description:
Considering adaptive batch scheduler currently only supports ALL-EDGES-BLOCKING
jobs, so we can schedule execution vertices in vertex wise). That means when an
execution vertex is finished, the following execution vertices will be checked
to see if they can be scheduled:
# All its consumer vertices.
# All execution vertices that are newly added to topology. It is needed
because otherwise some vertices will never be scheduled. e.g. If A1 finishes
first and A2 finishes later, no vertex will be scheduled when A1 finishes
because B1 has not been added to scheduling topology. And only B2 will be
scheduled when A2 finishes, B1 will never be scheduled if we don't try to
schedule the newly added vertices.
was:The scheduling of the adaptive batch scheduler only supports
ALL_BLOCKING_EDGES should be stage granularity, because the information for
deciding parallelism can only be collected after the upstream stage is fully
finished, so we need to introduce a new scheduling strategy: Stage-wised
Scheduling Strategy.
> Introduce vertex wise scheduling strategy
> -----------------------------------------
>
> Key: FLINK-25036
> URL: https://issues.apache.org/jira/browse/FLINK-25036
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / Coordination
> Reporter: Lijie Wang
> Assignee: Lijie Wang
> Priority: Major
> Labels: pull-request-available
>
> Considering adaptive batch scheduler currently only supports
> ALL-EDGES-BLOCKING jobs, so we can schedule execution vertices in vertex
> wise). That means when an execution vertex is finished, the following
> execution vertices will be checked to see if they can be scheduled:
> # All its consumer vertices.
> # All execution vertices that are newly added to topology. It is needed
> because otherwise some vertices will never be scheduled. e.g. If A1 finishes
> first and A2 finishes later, no vertex will be scheduled when A1 finishes
> because B1 has not been added to scheduling topology. And only B2 will be
> scheduled when A2 finishes, B1 will never be scheduled if we don't try to
> schedule the newly added vertices.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
