[
https://issues.apache.org/jira/browse/FLINK-14909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhu Zhu updated FLINK-14909:
----------------------------
Description:
I'd propose to let tasks in a batch get scheduled in topological order and
subtaskIndex ascending pattern.
There can be 2 benefits:
1. there would be less chance for a task to get launched before its upstream
tasks, which reduces {{requestPartitionState}} RPCs to JobMaster.
2. logs could be more readable, e.g.
ordered:
Source: source (1/20) ... switched from CREATED to SCHEDULED.
Source: source (2/20) ... switched from CREATED to SCHEDULED.
...
Source: source (20/20) ... switched from CREATED to SCHEDULED.
Flat Map (1/20) ... switched from CREATED to SCHEDULED.
...
Flat Map (20/20) ... switched from CREATED to SCHEDULED.
disordered:
Source: source (1/20) ... switched from CREATED to SCHEDULED.
Flat Map (11/20) ... switched from CREATED to SCHEDULED.
Source: source (19/20) ... switched from CREATED to SCHEDULED.
Flat Map (2/20) ... switched from CREATED to SCHEDULED.
...
The detailed proposal is:
1. change scheduling related methods to take and return tasks as {{List}}
instead of {{Collection}} in {{DefaultScheduler}} and related classes
2. sort the tasks in
was:
I'd propose to let tasks in a batch get scheduled in topological order and
subtaskIndex ascending pattern.
There can be 2 benefits:
1. there would be less chance for a task to get launched before its upstream
tasks, which reduces {{requestPartitionState}} RPCs to JobMaster.
2. logs could be more readable, e.g.
ordered:
Source: source (1/20) ... switched from CREATED to SCHEDULED.
Source: source (2/20) ... switched from CREATED to SCHEDULED.
...
Source: source (20/20) ... switched from CREATED to SCHEDULED.
Flat Map (1/20) ... switched from CREATED to SCHEDULED.
...
Flat Map (20/20) ... switched from CREATED to SCHEDULED.
disordered:
Source: source (1/20) ... switched from CREATED to SCHEDULED.
Flat Map (11/20) ... switched from CREATED to SCHEDULED.
Source: source (19/20) ... switched from CREATED to SCHEDULED.
Flat Map (2/20) ... switched from CREATED to SCHEDULED.
...
Changes should be made to both {{EagerSchedulingStrategy}} and
{{LazyFromSourcesSchedulingStrategy}}.
> Let tasks in a batch get scheduled in topological order and subtaskIndex
> ascending pattern
> ------------------------------------------------------------------------------------------
>
> Key: FLINK-14909
> URL: https://issues.apache.org/jira/browse/FLINK-14909
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / Coordination
> Affects Versions: 1.10.0
> Reporter: Zhu Zhu
> Assignee: Zhu Zhu
> Priority: Major
> Fix For: 1.10.0
>
>
> I'd propose to let tasks in a batch get scheduled in topological order and
> subtaskIndex ascending pattern.
> There can be 2 benefits:
> 1. there would be less chance for a task to get launched before its upstream
> tasks, which reduces {{requestPartitionState}} RPCs to JobMaster.
> 2. logs could be more readable, e.g.
> ordered:
> Source: source (1/20) ... switched from CREATED to SCHEDULED.
> Source: source (2/20) ... switched from CREATED to SCHEDULED.
> ...
> Source: source (20/20) ... switched from CREATED to SCHEDULED.
> Flat Map (1/20) ... switched from CREATED to SCHEDULED.
> ...
> Flat Map (20/20) ... switched from CREATED to SCHEDULED.
> disordered:
> Source: source (1/20) ... switched from CREATED to SCHEDULED.
> Flat Map (11/20) ... switched from CREATED to SCHEDULED.
> Source: source (19/20) ... switched from CREATED to SCHEDULED.
> Flat Map (2/20) ... switched from CREATED to SCHEDULED.
> ...
> The detailed proposal is:
> 1. change scheduling related methods to take and return tasks as {{List}}
> instead of {{Collection}} in {{DefaultScheduler}} and related classes
> 2. sort the tasks in
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
