Zhu Zhu created FLINK-14162:
-------------------------------
Summary: Unify SchedulerOperations#allocateSlotsAndDeploy
implementation for all scheduling strategies
Key: FLINK-14162
URL: https://issues.apache.org/jira/browse/FLINK-14162
Project: Flink
Issue Type: Sub-task
Components: Runtime / Coordination
Affects Versions: 1.10.0
Reporter: Zhu Zhu
Fix For: 1.10.0
In scheduler NG, scheduling strategies invokes
{{SchedulerOperations#allocateSlotsAndDeploy(Collection<ExecutionVertexDeploymentOption>)}}
to trigger scheduling of tasks.
However, {{EagerSchedulingStrategy}} and {{LazyFromSourcesSchedulingStrategy}}
both invokes it by passing a batch of tasks, but requires the scheduling
process to be conducted in 2 different ways:
* {{EagerSchedulingStrategy}} requires the batch of tasks to deploy after all
of them have acquired slots. This is essential to avoid partition update RPCs
in streaming job scheduling.
* {{LazyFromSourcesSchedulingStrategy}} requires tasks in the batch to
allocate slot and get deployed individually, so that it can deploy a few tasks
even if the slots is not enough for all tasks in the batch. This is helpful for
batch job scheduling.
The scheduler then have to decide the scheduling pattern based whether the
scheduling strategy is a {{LazyFromSourcesSchedulingStrategy}}. This is not
good, as there can be more strategies in the future, and even customized
scheduling strategies.
I think it's better to define the
{{SchedulerOperations#allocateSlotsAndDeploy(Collection<ExecutionVertexDeploymentOption>)}}
to be that all tasks in the batch need to be assigned and deployed together,
like what we do for {{EagerSchedulingStrategy}}.
All scheduling strategies need to follow this rule. If tasks should be
scheduled individually, the strategy should invoke {{allocateSlotsAndDeploy}}
multiple times, one for each task. As a result, the
{{LazyFromSourcesSchedulingStrategy}} needs to be adjusted for that.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
