Zhu Zhu created FLINK-14162:
-------------------------------

             Summary: Unify SchedulerOperations#allocateSlotsAndDeploy 
implementation for all scheduling strategies
                 Key: FLINK-14162
                 URL: https://issues.apache.org/jira/browse/FLINK-14162
             Project: Flink
          Issue Type: Sub-task
          Components: Runtime / Coordination
    Affects Versions: 1.10.0
            Reporter: Zhu Zhu
             Fix For: 1.10.0


In scheduler NG, scheduling strategies invokes 
{{SchedulerOperations#allocateSlotsAndDeploy(Collection<ExecutionVertexDeploymentOption>)}}
 to trigger scheduling of tasks.

However, {{EagerSchedulingStrategy}} and {{LazyFromSourcesSchedulingStrategy}} 
both invokes it by passing a batch of tasks, but requires the scheduling 
process to be conducted in 2 different ways:
 * {{EagerSchedulingStrategy}} requires the batch of tasks to deploy after all 
of them have acquired slots. This is essential to avoid partition update RPCs 
in streaming job scheduling.
 * {{LazyFromSourcesSchedulingStrategy}} requires tasks in the batch to 
allocate slot and get deployed individually, so that it can deploy a few tasks 
even if the slots is not enough for all tasks in the batch. This is helpful for 
batch job scheduling.

The scheduler then have to decide the scheduling pattern based whether the 
scheduling strategy is a {{LazyFromSourcesSchedulingStrategy}}. This is not 
good, as there can be more strategies in the future, and even customized 
scheduling strategies.

I think it's better to define the 
{{SchedulerOperations#allocateSlotsAndDeploy(Collection<ExecutionVertexDeploymentOption>)}}
 to be that all tasks in the batch need to be assigned and deployed together, 
like what we do for {{EagerSchedulingStrategy}}.
All scheduling strategies need to follow this rule. If tasks should be 
scheduled individually, the strategy should invoke {{allocateSlotsAndDeploy}} 
multiple times, one for each task. As a result, the 
{{LazyFromSourcesSchedulingStrategy}} needs to be adjusted for that.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Reply via email to