[
https://issues.apache.org/jira/browse/FLINK-10240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17227397#comment-17227397
]
Andrey Zagrebin edited comment on FLINK-10240 at 11/6/20, 1:16 PM:
-------------------------------------------------------------------
[~zhuzh] do we still want to keep this ticket open? Its context seems to be
already too old.
The SchedulingStrategy is already pluggable and the scheduling is going to be
changed again quite a bit with declarative design.
Should we open another issue once it is clear what we want for this
optimisation with the new declarative scheduler?
was (Author: azagrebin):
[~zhuzh] do we still want to keep this ticket open? Its context seems to be
already too old.
The SchedulingStrategy is already pluggable and the scheduling is going to be
changed again quite a bit with declarative design.
Should we open another issue once it is clear what we want for this optimiation
with the new declarative scheduler?
> Pluggable scheduling strategy for batch jobs
> --------------------------------------------
>
> Key: FLINK-10240
> URL: https://issues.apache.org/jira/browse/FLINK-10240
> Project: Flink
> Issue Type: New Feature
> Components: Runtime / Coordination
> Affects Versions: 1.7.0
> Reporter: Zhu Zhu
> Priority: Major
> Labels: scheduling
>
> Currently batch jobs are scheduled with LAZY_FROM_SOURCES strategy: source
> tasks are scheduled in the beginning, and other tasks are scheduled once
> their input data is consumable.
> However, input data consumable does not always mean the task can work at
> once.
>
> One example is the hash join operation, where the operator first consumes one
> side(we call it build side) to setup a table, then consumes the other side(we
> call it probe side) to do the real join work. If the probe side is started
> early, it just get stuck on back pressure as the join operator will not
> consume data from it before the building stage is done, causing a waste of
> resources.
> If we have the probe side task started after the build stage is done, both
> the build and probe side can have more computing resources as they are
> staggered.
>
> That's why we think a flexible scheduling strategy is needed, allowing job
> owners to customize the vertex schedule order and constraints. Better
> resource utilization usually means better performance.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
