[
https://issues.apache.org/jira/browse/FLINK-10240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16614335#comment-16614335
]
陈梓立 commented on FLINK-10240:
-----------------------------
Introduce pluggable schedule strategy is an excellent idea that could expand a
lot the cases Flink is able to handle.
I like this idea and can help. However, the document attached above is
read-only. So I remain my comments as a link to a copy of it below. Most of
them are layout improvements and minor reword, the body of document is no more
than the original design.
https://docs.google.com/document/d/15pUYc5_yrY2IwmnADCoNWZwOIYCOcroWmuuZHs-vdlU/edit?usp=sharing
Note that this is a EDITABLE document and everyone interest on it can remains
comments or edit it directly. As an open source software we just trust our
contributors and the document could be frozen and left comment-only if the
discussion reaches a consensus.
> Pluggable scheduling strategy for batch job
> -------------------------------------------
>
> Key: FLINK-10240
> URL: https://issues.apache.org/jira/browse/FLINK-10240
> Project: Flink
> Issue Type: New Feature
> Components: Distributed Coordination
> Reporter: Zhu Zhu
> Priority: Major
> Labels: scheduling
>
> Currently batch jobs are scheduled with LAZY_FROM_SOURCES strategy: source
> tasks are scheduled in the beginning, and other tasks are scheduled once
> there input data are consumable.
> However, input data consumable does not always mean the task can work at
> once.
>
> One example is the hash join operation, where the operator first consumes one
> side(we call it build side) to setup a table, then consumes the other side(we
> call it probe side) to do the real join work. If the probe side is started
> early, it just get stuck on back pressure as the join operator will not
> consume data from it before the building stage is done, causing a waste of
> resources.
> If we have the probe side task started after the build stage is done, both
> the build and probe side can have more computing resources as they are
> staggered.
>
> That's why we think a flexible scheduling strategy is needed, allowing job
> owners to customize the vertex schedule order and constraints. Better
> resource utilization usually means better performance.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
