[
https://issues.apache.org/jira/browse/FLINK-15959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17034238#comment-17034238
]
YufeiLiu edited comment on FLINK-15959 at 2/11/20 8:33 AM:
-----------------------------------------------------------
[~sewen] I also feel like it's a backwards when I propose this feature. But I
did have some load balancing issue in our production, I think it's not bad to
give user a right to choose.
was (Author: liuyufei):
[~sewen] I also feel like it's a backwards when I propose this feature. But I
did have some load balancing issue in our production, I think it's not bad to
give user a right to choice.
> Add TaskExecutor number option in FlinkYarnSessionCli
> -----------------------------------------------------
>
> Key: FLINK-15959
> URL: https://issues.apache.org/jira/browse/FLINK-15959
> Project: Flink
> Issue Type: New Feature
> Components: Runtime / Coordination
> Affects Versions: 1.11.0
> Reporter: YufeiLiu
> Priority: Major
>
> Flink removed `-n` option after FLIP-6, change to ResourceManager start a new
> worker when required. But I think maintain a TaskExecutor number option is
> necessary. These workers will start immediately when ResourceManager starts
> and would not release even if all slots are free.
> Here are some resons:
> # Users actually know how many resources are needed when run a single job,
> initialize all workers when cluster starts can speed up startup process.
> # Job schedule in topology order, next operator won't schedule until prior
> execution slot allocated. The TaskExecutors will start in several batchs in
> some cases, it might slow down the startup speed.
> # Flink support
> [FLINK-12122|https://issues.apache.org/jira/browse/FLINK-12122] [Spread out
> tasks evenly across all available registered TaskManagers], but it will only
> effect if all TMs are registered. Start all TMs at begining can slove this
> problem.
> *suggestion:*
> I only changed YarnResourceManager, start all container in `initialize`
> stage, and don't comlete slot request until minimum number of slots are
> registered.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
