[
https://issues.apache.org/jira/browse/FLINK-15959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
YufeiLiu updated FLINK-15959:
-----------------------------
Description:
Flink removed `-n` option after FLIP-6, change to ResourceManager start a new
worker when required. But I think maintain a TaskExecutor number option is
necessary. These workers will start immediately when ResourceManager starts and
would not release even if all slots are free.
Here are some resons:
# Users actually know how many resources are needed when run a single job,
initialize all workers when cluster starts can speed up startup process.
# Job schedule in topology order, next operator won't schedule until prior
execution slot allocated. The TaskExecutors will start in several batchs in
some cases, it might slow down the startup speed.
# Flink support [FLINK-12122|https://issues.apache.org/jira/browse/FLINK-12122]
[Spread out tasks evenly across all available registered TaskManagers], but it
will only effect if all TMs are registered. Start all TMs at begining can slove
this problem.
*suggestion:*
I only changed YarnResourceManager, start all container in `initialize` stage,
and don't comlete slot request until minimum number of slots are registered.
was:
Flink removed `-n` option after FLIP-6, change to ResourceManager start a new
worker when required. But I think maintain a TaskExecutor number option is
necessary. These workers will start immediately when ResourceManager starts and
would not release even if all slots are free.
Here are some resons:
# Users actually know how many resources are needed when run a single job,
initialize all workers when cluster starts can speed up startup process.
# Job schedule in topology order, next operator won't schedule until prior
execution slot allocated. The TaskExecutors will start in several batchs in
some cases, it might slow down the startup speed.
# Flink support [FLINK-12122|https://issues.apache.org/jira/browse/FLINK-12122]
[Spread out tasks evenly across all available registered TaskManagers], but it
will only effect if all TMs are registered. Start all TMs at begining can slove
this problem.
> Add TaskExecutor number option in FlinkYarnSessionCli
> -----------------------------------------------------
>
> Key: FLINK-15959
> URL: https://issues.apache.org/jira/browse/FLINK-15959
> Project: Flink
> Issue Type: New Feature
> Components: Runtime / Coordination
> Affects Versions: 1.11.0
> Reporter: YufeiLiu
> Priority: Major
>
> Flink removed `-n` option after FLIP-6, change to ResourceManager start a new
> worker when required. But I think maintain a TaskExecutor number option is
> necessary. These workers will start immediately when ResourceManager starts
> and would not release even if all slots are free.
> Here are some resons:
> # Users actually know how many resources are needed when run a single job,
> initialize all workers when cluster starts can speed up startup process.
> # Job schedule in topology order, next operator won't schedule until prior
> execution slot allocated. The TaskExecutors will start in several batchs in
> some cases, it might slow down the startup speed.
> # Flink support
> [FLINK-12122|https://issues.apache.org/jira/browse/FLINK-12122] [Spread out
> tasks evenly across all available registered TaskManagers], but it will only
> effect if all TMs are registered. Start all TMs at begining can slove this
> problem.
> *suggestion:*
> I only changed YarnResourceManager, start all container in `initialize`
> stage, and don't comlete slot request until minimum number of slots are
> registered.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
