[
https://issues.apache.org/jira/browse/FLINK-14582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16963942#comment-16963942
]
Yang Wang commented on FLINK-14582:
-----------------------------------
>From the experience of our cluster, uploading a conf file to hdfs will take
>more than 50ms. If the name node of hdfs cluster is busy, it will task several
>seconds on a big hdfs cluster.
The time cost leads that we could only remove 10 container request in
{{onContainersAllocated}} when {{containerRequestHeartbeat}} is 500ms. So the
{{YarnResourceManager}} will allocate more containers than it really needs.
> Do not upload {uuid}-taskmanager-conf.yaml for each task manager container
> --------------------------------------------------------------------------
>
> Key: FLINK-14582
> URL: https://issues.apache.org/jira/browse/FLINK-14582
> Project: Flink
> Issue Type: Improvement
> Components: Deployment / YARN
> Affects Versions: 1.10.0, 1.9.1
> Reporter: Yang Wang
> Priority: Minor
>
> Currently when we {{createTaskExecutorLaunchContext}}, a new task manager
> conf will be created and uploaded to hdfs. The time cost is unnecessary and
> could be reduced by using java options of task manager main class instead.
> By reducing the time of launch container in {{onContainersAllocated}}, we
> could avoid allocating excess containers as much as possible.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
