wangyang0918 commented on a change in pull request #10143:
[FLINK-13184]Starting a TaskExecutor blocks the YarnResourceManager's main
thread
URL: https://github.com/apache/flink/pull/10143#discussion_r344660476
##########
File path: flink-yarn/src/main/java/org/apache/flink/yarn/Utils.java
##########
@@ -491,41 +492,9 @@ static ContainerLaunchContext createTaskExecutorContext(
flinkJar = registerLocalResource(fs, remoteJarPath);
}
- // register conf with local fs
- final LocalResource flinkConf;
Review comment:
Uploading a taskmanager-conf.yaml to hdfs will take 50ms in many production
hdfs cluster. If we receive more that 1000 containers, this will take long time
in `YarnResourceManager` main thread. In addition, all the
{uuid}-taskmanager-conf.yaml are same and have very small difference with flink
config uploaded by client.
So i suggest to remove the io operation in start TaskExecutor. About
`NMClientAsync`, it is another optimazion to eliminate the rpc blocking call in
`YarnResourceManager` main thread.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
