walterddr commented on a change in pull request #10143: [FLINK-13184]Starting a
TaskExecutor blocks the YarnResourceManager's main thread
URL: https://github.com/apache/flink/pull/10143#discussion_r345536226
##########
File path: flink-yarn/src/main/java/org/apache/flink/yarn/Utils.java
##########
@@ -491,41 +492,9 @@ static ContainerLaunchContext createTaskExecutorContext(
flinkJar = registerLocalResource(fs, remoteJarPath);
}
- // register conf with local fs
- final LocalResource flinkConf;
Review comment:
in this case, I think the real bottle neck is the HDFS synchronized upload.
I poked around and it seems like there's really no way to make it async,
please correct me if I were wrong:
> if there's no guarantee that all 100 files had been uploaded to HDFS,
launching the request to YARN RM will result in a container localization
failure (because when RM tries to ask NM to localize the container, the files
may not be available yet in the HDFS)
follow up question: is it possible to upload just one file (by using the
setupLocalResource) once and reuse them across different TMs? this makes the
argument length of the startup script (of the TM) much shorter.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
