walterddr commented on a change in pull request #10143: [FLINK-13184]Starting a
TaskExecutor blocks the YarnResourceManager's main thread
URL: https://github.com/apache/flink/pull/10143#discussion_r345537005
##########
File path: flink-yarn/src/main/java/org/apache/flink/yarn/Utils.java
##########
@@ -641,4 +610,32 @@ static void require(boolean condition, String message,
Object... values) {
}
}
+ /**
+ * Get dynamic properties based on two Flink configuration. If base
config does not contain and target config
+ * contains the key or the value is different, it should be added to
results. Otherwise, if the base config contains
+ * and target config does not contain the key, it will be ignored.
+ * @param baseConfig The base configuration.
+ * @param targetConfig The target configuration.
+ * @return Dynamic properties as string, separated by space.
+ */
+ static String getDynamicProperties(
Review comment:
I asked around, seems like the startup script is cached but the NM thread in
some secure path (probably in `nmPrivate`, or somewhere private to the
container itself).
In this case, is the following claim true?
> In the situation where, say one worker container is dead, YARN will have
to re-ship the entire payload of the launch command to another new host and
restart it again. vs if the file is on HDFS, YARN RM can simply just ask the NM
to re-localize the file from HDFS.
So the follow up question is: If the launch command is huge (say the
properties we attached to the launch command as dynamic properties is beyond
1MB) would that cause any problem?
disclaimer: I haven't checked YARN code to verify this finding yet.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
