lewismc commented on code in PR #8331:
URL: https://github.com/apache/hadoop/pull/8331#discussion_r2918077873
##########
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java:
##########
@@ -503,7 +503,9 @@ private List<String> setupAMCommand(Configuration jobConf) {
}
}
- vargs.add(MRJobConfig.APPLICATION_MASTER_CLASS);
+ String amClass = jobConf.get("yarn.app.mapreduce.am",
Review Comment:
Hi @RexXiong thanks.
I wasn't able to get that working because of how Hadoop builds the AM
container command and what yarn.app.mapreduce.am.command-opts is used for.
# How the AM is launched
In `YARNRunner`, the command for the AM container is built in two separate
steps:
1. Main class – One place in code does:
* `vargs.add(MRJobConfig.APPLICATION_MASTER_CLASS);` So the main class is
always `org.apache.hadoop.mapreduce.v2.app.MRAppMaster`. That value is
hardcoded; no config key is read for it.
2. Command opts – `yarn.app.mapreduce.am.command-opts` is used elsewhere for
JVM options or extra arguments. Those are merged into the same command, but
they are not used as the main class. So they end up either:
* as JVM args (e.g. `-Xmx...`), or
* as arguments passed to the main class (i.e. to `MRAppMaster`).
So the actual process looks like:
`java [options from command-opts]
org.apache.hadoop.mapreduce.v2.app.MRAppMaster [any extra args] 1>... 2>...`
If you set:
`-Dyarn.app.mapreduce.am.command-opts=org.apache.celeborn.mapreduce.v2.app.MRAppMasterWithCeleborn`
then that string is treated as part of “options” or “extra args”. It does
not replace the main class, so:
* JVM still runs `MRAppMaster` as main.
* `MRAppMasterWithCeleborn` is at best an argument to `MRAppMaster`, not the
entry point.
The JVM never executes `MRAppMasterWithCeleborn` as the main class; it
always runs `MRAppMaster`. I wasn't able to get the example from the Celeborn
doc’s method running without this patch. The main class is fixed in code, and
command-opts never controls it.
Unless I am mistaken, to actually run Celeborn’s AM, the main class in that
launch command must be `MRAppMasterWithCeleborn`. The only way to do that with
the current design is to change the code that builds the command so it takes
the main class from config (e.g. `yarn.app.mapreduce.am` or as proposed by
@pan3793 `yarn.app.mapreduce.am.class`) instead of always using
`APPLICATION_MASTER_CLASS`. That’s what this patch for `YARNRunner` does;
command-opts alone can’t do it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
