Lewis John McGibbney created HADOOP-19835:
---------------------------------------------
Summary: Make MapReduce Application Master class configurable in
YARNRunner
Key: HADOOP-19835
URL: https://issues.apache.org/jira/browse/HADOOP-19835
Project: Hadoop Common
Issue Type: Bug
Components: mapreduce
Affects Versions: 3.4.3
Reporter: Lewis John McGibbney
Fix For: 3.4.4
YARNRunner currently always uses {{MRJobConfig.APPLICATION_MASTER_CLASS}} when
building the command to start the Application Master, and does not read
{{{}yarn.app.mapreduce.am{}}}. As a result, custom AM classes (e.g. Celeborn’s
{{{}MRAppMasterWithCeleborn{}}}) are never used when submitting jobs via the
standard job client.
This patch changes YARNRunner to take the AM class from configuration: it uses
{{jobConf.get("yarn.app.mapreduce.am", MRJobConfig.APPLICATION_MASTER_CLASS)}}
and passes that value into the launch arguments. If {{yarn.app.mapreduce.am}}
is set, that class is used; otherwise behavior is unchanged (default AM class).
This allows remote shuffle and other custom AMs to be used without forking the
job client.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
