[
https://issues.apache.org/jira/browse/TEZ-4175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
László Bodor updated TEZ-4175:
------------------------------
Description:
A comment in DAGAppmaster made me think that we don't need to rely on
[YarnConfiguration|https://github.com/apache/hadoop/blob/branch-3.1.3/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java]
in all cases, what if it can be replace with base Configuration...
{code}
// TODO Does this really need to be a YarnConfiguration ?
Configuration conf = new Configuration(new YarnConfiguration());
{code}
In hadoop 3.1.3 source, I cannot see that it adds e.g. yarn-site as a resource:
{code}
public YarnConfiguration() {
super();
}
public YarnConfiguration(Configuration conf) {
super(conf);
if (! (conf instanceof YarnConfiguration)) {
this.reloadConfiguration();
}
}
{code}
UPDATE: YarnConfiguration adds yarn-site as a default resource [in a static
initializer|https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java#L100-L102],
so it should be used when yarn configs are expected to be present
in current codebase:
{code}
grep -iRH "new YarnConfiguration" --include="*.java"
tez-plugins/tez-history-parser/src/main/java/org/apache/tez/history/ATSImportTool.java:
YarnConfiguration yarnConf = new YarnConfiguration(conf);
tez-plugins/tez-aux-services/src/main/java/org/apache/tez/auxservices/ShuffleHandler.java:
super.serviceInit(new YarnConfiguration(conf));
tez-api/src/test/java/org/apache/tez/dag/api/client/rpc/TestDAGClient.java:
YarnConfiguration yarnConf = new YarnConfiguration(tezConf);
tez-api/src/test/java/org/apache/tez/dag/api/client/rpc/TestDAGClient.java:
YarnConfiguration yarnConf = new YarnConfiguration(tezConf);
tez-api/src/test/java/org/apache/tez/dag/api/client/rpc/TestDAGClient.java:
YarnConfiguration yarnConf = new YarnConfiguration(tezConf);
tez-api/src/test/java/org/apache/tez/client/TestTezClient.java:
tezClient.init(new TezConfiguration(false), new YarnConfiguration());
tez-api/src/main/java/org/apache/tez/client/TezClient.java:
amConfig.setYarnConfiguration(new
YarnConfiguration(amConfig.getTezConfiguration()));
tez-api/src/main/java/org/apache/tez/client/TezClient.java:
amConfig.setYarnConfiguration(new
YarnConfiguration(amConfig.getTezConfiguration()));
tez-api/src/main/java/org/apache/tez/client/TezClient.java: return
getDAGClient(appId, tezConf, new YarnConfiguration(tezConf), frameworkClient,
ugi);
tez-tests/src/test/java/org/apache/tez/test/FaultToleranceTestRunner.java:
tezConf = new TezConfiguration(new YarnConfiguration());
tez-tests/src/test/java/org/apache/tez/test/FaultToleranceTestRunner.java:
tezConf = new TezConfiguration(new YarnConfiguration(this.conf));
tez-mapreduce/src/test/java/org/apache/tez/mapreduce/hadoop/TestMRInputHelpers.java:
Configuration testConf = new YarnConfiguration(
tez-mapreduce/src/main/java/org/apache/tez/mapreduce/client/YARNRunner.java:
this(conf, new ResourceMgrDelegate(new YarnConfiguration(conf)));
tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java:
Configuration conf = new Configuration(new YarnConfiguration());
tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java:
Configuration conf = new Configuration(new YarnConfiguration());
tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java:
Configuration tezConf = new Configuration(new YarnConfiguration());
tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java:
Configuration tezConf = new Configuration(new YarnConfiguration());
tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java:
Configuration tezConf = new Configuration(new YarnConfiguration());
tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java:
Configuration tezConf = new Configuration(new YarnConfiguration());
tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java:
Configuration tezConf = new Configuration(new YarnConfiguration());
tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java:
Configuration tezConf = new Configuration(new YarnConfiguration());
tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java:
Configuration tezConf = new Configuration(new YarnConfiguration());
tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java:
Configuration tezConf = new Configuration(new YarnConfiguration());
tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java:
Configuration tezConf = new Configuration(new YarnConfiguration());
tez-dag/src/main/java/org/apache/tez/dag/app/DAGAppMaster.java:
Configuration conf = new Configuration(new YarnConfiguration());
{code}
was:
A comment in DAGAppmaster made me think that we don't need to rely on
[YarnConfiguration|https://github.com/apache/hadoop/blob/branch-3.1.3/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java]
in all cases, what if it can be replace with base Configuration...
{code}
// TODO Does this really need to be a YarnConfiguration ?
Configuration conf = new Configuration(new YarnConfiguration());
{code}
In hadoop 3.1.3 source, I cannot see that it adds e.g. yarn-site as a resource:
{code}
public YarnConfiguration() {
super();
}
public YarnConfiguration(Configuration conf) {
super(conf);
if (! (conf instanceof YarnConfiguration)) {
this.reloadConfiguration();
}
}
{code}
UPDATE: it adds yarn-site as a default resource in its static initializer, so
it should be used in cases where the Configuration user expects yarn configs to
be present
in current codebase:
{code}
grep -iRH "new YarnConfiguration" --include="*.java"
tez-plugins/tez-history-parser/src/main/java/org/apache/tez/history/ATSImportTool.java:
YarnConfiguration yarnConf = new YarnConfiguration(conf);
tez-plugins/tez-aux-services/src/main/java/org/apache/tez/auxservices/ShuffleHandler.java:
super.serviceInit(new YarnConfiguration(conf));
tez-api/src/test/java/org/apache/tez/dag/api/client/rpc/TestDAGClient.java:
YarnConfiguration yarnConf = new YarnConfiguration(tezConf);
tez-api/src/test/java/org/apache/tez/dag/api/client/rpc/TestDAGClient.java:
YarnConfiguration yarnConf = new YarnConfiguration(tezConf);
tez-api/src/test/java/org/apache/tez/dag/api/client/rpc/TestDAGClient.java:
YarnConfiguration yarnConf = new YarnConfiguration(tezConf);
tez-api/src/test/java/org/apache/tez/client/TestTezClient.java:
tezClient.init(new TezConfiguration(false), new YarnConfiguration());
tez-api/src/main/java/org/apache/tez/client/TezClient.java:
amConfig.setYarnConfiguration(new
YarnConfiguration(amConfig.getTezConfiguration()));
tez-api/src/main/java/org/apache/tez/client/TezClient.java:
amConfig.setYarnConfiguration(new
YarnConfiguration(amConfig.getTezConfiguration()));
tez-api/src/main/java/org/apache/tez/client/TezClient.java: return
getDAGClient(appId, tezConf, new YarnConfiguration(tezConf), frameworkClient,
ugi);
tez-tests/src/test/java/org/apache/tez/test/FaultToleranceTestRunner.java:
tezConf = new TezConfiguration(new YarnConfiguration());
tez-tests/src/test/java/org/apache/tez/test/FaultToleranceTestRunner.java:
tezConf = new TezConfiguration(new YarnConfiguration(this.conf));
tez-mapreduce/src/test/java/org/apache/tez/mapreduce/hadoop/TestMRInputHelpers.java:
Configuration testConf = new YarnConfiguration(
tez-mapreduce/src/main/java/org/apache/tez/mapreduce/client/YARNRunner.java:
this(conf, new ResourceMgrDelegate(new YarnConfiguration(conf)));
tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java:
Configuration conf = new Configuration(new YarnConfiguration());
tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java:
Configuration conf = new Configuration(new YarnConfiguration());
tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java:
Configuration tezConf = new Configuration(new YarnConfiguration());
tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java:
Configuration tezConf = new Configuration(new YarnConfiguration());
tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java:
Configuration tezConf = new Configuration(new YarnConfiguration());
tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java:
Configuration tezConf = new Configuration(new YarnConfiguration());
tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java:
Configuration tezConf = new Configuration(new YarnConfiguration());
tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java:
Configuration tezConf = new Configuration(new YarnConfiguration());
tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java:
Configuration tezConf = new Configuration(new YarnConfiguration());
tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java:
Configuration tezConf = new Configuration(new YarnConfiguration());
tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java:
Configuration tezConf = new Configuration(new YarnConfiguration());
tez-dag/src/main/java/org/apache/tez/dag/app/DAGAppMaster.java:
Configuration conf = new Configuration(new YarnConfiguration());
{code}
> Consider removing YarnConfiguration where it's possible
> -------------------------------------------------------
>
> Key: TEZ-4175
> URL: https://issues.apache.org/jira/browse/TEZ-4175
> Project: Apache Tez
> Issue Type: Improvement
> Reporter: László Bodor
> Assignee: László Bodor
> Priority: Major
> Attachments: TEZ-4175.01.patch, TEZ-4175.02.patch, TEZ-4175.03.patch,
> TEZ-4175.03.patch, TEZ-4175.04.patch, TEZ-4175.05.patch
>
>
> A comment in DAGAppmaster made me think that we don't need to rely on
> [YarnConfiguration|https://github.com/apache/hadoop/blob/branch-3.1.3/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java]
> in all cases, what if it can be replace with base Configuration...
> {code}
> // TODO Does this really need to be a YarnConfiguration ?
> Configuration conf = new Configuration(new YarnConfiguration());
> {code}
> In hadoop 3.1.3 source, I cannot see that it adds e.g. yarn-site as a
> resource:
> {code}
> public YarnConfiguration() {
> super();
> }
>
> public YarnConfiguration(Configuration conf) {
> super(conf);
> if (! (conf instanceof YarnConfiguration)) {
> this.reloadConfiguration();
> }
> }
> {code}
> UPDATE: YarnConfiguration adds yarn-site as a default resource [in a static
> initializer|https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java#L100-L102],
> so it should be used when yarn configs are expected to be present
> in current codebase:
> {code}
> grep -iRH "new YarnConfiguration" --include="*.java"
> tez-plugins/tez-history-parser/src/main/java/org/apache/tez/history/ATSImportTool.java:
> YarnConfiguration yarnConf = new YarnConfiguration(conf);
> tez-plugins/tez-aux-services/src/main/java/org/apache/tez/auxservices/ShuffleHandler.java:
> super.serviceInit(new YarnConfiguration(conf));
> tez-api/src/test/java/org/apache/tez/dag/api/client/rpc/TestDAGClient.java:
> YarnConfiguration yarnConf = new YarnConfiguration(tezConf);
> tez-api/src/test/java/org/apache/tez/dag/api/client/rpc/TestDAGClient.java:
> YarnConfiguration yarnConf = new YarnConfiguration(tezConf);
> tez-api/src/test/java/org/apache/tez/dag/api/client/rpc/TestDAGClient.java:
> YarnConfiguration yarnConf = new YarnConfiguration(tezConf);
> tez-api/src/test/java/org/apache/tez/client/TestTezClient.java:
> tezClient.init(new TezConfiguration(false), new YarnConfiguration());
> tez-api/src/main/java/org/apache/tez/client/TezClient.java:
> amConfig.setYarnConfiguration(new
> YarnConfiguration(amConfig.getTezConfiguration()));
> tez-api/src/main/java/org/apache/tez/client/TezClient.java:
> amConfig.setYarnConfiguration(new
> YarnConfiguration(amConfig.getTezConfiguration()));
> tez-api/src/main/java/org/apache/tez/client/TezClient.java: return
> getDAGClient(appId, tezConf, new YarnConfiguration(tezConf), frameworkClient,
> ugi);
> tez-tests/src/test/java/org/apache/tez/test/FaultToleranceTestRunner.java:
> tezConf = new TezConfiguration(new YarnConfiguration());
> tez-tests/src/test/java/org/apache/tez/test/FaultToleranceTestRunner.java:
> tezConf = new TezConfiguration(new YarnConfiguration(this.conf));
> tez-mapreduce/src/test/java/org/apache/tez/mapreduce/hadoop/TestMRInputHelpers.java:
> Configuration testConf = new YarnConfiguration(
> tez-mapreduce/src/main/java/org/apache/tez/mapreduce/client/YARNRunner.java:
> this(conf, new ResourceMgrDelegate(new YarnConfiguration(conf)));
> tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java:
> Configuration conf = new Configuration(new YarnConfiguration());
> tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java:
> Configuration conf = new Configuration(new YarnConfiguration());
> tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java:
> Configuration tezConf = new Configuration(new YarnConfiguration());
> tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java:
> Configuration tezConf = new Configuration(new YarnConfiguration());
> tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java:
> Configuration tezConf = new Configuration(new YarnConfiguration());
> tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java:
> Configuration tezConf = new Configuration(new YarnConfiguration());
> tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java:
> Configuration tezConf = new Configuration(new YarnConfiguration());
> tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java:
> Configuration tezConf = new Configuration(new YarnConfiguration());
> tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java:
> Configuration tezConf = new Configuration(new YarnConfiguration());
> tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java:
> Configuration tezConf = new Configuration(new YarnConfiguration());
> tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java:
> Configuration tezConf = new Configuration(new YarnConfiguration());
> tez-dag/src/main/java/org/apache/tez/dag/app/DAGAppMaster.java:
> Configuration conf = new Configuration(new YarnConfiguration());
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
