[ https://issues.apache.org/jira/browse/TEZ-4175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17103232#comment-17103232 ]
László Bodor commented on TEZ-4175: ----------------------------------- uploaded experimental patch [^TEZ-4175.01.patch] for kicking off tests > Consider removing YarnConfiguration where it's possible > ------------------------------------------------------- > > Key: TEZ-4175 > URL: https://issues.apache.org/jira/browse/TEZ-4175 > Project: Apache Tez > Issue Type: Improvement > Reporter: László Bodor > Assignee: László Bodor > Priority: Major > Attachments: TEZ-4175.01.patch > > > A comment in DAGAppmaster made me think that we don't need to rely on > [YarnConfiguration|https://github.com/apache/hadoop/blob/branch-3.1.3/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java] > in all cases, what if it can be replace with base Configuration... > {code} > // TODO Does this really need to be a YarnConfiguration ? > Configuration conf = new Configuration(new YarnConfiguration()); > {code} > From hadoop 3.1.3, I cannot see that it adds e.g. yarn-site as a resource: > {code} > public YarnConfiguration() { > super(); > } > > public YarnConfiguration(Configuration conf) { > super(conf); > if (! (conf instanceof YarnConfiguration)) { > this.reloadConfiguration(); > } > } > {code} > in current codebase: > {code} > grep -iRH "new YarnConfiguration" --include="*.java" > tez-plugins/tez-history-parser/src/main/java/org/apache/tez/history/ATSImportTool.java: > YarnConfiguration yarnConf = new YarnConfiguration(conf); > tez-plugins/tez-aux-services/src/main/java/org/apache/tez/auxservices/ShuffleHandler.java: > super.serviceInit(new YarnConfiguration(conf)); > tez-api/src/test/java/org/apache/tez/dag/api/client/rpc/TestDAGClient.java: > YarnConfiguration yarnConf = new YarnConfiguration(tezConf); > tez-api/src/test/java/org/apache/tez/dag/api/client/rpc/TestDAGClient.java: > YarnConfiguration yarnConf = new YarnConfiguration(tezConf); > tez-api/src/test/java/org/apache/tez/dag/api/client/rpc/TestDAGClient.java: > YarnConfiguration yarnConf = new YarnConfiguration(tezConf); > tez-api/src/test/java/org/apache/tez/client/TestTezClient.java: > tezClient.init(new TezConfiguration(false), new YarnConfiguration()); > tez-api/src/main/java/org/apache/tez/client/TezClient.java: > amConfig.setYarnConfiguration(new > YarnConfiguration(amConfig.getTezConfiguration())); > tez-api/src/main/java/org/apache/tez/client/TezClient.java: > amConfig.setYarnConfiguration(new > YarnConfiguration(amConfig.getTezConfiguration())); > tez-api/src/main/java/org/apache/tez/client/TezClient.java: return > getDAGClient(appId, tezConf, new YarnConfiguration(tezConf), frameworkClient, > ugi); > tez-tests/src/test/java/org/apache/tez/test/FaultToleranceTestRunner.java: > tezConf = new TezConfiguration(new YarnConfiguration()); > tez-tests/src/test/java/org/apache/tez/test/FaultToleranceTestRunner.java: > tezConf = new TezConfiguration(new YarnConfiguration(this.conf)); > tez-mapreduce/src/test/java/org/apache/tez/mapreduce/hadoop/TestMRInputHelpers.java: > Configuration testConf = new YarnConfiguration( > tez-mapreduce/src/main/java/org/apache/tez/mapreduce/client/YARNRunner.java: > this(conf, new ResourceMgrDelegate(new YarnConfiguration(conf))); > tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java: > Configuration conf = new Configuration(new YarnConfiguration()); > tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java: > Configuration conf = new Configuration(new YarnConfiguration()); > tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java: > Configuration tezConf = new Configuration(new YarnConfiguration()); > tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java: > Configuration tezConf = new Configuration(new YarnConfiguration()); > tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java: > Configuration tezConf = new Configuration(new YarnConfiguration()); > tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java: > Configuration tezConf = new Configuration(new YarnConfiguration()); > tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java: > Configuration tezConf = new Configuration(new YarnConfiguration()); > tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java: > Configuration tezConf = new Configuration(new YarnConfiguration()); > tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java: > Configuration tezConf = new Configuration(new YarnConfiguration()); > tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java: > Configuration tezConf = new Configuration(new YarnConfiguration()); > tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java: > Configuration tezConf = new Configuration(new YarnConfiguration()); > tez-dag/src/main/java/org/apache/tez/dag/app/DAGAppMaster.java: > Configuration conf = new Configuration(new YarnConfiguration()); > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005)