[
https://issues.apache.org/jira/browse/TEZ-4175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17164098#comment-17164098
]
Rajesh Balamohan commented on TEZ-4175:
---------------------------------------
[~abstractdog] , thanks for sharing the patch. It would be fine to fix it in
DAGAppMaster.
But for changes like TezClient, it may need to be revisited. E.g TezYarnClient
would need yarn conf for properly initing yarnClient.
Otherwise, it would be good to double confirm whether the TezConf passed to it
already has the relevant yarn details.
https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java#L311
https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java#L354
> Consider removing YarnConfiguration where it's possible
> -------------------------------------------------------
>
> Key: TEZ-4175
> URL: https://issues.apache.org/jira/browse/TEZ-4175
> Project: Apache Tez
> Issue Type: Improvement
> Reporter: László Bodor
> Assignee: László Bodor
> Priority: Major
> Attachments: TEZ-4175.01.patch, TEZ-4175.02.patch, TEZ-4175.03.patch,
> TEZ-4175.03.patch
>
>
> A comment in DAGAppmaster made me think that we don't need to rely on
> [YarnConfiguration|https://github.com/apache/hadoop/blob/branch-3.1.3/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java]
> in all cases, what if it can be replace with base Configuration...
> {code}
> // TODO Does this really need to be a YarnConfiguration ?
> Configuration conf = new Configuration(new YarnConfiguration());
> {code}
> In hadoop 3.1.3 source, I cannot see that it adds e.g. yarn-site as a
> resource:
> {code}
> public YarnConfiguration() {
> super();
> }
>
> public YarnConfiguration(Configuration conf) {
> super(conf);
> if (! (conf instanceof YarnConfiguration)) {
> this.reloadConfiguration();
> }
> }
> {code}
> in current codebase:
> {code}
> grep -iRH "new YarnConfiguration" --include="*.java"
> tez-plugins/tez-history-parser/src/main/java/org/apache/tez/history/ATSImportTool.java:
> YarnConfiguration yarnConf = new YarnConfiguration(conf);
> tez-plugins/tez-aux-services/src/main/java/org/apache/tez/auxservices/ShuffleHandler.java:
> super.serviceInit(new YarnConfiguration(conf));
> tez-api/src/test/java/org/apache/tez/dag/api/client/rpc/TestDAGClient.java:
> YarnConfiguration yarnConf = new YarnConfiguration(tezConf);
> tez-api/src/test/java/org/apache/tez/dag/api/client/rpc/TestDAGClient.java:
> YarnConfiguration yarnConf = new YarnConfiguration(tezConf);
> tez-api/src/test/java/org/apache/tez/dag/api/client/rpc/TestDAGClient.java:
> YarnConfiguration yarnConf = new YarnConfiguration(tezConf);
> tez-api/src/test/java/org/apache/tez/client/TestTezClient.java:
> tezClient.init(new TezConfiguration(false), new YarnConfiguration());
> tez-api/src/main/java/org/apache/tez/client/TezClient.java:
> amConfig.setYarnConfiguration(new
> YarnConfiguration(amConfig.getTezConfiguration()));
> tez-api/src/main/java/org/apache/tez/client/TezClient.java:
> amConfig.setYarnConfiguration(new
> YarnConfiguration(amConfig.getTezConfiguration()));
> tez-api/src/main/java/org/apache/tez/client/TezClient.java: return
> getDAGClient(appId, tezConf, new YarnConfiguration(tezConf), frameworkClient,
> ugi);
> tez-tests/src/test/java/org/apache/tez/test/FaultToleranceTestRunner.java:
> tezConf = new TezConfiguration(new YarnConfiguration());
> tez-tests/src/test/java/org/apache/tez/test/FaultToleranceTestRunner.java:
> tezConf = new TezConfiguration(new YarnConfiguration(this.conf));
> tez-mapreduce/src/test/java/org/apache/tez/mapreduce/hadoop/TestMRInputHelpers.java:
> Configuration testConf = new YarnConfiguration(
> tez-mapreduce/src/main/java/org/apache/tez/mapreduce/client/YARNRunner.java:
> this(conf, new ResourceMgrDelegate(new YarnConfiguration(conf)));
> tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java:
> Configuration conf = new Configuration(new YarnConfiguration());
> tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java:
> Configuration conf = new Configuration(new YarnConfiguration());
> tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java:
> Configuration tezConf = new Configuration(new YarnConfiguration());
> tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java:
> Configuration tezConf = new Configuration(new YarnConfiguration());
> tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java:
> Configuration tezConf = new Configuration(new YarnConfiguration());
> tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java:
> Configuration tezConf = new Configuration(new YarnConfiguration());
> tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java:
> Configuration tezConf = new Configuration(new YarnConfiguration());
> tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java:
> Configuration tezConf = new Configuration(new YarnConfiguration());
> tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java:
> Configuration tezConf = new Configuration(new YarnConfiguration());
> tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java:
> Configuration tezConf = new Configuration(new YarnConfiguration());
> tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java:
> Configuration tezConf = new Configuration(new YarnConfiguration());
> tez-dag/src/main/java/org/apache/tez/dag/app/DAGAppMaster.java:
> Configuration conf = new Configuration(new YarnConfiguration());
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
