[jira] [Commented] (TEZ-4175) Consider removing YarnConfiguration where it's possible
[ https://issues.apache.org/jira/browse/TEZ-4175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17185403#comment-17185403 ] László Bodor commented on TEZ-4175: --- pushed to master, thanks [~rajesh.balamohan], [~mustafaiman], [~ashutoshc] for the review! > Consider removing YarnConfiguration where it's possible > --- > > Key: TEZ-4175 > URL: https://issues.apache.org/jira/browse/TEZ-4175 > Project: Apache Tez > Issue Type: Improvement >Reporter: László Bodor >Assignee: László Bodor >Priority: Major > Fix For: 0.10.0 > > Attachments: TEZ-4175.01.patch, TEZ-4175.02.patch, TEZ-4175.03.patch, > TEZ-4175.03.patch, TEZ-4175.04.patch, TEZ-4175.05.patch > > > A comment in DAGAppmaster made me think that we don't need to rely on > [YarnConfiguration|https://github.com/apache/hadoop/blob/branch-3.1.3/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java] > in all cases, what if it can be replace with base Configuration... > {code} > // TODO Does this really need to be a YarnConfiguration ? > Configuration conf = new Configuration(new YarnConfiguration()); > {code} > In hadoop 3.1.3 source, I cannot see that it adds e.g. yarn-site as a > resource: > {code} > public YarnConfiguration() { > super(); > } > > public YarnConfiguration(Configuration conf) { > super(conf); > if (! (conf instanceof YarnConfiguration)) { > this.reloadConfiguration(); > } > } > {code} > UPDATE: YarnConfiguration adds yarn-site as a default resource [in a static > initializer|https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java#L100-L102], > so it should be used when yarn configs are expected to be present > in current codebase: > {code} > grep -iRH "new YarnConfiguration" --include="*.java" > tez-plugins/tez-history-parser/src/main/java/org/apache/tez/history/ATSImportTool.java: > YarnConfiguration yarnConf = new YarnConfiguration(conf); > tez-plugins/tez-aux-services/src/main/java/org/apache/tez/auxservices/ShuffleHandler.java: > super.serviceInit(new YarnConfiguration(conf)); > tez-api/src/test/java/org/apache/tez/dag/api/client/rpc/TestDAGClient.java: > YarnConfiguration yarnConf = new YarnConfiguration(tezConf); > tez-api/src/test/java/org/apache/tez/dag/api/client/rpc/TestDAGClient.java: > YarnConfiguration yarnConf = new YarnConfiguration(tezConf); > tez-api/src/test/java/org/apache/tez/dag/api/client/rpc/TestDAGClient.java: > YarnConfiguration yarnConf = new YarnConfiguration(tezConf); > tez-api/src/test/java/org/apache/tez/client/TestTezClient.java: > tezClient.init(new TezConfiguration(false), new YarnConfiguration()); > tez-api/src/main/java/org/apache/tez/client/TezClient.java: > amConfig.setYarnConfiguration(new > YarnConfiguration(amConfig.getTezConfiguration())); > tez-api/src/main/java/org/apache/tez/client/TezClient.java: > amConfig.setYarnConfiguration(new > YarnConfiguration(amConfig.getTezConfiguration())); > tez-api/src/main/java/org/apache/tez/client/TezClient.java:return > getDAGClient(appId, tezConf, new YarnConfiguration(tezConf), frameworkClient, > ugi); > tez-tests/src/test/java/org/apache/tez/test/FaultToleranceTestRunner.java: > tezConf = new TezConfiguration(new YarnConfiguration()); > tez-tests/src/test/java/org/apache/tez/test/FaultToleranceTestRunner.java: >tezConf = new TezConfiguration(new YarnConfiguration(this.conf)); > tez-mapreduce/src/test/java/org/apache/tez/mapreduce/hadoop/TestMRInputHelpers.java: > Configuration testConf = new YarnConfiguration( > tez-mapreduce/src/main/java/org/apache/tez/mapreduce/client/YARNRunner.java: > this(conf, new ResourceMgrDelegate(new YarnConfiguration(conf))); > tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java: > Configuration conf = new Configuration(new YarnConfiguration()); > tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java: > Configuration conf = new Configuration(new YarnConfiguration()); > tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java: > Configuration tezConf = new Configuration(new YarnConfiguration()); > tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java: > Configuration tezConf = new Configuration(new YarnConfiguration()); > tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java: > Configuration tezConf = new Configuration(new YarnConfiguration()); > tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java: > Configuration tezConf = new Configuration(new YarnConfiguration()); >
[jira] [Commented] (TEZ-4175) Consider removing YarnConfiguration where it's possible
[ https://issues.apache.org/jira/browse/TEZ-4175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17185239#comment-17185239 ] Ashutosh Chauhan commented on TEZ-4175: --- +1 > Consider removing YarnConfiguration where it's possible > --- > > Key: TEZ-4175 > URL: https://issues.apache.org/jira/browse/TEZ-4175 > Project: Apache Tez > Issue Type: Improvement >Reporter: László Bodor >Assignee: László Bodor >Priority: Major > Attachments: TEZ-4175.01.patch, TEZ-4175.02.patch, TEZ-4175.03.patch, > TEZ-4175.03.patch, TEZ-4175.04.patch, TEZ-4175.05.patch > > > A comment in DAGAppmaster made me think that we don't need to rely on > [YarnConfiguration|https://github.com/apache/hadoop/blob/branch-3.1.3/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java] > in all cases, what if it can be replace with base Configuration... > {code} > // TODO Does this really need to be a YarnConfiguration ? > Configuration conf = new Configuration(new YarnConfiguration()); > {code} > In hadoop 3.1.3 source, I cannot see that it adds e.g. yarn-site as a > resource: > {code} > public YarnConfiguration() { > super(); > } > > public YarnConfiguration(Configuration conf) { > super(conf); > if (! (conf instanceof YarnConfiguration)) { > this.reloadConfiguration(); > } > } > {code} > UPDATE: YarnConfiguration adds yarn-site as a default resource [in a static > initializer|https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java#L100-L102], > so it should be used when yarn configs are expected to be present > in current codebase: > {code} > grep -iRH "new YarnConfiguration" --include="*.java" > tez-plugins/tez-history-parser/src/main/java/org/apache/tez/history/ATSImportTool.java: > YarnConfiguration yarnConf = new YarnConfiguration(conf); > tez-plugins/tez-aux-services/src/main/java/org/apache/tez/auxservices/ShuffleHandler.java: > super.serviceInit(new YarnConfiguration(conf)); > tez-api/src/test/java/org/apache/tez/dag/api/client/rpc/TestDAGClient.java: > YarnConfiguration yarnConf = new YarnConfiguration(tezConf); > tez-api/src/test/java/org/apache/tez/dag/api/client/rpc/TestDAGClient.java: > YarnConfiguration yarnConf = new YarnConfiguration(tezConf); > tez-api/src/test/java/org/apache/tez/dag/api/client/rpc/TestDAGClient.java: > YarnConfiguration yarnConf = new YarnConfiguration(tezConf); > tez-api/src/test/java/org/apache/tez/client/TestTezClient.java: > tezClient.init(new TezConfiguration(false), new YarnConfiguration()); > tez-api/src/main/java/org/apache/tez/client/TezClient.java: > amConfig.setYarnConfiguration(new > YarnConfiguration(amConfig.getTezConfiguration())); > tez-api/src/main/java/org/apache/tez/client/TezClient.java: > amConfig.setYarnConfiguration(new > YarnConfiguration(amConfig.getTezConfiguration())); > tez-api/src/main/java/org/apache/tez/client/TezClient.java:return > getDAGClient(appId, tezConf, new YarnConfiguration(tezConf), frameworkClient, > ugi); > tez-tests/src/test/java/org/apache/tez/test/FaultToleranceTestRunner.java: > tezConf = new TezConfiguration(new YarnConfiguration()); > tez-tests/src/test/java/org/apache/tez/test/FaultToleranceTestRunner.java: >tezConf = new TezConfiguration(new YarnConfiguration(this.conf)); > tez-mapreduce/src/test/java/org/apache/tez/mapreduce/hadoop/TestMRInputHelpers.java: > Configuration testConf = new YarnConfiguration( > tez-mapreduce/src/main/java/org/apache/tez/mapreduce/client/YARNRunner.java: > this(conf, new ResourceMgrDelegate(new YarnConfiguration(conf))); > tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java: > Configuration conf = new Configuration(new YarnConfiguration()); > tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java: > Configuration conf = new Configuration(new YarnConfiguration()); > tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java: > Configuration tezConf = new Configuration(new YarnConfiguration()); > tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java: > Configuration tezConf = new Configuration(new YarnConfiguration()); > tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java: > Configuration tezConf = new Configuration(new YarnConfiguration()); > tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java: > Configuration tezConf = new Configuration(new YarnConfiguration()); > tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java: > Configuration tezConf = new Configuration(new YarnConfiguration()); >
[jira] [Commented] (TEZ-4175) Consider removing YarnConfiguration where it's possible
[ https://issues.apache.org/jira/browse/TEZ-4175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17177362#comment-17177362 ] Mustafa Iman commented on TEZ-4175: --- Thanks [~abstractdog] TEZ-4175.05.patch looks good to me > Consider removing YarnConfiguration where it's possible > --- > > Key: TEZ-4175 > URL: https://issues.apache.org/jira/browse/TEZ-4175 > Project: Apache Tez > Issue Type: Improvement >Reporter: László Bodor >Assignee: László Bodor >Priority: Major > Attachments: TEZ-4175.01.patch, TEZ-4175.02.patch, TEZ-4175.03.patch, > TEZ-4175.03.patch, TEZ-4175.04.patch, TEZ-4175.05.patch > > > A comment in DAGAppmaster made me think that we don't need to rely on > [YarnConfiguration|https://github.com/apache/hadoop/blob/branch-3.1.3/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java] > in all cases, what if it can be replace with base Configuration... > {code} > // TODO Does this really need to be a YarnConfiguration ? > Configuration conf = new Configuration(new YarnConfiguration()); > {code} > In hadoop 3.1.3 source, I cannot see that it adds e.g. yarn-site as a > resource: > {code} > public YarnConfiguration() { > super(); > } > > public YarnConfiguration(Configuration conf) { > super(conf); > if (! (conf instanceof YarnConfiguration)) { > this.reloadConfiguration(); > } > } > {code} > in current codebase: > {code} > grep -iRH "new YarnConfiguration" --include="*.java" > tez-plugins/tez-history-parser/src/main/java/org/apache/tez/history/ATSImportTool.java: > YarnConfiguration yarnConf = new YarnConfiguration(conf); > tez-plugins/tez-aux-services/src/main/java/org/apache/tez/auxservices/ShuffleHandler.java: > super.serviceInit(new YarnConfiguration(conf)); > tez-api/src/test/java/org/apache/tez/dag/api/client/rpc/TestDAGClient.java: > YarnConfiguration yarnConf = new YarnConfiguration(tezConf); > tez-api/src/test/java/org/apache/tez/dag/api/client/rpc/TestDAGClient.java: > YarnConfiguration yarnConf = new YarnConfiguration(tezConf); > tez-api/src/test/java/org/apache/tez/dag/api/client/rpc/TestDAGClient.java: > YarnConfiguration yarnConf = new YarnConfiguration(tezConf); > tez-api/src/test/java/org/apache/tez/client/TestTezClient.java: > tezClient.init(new TezConfiguration(false), new YarnConfiguration()); > tez-api/src/main/java/org/apache/tez/client/TezClient.java: > amConfig.setYarnConfiguration(new > YarnConfiguration(amConfig.getTezConfiguration())); > tez-api/src/main/java/org/apache/tez/client/TezClient.java: > amConfig.setYarnConfiguration(new > YarnConfiguration(amConfig.getTezConfiguration())); > tez-api/src/main/java/org/apache/tez/client/TezClient.java:return > getDAGClient(appId, tezConf, new YarnConfiguration(tezConf), frameworkClient, > ugi); > tez-tests/src/test/java/org/apache/tez/test/FaultToleranceTestRunner.java: > tezConf = new TezConfiguration(new YarnConfiguration()); > tez-tests/src/test/java/org/apache/tez/test/FaultToleranceTestRunner.java: >tezConf = new TezConfiguration(new YarnConfiguration(this.conf)); > tez-mapreduce/src/test/java/org/apache/tez/mapreduce/hadoop/TestMRInputHelpers.java: > Configuration testConf = new YarnConfiguration( > tez-mapreduce/src/main/java/org/apache/tez/mapreduce/client/YARNRunner.java: > this(conf, new ResourceMgrDelegate(new YarnConfiguration(conf))); > tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java: > Configuration conf = new Configuration(new YarnConfiguration()); > tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java: > Configuration conf = new Configuration(new YarnConfiguration()); > tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java: > Configuration tezConf = new Configuration(new YarnConfiguration()); > tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java: > Configuration tezConf = new Configuration(new YarnConfiguration()); > tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java: > Configuration tezConf = new Configuration(new YarnConfiguration()); > tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java: > Configuration tezConf = new Configuration(new YarnConfiguration()); > tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java: > Configuration tezConf = new Configuration(new YarnConfiguration()); > tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java: > Configuration tezConf = new Configuration(new YarnConfiguration()); > tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java: > Configuration tezConf = new Configuration(new
[jira] [Commented] (TEZ-4175) Consider removing YarnConfiguration where it's possible
[ https://issues.apache.org/jira/browse/TEZ-4175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17176986#comment-17176986 ] TezQA commented on TEZ-4175: | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 28s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m 0s{color} | {color:green} No case conflicting files found. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 5 new or modified test files. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 26s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m 17s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 37s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 31s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 42s{color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 0m 37s{color} | {color:blue} Used deprecated FindBugs config; considering switching to SpotBugs. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 35s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 8s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 13s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 3s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 3s{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 0m 15s{color} | {color:orange} tez-api: The patch generated 1 new + 126 unchanged - 3 fixed = 127 total (was 129) {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 0m 11s{color} | {color:orange} tez-mapreduce: The patch generated 1 new + 16 unchanged - 1 fixed = 17 total (was 17) {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 19s{color} | {color:green} tez-dag: The patch generated 0 new + 167 unchanged - 2 fixed = 167 total (was 169) {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 12s{color} | {color:green} tez-tests: The patch generated 0 new + 8 unchanged - 2 fixed = 8 total (was 10) {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 18s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m 5s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 50s{color} | {color:green} tez-api in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 4s{color} | {color:green} tez-mapreduce in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m 6s{color} | {color:green} tez-dag in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 32m 40s{color} | {color:green} tez-tests in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 33s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 64m 10s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://builds.apache.org/job/PreCommit-TEZ-Build/522/artifact/out/Dockerfile | | JIRA Issue | TEZ-4175 | | JIRA Patch URL |
[jira] [Commented] (TEZ-4175) Consider removing YarnConfiguration where it's possible
[ https://issues.apache.org/jira/browse/TEZ-4175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17176859#comment-17176859 ] László Bodor commented on TEZ-4175: --- thanks [~mustafaiman] for the review! for the same reason, I reverted back to YarnConfiguration in ResourceMgrDelegate and ShuffleHandler as well > Consider removing YarnConfiguration where it's possible > --- > > Key: TEZ-4175 > URL: https://issues.apache.org/jira/browse/TEZ-4175 > Project: Apache Tez > Issue Type: Improvement >Reporter: László Bodor >Assignee: László Bodor >Priority: Major > Attachments: TEZ-4175.01.patch, TEZ-4175.02.patch, TEZ-4175.03.patch, > TEZ-4175.03.patch, TEZ-4175.04.patch, TEZ-4175.05.patch > > > A comment in DAGAppmaster made me think that we don't need to rely on > [YarnConfiguration|https://github.com/apache/hadoop/blob/branch-3.1.3/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java] > in all cases, what if it can be replace with base Configuration... > {code} > // TODO Does this really need to be a YarnConfiguration ? > Configuration conf = new Configuration(new YarnConfiguration()); > {code} > In hadoop 3.1.3 source, I cannot see that it adds e.g. yarn-site as a > resource: > {code} > public YarnConfiguration() { > super(); > } > > public YarnConfiguration(Configuration conf) { > super(conf); > if (! (conf instanceof YarnConfiguration)) { > this.reloadConfiguration(); > } > } > {code} > in current codebase: > {code} > grep -iRH "new YarnConfiguration" --include="*.java" > tez-plugins/tez-history-parser/src/main/java/org/apache/tez/history/ATSImportTool.java: > YarnConfiguration yarnConf = new YarnConfiguration(conf); > tez-plugins/tez-aux-services/src/main/java/org/apache/tez/auxservices/ShuffleHandler.java: > super.serviceInit(new YarnConfiguration(conf)); > tez-api/src/test/java/org/apache/tez/dag/api/client/rpc/TestDAGClient.java: > YarnConfiguration yarnConf = new YarnConfiguration(tezConf); > tez-api/src/test/java/org/apache/tez/dag/api/client/rpc/TestDAGClient.java: > YarnConfiguration yarnConf = new YarnConfiguration(tezConf); > tez-api/src/test/java/org/apache/tez/dag/api/client/rpc/TestDAGClient.java: > YarnConfiguration yarnConf = new YarnConfiguration(tezConf); > tez-api/src/test/java/org/apache/tez/client/TestTezClient.java: > tezClient.init(new TezConfiguration(false), new YarnConfiguration()); > tez-api/src/main/java/org/apache/tez/client/TezClient.java: > amConfig.setYarnConfiguration(new > YarnConfiguration(amConfig.getTezConfiguration())); > tez-api/src/main/java/org/apache/tez/client/TezClient.java: > amConfig.setYarnConfiguration(new > YarnConfiguration(amConfig.getTezConfiguration())); > tez-api/src/main/java/org/apache/tez/client/TezClient.java:return > getDAGClient(appId, tezConf, new YarnConfiguration(tezConf), frameworkClient, > ugi); > tez-tests/src/test/java/org/apache/tez/test/FaultToleranceTestRunner.java: > tezConf = new TezConfiguration(new YarnConfiguration()); > tez-tests/src/test/java/org/apache/tez/test/FaultToleranceTestRunner.java: >tezConf = new TezConfiguration(new YarnConfiguration(this.conf)); > tez-mapreduce/src/test/java/org/apache/tez/mapreduce/hadoop/TestMRInputHelpers.java: > Configuration testConf = new YarnConfiguration( > tez-mapreduce/src/main/java/org/apache/tez/mapreduce/client/YARNRunner.java: > this(conf, new ResourceMgrDelegate(new YarnConfiguration(conf))); > tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java: > Configuration conf = new Configuration(new YarnConfiguration()); > tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java: > Configuration conf = new Configuration(new YarnConfiguration()); > tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java: > Configuration tezConf = new Configuration(new YarnConfiguration()); > tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java: > Configuration tezConf = new Configuration(new YarnConfiguration()); > tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java: > Configuration tezConf = new Configuration(new YarnConfiguration()); > tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java: > Configuration tezConf = new Configuration(new YarnConfiguration()); > tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java: > Configuration tezConf = new Configuration(new YarnConfiguration()); > tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java: > Configuration tezConf = new Configuration(new YarnConfiguration()); >
[jira] [Commented] (TEZ-4175) Consider removing YarnConfiguration where it's possible
[ https://issues.apache.org/jira/browse/TEZ-4175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17172617#comment-17172617 ] Mustafa Iman commented on TEZ-4175: --- [~abstractdog] I'd love to get rid of YarnConfiguration wherever it is not necessary. Thanks for the patch. For TezYarnClient I think wrapping the config with YarnClient inside its own init method like in this patch should be fine. One concern: Looks like ATSImportTool is querying a Yarn config that might come from yarn-*.xml file. So we can leave ATSImportTool#hasHttpsPolicy untouched. It is an isolated tool. Otherwise it looks good to me. > Consider removing YarnConfiguration where it's possible > --- > > Key: TEZ-4175 > URL: https://issues.apache.org/jira/browse/TEZ-4175 > Project: Apache Tez > Issue Type: Improvement >Reporter: László Bodor >Assignee: László Bodor >Priority: Major > Attachments: TEZ-4175.01.patch, TEZ-4175.02.patch, TEZ-4175.03.patch, > TEZ-4175.03.patch, TEZ-4175.04.patch > > > A comment in DAGAppmaster made me think that we don't need to rely on > [YarnConfiguration|https://github.com/apache/hadoop/blob/branch-3.1.3/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java] > in all cases, what if it can be replace with base Configuration... > {code} > // TODO Does this really need to be a YarnConfiguration ? > Configuration conf = new Configuration(new YarnConfiguration()); > {code} > In hadoop 3.1.3 source, I cannot see that it adds e.g. yarn-site as a > resource: > {code} > public YarnConfiguration() { > super(); > } > > public YarnConfiguration(Configuration conf) { > super(conf); > if (! (conf instanceof YarnConfiguration)) { > this.reloadConfiguration(); > } > } > {code} > in current codebase: > {code} > grep -iRH "new YarnConfiguration" --include="*.java" > tez-plugins/tez-history-parser/src/main/java/org/apache/tez/history/ATSImportTool.java: > YarnConfiguration yarnConf = new YarnConfiguration(conf); > tez-plugins/tez-aux-services/src/main/java/org/apache/tez/auxservices/ShuffleHandler.java: > super.serviceInit(new YarnConfiguration(conf)); > tez-api/src/test/java/org/apache/tez/dag/api/client/rpc/TestDAGClient.java: > YarnConfiguration yarnConf = new YarnConfiguration(tezConf); > tez-api/src/test/java/org/apache/tez/dag/api/client/rpc/TestDAGClient.java: > YarnConfiguration yarnConf = new YarnConfiguration(tezConf); > tez-api/src/test/java/org/apache/tez/dag/api/client/rpc/TestDAGClient.java: > YarnConfiguration yarnConf = new YarnConfiguration(tezConf); > tez-api/src/test/java/org/apache/tez/client/TestTezClient.java: > tezClient.init(new TezConfiguration(false), new YarnConfiguration()); > tez-api/src/main/java/org/apache/tez/client/TezClient.java: > amConfig.setYarnConfiguration(new > YarnConfiguration(amConfig.getTezConfiguration())); > tez-api/src/main/java/org/apache/tez/client/TezClient.java: > amConfig.setYarnConfiguration(new > YarnConfiguration(amConfig.getTezConfiguration())); > tez-api/src/main/java/org/apache/tez/client/TezClient.java:return > getDAGClient(appId, tezConf, new YarnConfiguration(tezConf), frameworkClient, > ugi); > tez-tests/src/test/java/org/apache/tez/test/FaultToleranceTestRunner.java: > tezConf = new TezConfiguration(new YarnConfiguration()); > tez-tests/src/test/java/org/apache/tez/test/FaultToleranceTestRunner.java: >tezConf = new TezConfiguration(new YarnConfiguration(this.conf)); > tez-mapreduce/src/test/java/org/apache/tez/mapreduce/hadoop/TestMRInputHelpers.java: > Configuration testConf = new YarnConfiguration( > tez-mapreduce/src/main/java/org/apache/tez/mapreduce/client/YARNRunner.java: > this(conf, new ResourceMgrDelegate(new YarnConfiguration(conf))); > tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java: > Configuration conf = new Configuration(new YarnConfiguration()); > tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java: > Configuration conf = new Configuration(new YarnConfiguration()); > tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java: > Configuration tezConf = new Configuration(new YarnConfiguration()); > tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java: > Configuration tezConf = new Configuration(new YarnConfiguration()); > tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java: > Configuration tezConf = new Configuration(new YarnConfiguration()); > tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java: > Configuration tezConf = new Configuration(new YarnConfiguration()); >
[jira] [Commented] (TEZ-4175) Consider removing YarnConfiguration where it's possible
[ https://issues.apache.org/jira/browse/TEZ-4175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17167724#comment-17167724 ] László Bodor commented on TEZ-4175: --- [~rajesh.balamohan]: could you please take a look at [^TEZ-4175.04.patch]? I've validated this patch on a cluster (with hive on tez container mode), worked fine there were 2 things kept in mind: 1. use YarnConfiguration in TezYarnClient: double-checked and I've found that YarnConfiguration [adds default resources|https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java#L100-L102], and by using TezConfiguration everywhere, we can miss yarn config (as if YarnConfiguration class is not referenced, we won't add yarn-*.xml as default resource in its static initializer) 2. Use Configuration instead of YarnConfiguration in DAGAppMaster...YarnConfiguration was added in TEZ-1879 with the [comment|https://github.com/apache/tez/commit/ea46f459c0e88b0f9b0c714f8c6ac9d9a6f03c5e#diff-755c0ec043a1800cd6cbf31823a59c8fR1845], I confirmed on the cluster that using a Configuration didn't cause any issues > Consider removing YarnConfiguration where it's possible > --- > > Key: TEZ-4175 > URL: https://issues.apache.org/jira/browse/TEZ-4175 > Project: Apache Tez > Issue Type: Improvement >Reporter: László Bodor >Assignee: László Bodor >Priority: Major > Attachments: TEZ-4175.01.patch, TEZ-4175.02.patch, TEZ-4175.03.patch, > TEZ-4175.03.patch, TEZ-4175.04.patch > > > A comment in DAGAppmaster made me think that we don't need to rely on > [YarnConfiguration|https://github.com/apache/hadoop/blob/branch-3.1.3/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java] > in all cases, what if it can be replace with base Configuration... > {code} > // TODO Does this really need to be a YarnConfiguration ? > Configuration conf = new Configuration(new YarnConfiguration()); > {code} > In hadoop 3.1.3 source, I cannot see that it adds e.g. yarn-site as a > resource: > {code} > public YarnConfiguration() { > super(); > } > > public YarnConfiguration(Configuration conf) { > super(conf); > if (! (conf instanceof YarnConfiguration)) { > this.reloadConfiguration(); > } > } > {code} > in current codebase: > {code} > grep -iRH "new YarnConfiguration" --include="*.java" > tez-plugins/tez-history-parser/src/main/java/org/apache/tez/history/ATSImportTool.java: > YarnConfiguration yarnConf = new YarnConfiguration(conf); > tez-plugins/tez-aux-services/src/main/java/org/apache/tez/auxservices/ShuffleHandler.java: > super.serviceInit(new YarnConfiguration(conf)); > tez-api/src/test/java/org/apache/tez/dag/api/client/rpc/TestDAGClient.java: > YarnConfiguration yarnConf = new YarnConfiguration(tezConf); > tez-api/src/test/java/org/apache/tez/dag/api/client/rpc/TestDAGClient.java: > YarnConfiguration yarnConf = new YarnConfiguration(tezConf); > tez-api/src/test/java/org/apache/tez/dag/api/client/rpc/TestDAGClient.java: > YarnConfiguration yarnConf = new YarnConfiguration(tezConf); > tez-api/src/test/java/org/apache/tez/client/TestTezClient.java: > tezClient.init(new TezConfiguration(false), new YarnConfiguration()); > tez-api/src/main/java/org/apache/tez/client/TezClient.java: > amConfig.setYarnConfiguration(new > YarnConfiguration(amConfig.getTezConfiguration())); > tez-api/src/main/java/org/apache/tez/client/TezClient.java: > amConfig.setYarnConfiguration(new > YarnConfiguration(amConfig.getTezConfiguration())); > tez-api/src/main/java/org/apache/tez/client/TezClient.java:return > getDAGClient(appId, tezConf, new YarnConfiguration(tezConf), frameworkClient, > ugi); > tez-tests/src/test/java/org/apache/tez/test/FaultToleranceTestRunner.java: > tezConf = new TezConfiguration(new YarnConfiguration()); > tez-tests/src/test/java/org/apache/tez/test/FaultToleranceTestRunner.java: >tezConf = new TezConfiguration(new YarnConfiguration(this.conf)); > tez-mapreduce/src/test/java/org/apache/tez/mapreduce/hadoop/TestMRInputHelpers.java: > Configuration testConf = new YarnConfiguration( > tez-mapreduce/src/main/java/org/apache/tez/mapreduce/client/YARNRunner.java: > this(conf, new ResourceMgrDelegate(new YarnConfiguration(conf))); > tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java: > Configuration conf = new Configuration(new YarnConfiguration()); > tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java: > Configuration conf = new Configuration(new YarnConfiguration()); > tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java: > Configuration tezConf =
[jira] [Commented] (TEZ-4175) Consider removing YarnConfiguration where it's possible
[ https://issues.apache.org/jira/browse/TEZ-4175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17167318#comment-17167318 ] TezQA commented on TEZ-4175: | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 12m 26s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m 0s{color} | {color:green} No case conflicting files found. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 5 new or modified test files. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 41s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m 22s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 22s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m 14s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 34s{color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 0m 35s{color} | {color:blue} Used deprecated FindBugs config; considering switching to SpotBugs. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m 52s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m 15s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 36s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 36s{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 0m 15s{color} | {color:orange} tez-api: The patch generated 1 new + 125 unchanged - 3 fixed = 126 total (was 128) {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 12s{color} | {color:green} tez-mapreduce: The patch generated 0 new + 18 unchanged - 1 fixed = 18 total (was 19) {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 22s{color} | {color:green} tez-dag: The patch generated 0 new + 169 unchanged - 2 fixed = 169 total (was 171) {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 11s{color} | {color:green} tez-tests: The patch generated 0 new + 8 unchanged - 2 fixed = 8 total (was 10) {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 10s{color} | {color:green} The patch passed checkstyle in tez-history-parser {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 10s{color} | {color:green} The patch passed checkstyle in tez-aux-services {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 40s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m 6s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 4s{color} | {color:green} tez-api in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 9s{color} | {color:green} tez-mapreduce in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m 16s{color} | {color:green} tez-dag in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 35m 17s{color} | {color:green} tez-tests in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 11s{color} | {color:green} tez-history-parser in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit
[jira] [Commented] (TEZ-4175) Consider removing YarnConfiguration where it's possible
[ https://issues.apache.org/jira/browse/TEZ-4175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17164258#comment-17164258 ] László Bodor commented on TEZ-4175: --- thanks [~rajesh.balamohan]! I've already checked and I can see the yarn conf is just the leftover of old code, I mean, it just repacks tezConf everywhere it [comes from AMConfiguration to client|https://github.com/apache/tez/blob/master/tez-api/src/main/java/org/apache/tez/client/TezClient.java#L469], and checked all references of AMConfiguration.setYarnConfiguration: https://github.com/apache/tez/blob/master/tez-api/src/main/java/org/apache/tez/client/TezClient.java#L387 https://github.com/apache/tez/blob/master/tez-api/src/main/java/org/apache/tez/client/TezClient.java#L437 the only reason I can see to do "new YarnConfiguration(tezConf)" is e.g. recovering a yarn bug where yarn would touch tez config in a destructive way, but currently, I'm not aware of such issue...but even if that's the case, we can do that inside TezYarnClient.init, without polluting the api, IMHO > Consider removing YarnConfiguration where it's possible > --- > > Key: TEZ-4175 > URL: https://issues.apache.org/jira/browse/TEZ-4175 > Project: Apache Tez > Issue Type: Improvement >Reporter: László Bodor >Assignee: László Bodor >Priority: Major > Attachments: TEZ-4175.01.patch, TEZ-4175.02.patch, TEZ-4175.03.patch, > TEZ-4175.03.patch > > > A comment in DAGAppmaster made me think that we don't need to rely on > [YarnConfiguration|https://github.com/apache/hadoop/blob/branch-3.1.3/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java] > in all cases, what if it can be replace with base Configuration... > {code} > // TODO Does this really need to be a YarnConfiguration ? > Configuration conf = new Configuration(new YarnConfiguration()); > {code} > In hadoop 3.1.3 source, I cannot see that it adds e.g. yarn-site as a > resource: > {code} > public YarnConfiguration() { > super(); > } > > public YarnConfiguration(Configuration conf) { > super(conf); > if (! (conf instanceof YarnConfiguration)) { > this.reloadConfiguration(); > } > } > {code} > in current codebase: > {code} > grep -iRH "new YarnConfiguration" --include="*.java" > tez-plugins/tez-history-parser/src/main/java/org/apache/tez/history/ATSImportTool.java: > YarnConfiguration yarnConf = new YarnConfiguration(conf); > tez-plugins/tez-aux-services/src/main/java/org/apache/tez/auxservices/ShuffleHandler.java: > super.serviceInit(new YarnConfiguration(conf)); > tez-api/src/test/java/org/apache/tez/dag/api/client/rpc/TestDAGClient.java: > YarnConfiguration yarnConf = new YarnConfiguration(tezConf); > tez-api/src/test/java/org/apache/tez/dag/api/client/rpc/TestDAGClient.java: > YarnConfiguration yarnConf = new YarnConfiguration(tezConf); > tez-api/src/test/java/org/apache/tez/dag/api/client/rpc/TestDAGClient.java: > YarnConfiguration yarnConf = new YarnConfiguration(tezConf); > tez-api/src/test/java/org/apache/tez/client/TestTezClient.java: > tezClient.init(new TezConfiguration(false), new YarnConfiguration()); > tez-api/src/main/java/org/apache/tez/client/TezClient.java: > amConfig.setYarnConfiguration(new > YarnConfiguration(amConfig.getTezConfiguration())); > tez-api/src/main/java/org/apache/tez/client/TezClient.java: > amConfig.setYarnConfiguration(new > YarnConfiguration(amConfig.getTezConfiguration())); > tez-api/src/main/java/org/apache/tez/client/TezClient.java:return > getDAGClient(appId, tezConf, new YarnConfiguration(tezConf), frameworkClient, > ugi); > tez-tests/src/test/java/org/apache/tez/test/FaultToleranceTestRunner.java: > tezConf = new TezConfiguration(new YarnConfiguration()); > tez-tests/src/test/java/org/apache/tez/test/FaultToleranceTestRunner.java: >tezConf = new TezConfiguration(new YarnConfiguration(this.conf)); > tez-mapreduce/src/test/java/org/apache/tez/mapreduce/hadoop/TestMRInputHelpers.java: > Configuration testConf = new YarnConfiguration( > tez-mapreduce/src/main/java/org/apache/tez/mapreduce/client/YARNRunner.java: > this(conf, new ResourceMgrDelegate(new YarnConfiguration(conf))); > tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java: > Configuration conf = new Configuration(new YarnConfiguration()); > tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java: > Configuration conf = new Configuration(new YarnConfiguration()); > tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java: > Configuration tezConf = new Configuration(new YarnConfiguration()); > tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java: > Configuration
[jira] [Commented] (TEZ-4175) Consider removing YarnConfiguration where it's possible
[ https://issues.apache.org/jira/browse/TEZ-4175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17164098#comment-17164098 ] Rajesh Balamohan commented on TEZ-4175: --- [~abstractdog] , thanks for sharing the patch. It would be fine to fix it in DAGAppMaster. But for changes like TezClient, it may need to be revisited. E.g TezYarnClient would need yarn conf for properly initing yarnClient. Otherwise, it would be good to double confirm whether the TezConf passed to it already has the relevant yarn details. https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java#L311 https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java#L354 > Consider removing YarnConfiguration where it's possible > --- > > Key: TEZ-4175 > URL: https://issues.apache.org/jira/browse/TEZ-4175 > Project: Apache Tez > Issue Type: Improvement >Reporter: László Bodor >Assignee: László Bodor >Priority: Major > Attachments: TEZ-4175.01.patch, TEZ-4175.02.patch, TEZ-4175.03.patch, > TEZ-4175.03.patch > > > A comment in DAGAppmaster made me think that we don't need to rely on > [YarnConfiguration|https://github.com/apache/hadoop/blob/branch-3.1.3/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java] > in all cases, what if it can be replace with base Configuration... > {code} > // TODO Does this really need to be a YarnConfiguration ? > Configuration conf = new Configuration(new YarnConfiguration()); > {code} > In hadoop 3.1.3 source, I cannot see that it adds e.g. yarn-site as a > resource: > {code} > public YarnConfiguration() { > super(); > } > > public YarnConfiguration(Configuration conf) { > super(conf); > if (! (conf instanceof YarnConfiguration)) { > this.reloadConfiguration(); > } > } > {code} > in current codebase: > {code} > grep -iRH "new YarnConfiguration" --include="*.java" > tez-plugins/tez-history-parser/src/main/java/org/apache/tez/history/ATSImportTool.java: > YarnConfiguration yarnConf = new YarnConfiguration(conf); > tez-plugins/tez-aux-services/src/main/java/org/apache/tez/auxservices/ShuffleHandler.java: > super.serviceInit(new YarnConfiguration(conf)); > tez-api/src/test/java/org/apache/tez/dag/api/client/rpc/TestDAGClient.java: > YarnConfiguration yarnConf = new YarnConfiguration(tezConf); > tez-api/src/test/java/org/apache/tez/dag/api/client/rpc/TestDAGClient.java: > YarnConfiguration yarnConf = new YarnConfiguration(tezConf); > tez-api/src/test/java/org/apache/tez/dag/api/client/rpc/TestDAGClient.java: > YarnConfiguration yarnConf = new YarnConfiguration(tezConf); > tez-api/src/test/java/org/apache/tez/client/TestTezClient.java: > tezClient.init(new TezConfiguration(false), new YarnConfiguration()); > tez-api/src/main/java/org/apache/tez/client/TezClient.java: > amConfig.setYarnConfiguration(new > YarnConfiguration(amConfig.getTezConfiguration())); > tez-api/src/main/java/org/apache/tez/client/TezClient.java: > amConfig.setYarnConfiguration(new > YarnConfiguration(amConfig.getTezConfiguration())); > tez-api/src/main/java/org/apache/tez/client/TezClient.java:return > getDAGClient(appId, tezConf, new YarnConfiguration(tezConf), frameworkClient, > ugi); > tez-tests/src/test/java/org/apache/tez/test/FaultToleranceTestRunner.java: > tezConf = new TezConfiguration(new YarnConfiguration()); > tez-tests/src/test/java/org/apache/tez/test/FaultToleranceTestRunner.java: >tezConf = new TezConfiguration(new YarnConfiguration(this.conf)); > tez-mapreduce/src/test/java/org/apache/tez/mapreduce/hadoop/TestMRInputHelpers.java: > Configuration testConf = new YarnConfiguration( > tez-mapreduce/src/main/java/org/apache/tez/mapreduce/client/YARNRunner.java: > this(conf, new ResourceMgrDelegate(new YarnConfiguration(conf))); > tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java: > Configuration conf = new Configuration(new YarnConfiguration()); > tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java: > Configuration conf = new Configuration(new YarnConfiguration()); > tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java: > Configuration tezConf = new Configuration(new YarnConfiguration()); > tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java: > Configuration tezConf = new Configuration(new YarnConfiguration()); > tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java: > Configuration tezConf = new Configuration(new YarnConfiguration()); > tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java: > Configuration
[jira] [Commented] (TEZ-4175) Consider removing YarnConfiguration where it's possible
[ https://issues.apache.org/jira/browse/TEZ-4175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17161824#comment-17161824 ] László Bodor commented on TEZ-4175: --- could you please take a look at this patch [~jeagles], [~gopalv]? 1. eliminates the doubled tez + yarn config from AMConfiguration and relies on single tezconf 2. eliminates the yarn conf from classes extending FrameworkClient (the only relevant is TezYarnClient which can use tezconf with this patch) {code} public void init(TezConfiguration tezConf) { yarnClient.init(tezConf); } {code} I don't know the history of doubled tez/yarn confs, but is seems like this patch could prevent creating unnecessary conf objects > Consider removing YarnConfiguration where it's possible > --- > > Key: TEZ-4175 > URL: https://issues.apache.org/jira/browse/TEZ-4175 > Project: Apache Tez > Issue Type: Improvement >Reporter: László Bodor >Assignee: László Bodor >Priority: Major > Attachments: TEZ-4175.01.patch, TEZ-4175.02.patch, TEZ-4175.03.patch, > TEZ-4175.03.patch > > > A comment in DAGAppmaster made me think that we don't need to rely on > [YarnConfiguration|https://github.com/apache/hadoop/blob/branch-3.1.3/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java] > in all cases, what if it can be replace with base Configuration... > {code} > // TODO Does this really need to be a YarnConfiguration ? > Configuration conf = new Configuration(new YarnConfiguration()); > {code} > In hadoop 3.1.3 source, I cannot see that it adds e.g. yarn-site as a > resource: > {code} > public YarnConfiguration() { > super(); > } > > public YarnConfiguration(Configuration conf) { > super(conf); > if (! (conf instanceof YarnConfiguration)) { > this.reloadConfiguration(); > } > } > {code} > in current codebase: > {code} > grep -iRH "new YarnConfiguration" --include="*.java" > tez-plugins/tez-history-parser/src/main/java/org/apache/tez/history/ATSImportTool.java: > YarnConfiguration yarnConf = new YarnConfiguration(conf); > tez-plugins/tez-aux-services/src/main/java/org/apache/tez/auxservices/ShuffleHandler.java: > super.serviceInit(new YarnConfiguration(conf)); > tez-api/src/test/java/org/apache/tez/dag/api/client/rpc/TestDAGClient.java: > YarnConfiguration yarnConf = new YarnConfiguration(tezConf); > tez-api/src/test/java/org/apache/tez/dag/api/client/rpc/TestDAGClient.java: > YarnConfiguration yarnConf = new YarnConfiguration(tezConf); > tez-api/src/test/java/org/apache/tez/dag/api/client/rpc/TestDAGClient.java: > YarnConfiguration yarnConf = new YarnConfiguration(tezConf); > tez-api/src/test/java/org/apache/tez/client/TestTezClient.java: > tezClient.init(new TezConfiguration(false), new YarnConfiguration()); > tez-api/src/main/java/org/apache/tez/client/TezClient.java: > amConfig.setYarnConfiguration(new > YarnConfiguration(amConfig.getTezConfiguration())); > tez-api/src/main/java/org/apache/tez/client/TezClient.java: > amConfig.setYarnConfiguration(new > YarnConfiguration(amConfig.getTezConfiguration())); > tez-api/src/main/java/org/apache/tez/client/TezClient.java:return > getDAGClient(appId, tezConf, new YarnConfiguration(tezConf), frameworkClient, > ugi); > tez-tests/src/test/java/org/apache/tez/test/FaultToleranceTestRunner.java: > tezConf = new TezConfiguration(new YarnConfiguration()); > tez-tests/src/test/java/org/apache/tez/test/FaultToleranceTestRunner.java: >tezConf = new TezConfiguration(new YarnConfiguration(this.conf)); > tez-mapreduce/src/test/java/org/apache/tez/mapreduce/hadoop/TestMRInputHelpers.java: > Configuration testConf = new YarnConfiguration( > tez-mapreduce/src/main/java/org/apache/tez/mapreduce/client/YARNRunner.java: > this(conf, new ResourceMgrDelegate(new YarnConfiguration(conf))); > tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java: > Configuration conf = new Configuration(new YarnConfiguration()); > tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java: > Configuration conf = new Configuration(new YarnConfiguration()); > tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java: > Configuration tezConf = new Configuration(new YarnConfiguration()); > tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java: > Configuration tezConf = new Configuration(new YarnConfiguration()); > tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java: > Configuration tezConf = new Configuration(new YarnConfiguration()); > tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java: > Configuration tezConf = new Configuration(new
[jira] [Commented] (TEZ-4175) Consider removing YarnConfiguration where it's possible
[ https://issues.apache.org/jira/browse/TEZ-4175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17116070#comment-17116070 ] TezQA commented on TEZ-4175: | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 17m 57s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m 0s{color} | {color:green} No case conflicting files found. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 1s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 5 new or modified test files. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 26s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m 43s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 11s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m 2s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 19s{color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 0m 31s{color} | {color:blue} Used deprecated FindBugs config; considering switching to SpotBugs. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m 20s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 8s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 58s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 21s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 21s{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 0m 14s{color} | {color:orange} tez-api: The patch generated 1 new + 125 unchanged - 3 fixed = 126 total (was 128) {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 10s{color} | {color:green} tez-mapreduce: The patch generated 0 new + 18 unchanged - 1 fixed = 18 total (was 19) {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 20s{color} | {color:green} tez-dag: The patch generated 0 new + 169 unchanged - 2 fixed = 169 total (was 171) {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 10s{color} | {color:green} tez-tests: The patch generated 0 new + 8 unchanged - 2 fixed = 8 total (was 10) {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 9s{color} | {color:green} The patch passed checkstyle in tez-history-parser {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 9s{color} | {color:green} The patch passed checkstyle in tez-aux-services {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 30s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m 19s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 42s{color} | {color:green} tez-api in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 4s{color} | {color:green} tez-mapreduce in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m 27s{color} | {color:green} tez-dag in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 32m 49s{color} | {color:green} tez-tests in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 7s{color} | {color:green} tez-history-parser in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit
[jira] [Commented] (TEZ-4175) Consider removing YarnConfiguration where it's possible
[ https://issues.apache.org/jira/browse/TEZ-4175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17103691#comment-17103691 ] TezQA commented on TEZ-4175: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 32s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m 0s{color} | {color:green} No case conflicting files found. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 5 new or modified test files. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 21s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m 50s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 35s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m 31s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 45s{color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 0m 38s{color} | {color:blue} Used deprecated FindBugs config; considering switching to SpotBugs. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m 28s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 9s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m 2s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 29s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 29s{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 0m 15s{color} | {color:orange} tez-api: The patch generated 1 new + 125 unchanged - 3 fixed = 126 total (was 128) {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 12s{color} | {color:green} tez-mapreduce: The patch generated 0 new + 18 unchanged - 1 fixed = 18 total (was 19) {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 22s{color} | {color:green} tez-dag: The patch generated 0 new + 169 unchanged - 2 fixed = 169 total (was 171) {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 12s{color} | {color:green} tez-tests: The patch generated 0 new + 8 unchanged - 2 fixed = 8 total (was 10) {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 10s{color} | {color:green} The patch passed checkstyle in tez-history-parser {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 11s{color} | {color:green} The patch passed checkstyle in tez-aux-services {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 37s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m 7s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 45s{color} | {color:green} tez-api in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 6s{color} | {color:green} tez-mapreduce in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m 59s{color} | {color:green} tez-dag in the patch passed. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 32m 56s{color} | {color:red} tez-tests in the patch failed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 10s{color} | {color:green} tez-history-parser in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} |
[jira] [Commented] (TEZ-4175) Consider removing YarnConfiguration where it's possible
[ https://issues.apache.org/jira/browse/TEZ-4175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17103297#comment-17103297 ] TezQA commented on TEZ-4175: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 36s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m 0s{color} | {color:green} No case conflicting files found. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 5 new or modified test files. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 9s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m 56s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 40s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m 30s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 48s{color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 0m 32s{color} | {color:blue} Used deprecated FindBugs config; considering switching to SpotBugs. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m 32s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 8s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m 5s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 30s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 30s{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 0m 16s{color} | {color:orange} tez-api: The patch generated 1 new + 126 unchanged - 3 fixed = 127 total (was 129) {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 12s{color} | {color:green} tez-mapreduce: The patch generated 0 new + 18 unchanged - 1 fixed = 18 total (was 19) {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 21s{color} | {color:green} tez-dag: The patch generated 0 new + 169 unchanged - 2 fixed = 169 total (was 171) {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 11s{color} | {color:green} tez-tests: The patch generated 0 new + 8 unchanged - 2 fixed = 8 total (was 10) {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 10s{color} | {color:green} The patch passed checkstyle in tez-history-parser {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 10s{color} | {color:green} The patch passed checkstyle in tez-aux-services {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 21s{color} | {color:red} tez-api generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m 7s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} unit {color} | {color:red} 1m 46s{color} | {color:red} tez-api in the patch failed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 7s{color} | {color:green} tez-mapreduce in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m 28s{color} | {color:green} tez-dag in the patch passed. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 41m 43s{color} | {color:red} tez-tests in the patch failed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 11s{color} | {color:green} tez-history-parser in the patch passed. {color} | | {color:green}+1{color}
[jira] [Commented] (TEZ-4175) Consider removing YarnConfiguration where it's possible
[ https://issues.apache.org/jira/browse/TEZ-4175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17103259#comment-17103259 ] TezQA commented on TEZ-4175: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 9m 21s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m 0s{color} | {color:green} No case conflicting files found. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 5 new or modified test files. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 14s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m 10s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 10s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m 0s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 19s{color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 0m 34s{color} | {color:blue} Used deprecated FindBugs config; considering switching to SpotBugs. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m 19s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 8s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 58s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 22s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 22s{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 0m 14s{color} | {color:orange} tez-api: The patch generated 5 new + 125 unchanged - 3 fixed = 130 total (was 128) {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 0m 10s{color} | {color:orange} tez-mapreduce: The patch generated 1 new + 18 unchanged - 1 fixed = 19 total (was 19) {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 20s{color} | {color:green} tez-dag: The patch generated 0 new + 169 unchanged - 2 fixed = 169 total (was 171) {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 10s{color} | {color:green} tez-tests: The patch generated 0 new + 8 unchanged - 2 fixed = 8 total (was 10) {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 9s{color} | {color:green} The patch passed checkstyle in tez-history-parser {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 9s{color} | {color:green} The patch passed checkstyle in tez-aux-services {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 22s{color} | {color:red} tez-api generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m 20s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 40s{color} | {color:green} tez-api in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 3s{color} | {color:green} tez-mapreduce in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m 58s{color} | {color:green} tez-dag in the patch passed. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 39m 48s{color} | {color:red} tez-tests in the patch failed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 7s{color} | {color:green} tez-history-parser in the patch passed. {color} | |
[jira] [Commented] (TEZ-4175) Consider removing YarnConfiguration where it's possible
[ https://issues.apache.org/jira/browse/TEZ-4175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17103232#comment-17103232 ] László Bodor commented on TEZ-4175: --- uploaded experimental patch [^TEZ-4175.01.patch] for kicking off tests > Consider removing YarnConfiguration where it's possible > --- > > Key: TEZ-4175 > URL: https://issues.apache.org/jira/browse/TEZ-4175 > Project: Apache Tez > Issue Type: Improvement >Reporter: László Bodor >Assignee: László Bodor >Priority: Major > Attachments: TEZ-4175.01.patch > > > A comment in DAGAppmaster made me think that we don't need to rely on > [YarnConfiguration|https://github.com/apache/hadoop/blob/branch-3.1.3/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java] > in all cases, what if it can be replace with base Configuration... > {code} > // TODO Does this really need to be a YarnConfiguration ? > Configuration conf = new Configuration(new YarnConfiguration()); > {code} > From hadoop 3.1.3, I cannot see that it adds e.g. yarn-site as a resource: > {code} > public YarnConfiguration() { > super(); > } > > public YarnConfiguration(Configuration conf) { > super(conf); > if (! (conf instanceof YarnConfiguration)) { > this.reloadConfiguration(); > } > } > {code} > in current codebase: > {code} > grep -iRH "new YarnConfiguration" --include="*.java" > tez-plugins/tez-history-parser/src/main/java/org/apache/tez/history/ATSImportTool.java: > YarnConfiguration yarnConf = new YarnConfiguration(conf); > tez-plugins/tez-aux-services/src/main/java/org/apache/tez/auxservices/ShuffleHandler.java: > super.serviceInit(new YarnConfiguration(conf)); > tez-api/src/test/java/org/apache/tez/dag/api/client/rpc/TestDAGClient.java: > YarnConfiguration yarnConf = new YarnConfiguration(tezConf); > tez-api/src/test/java/org/apache/tez/dag/api/client/rpc/TestDAGClient.java: > YarnConfiguration yarnConf = new YarnConfiguration(tezConf); > tez-api/src/test/java/org/apache/tez/dag/api/client/rpc/TestDAGClient.java: > YarnConfiguration yarnConf = new YarnConfiguration(tezConf); > tez-api/src/test/java/org/apache/tez/client/TestTezClient.java: > tezClient.init(new TezConfiguration(false), new YarnConfiguration()); > tez-api/src/main/java/org/apache/tez/client/TezClient.java: > amConfig.setYarnConfiguration(new > YarnConfiguration(amConfig.getTezConfiguration())); > tez-api/src/main/java/org/apache/tez/client/TezClient.java: > amConfig.setYarnConfiguration(new > YarnConfiguration(amConfig.getTezConfiguration())); > tez-api/src/main/java/org/apache/tez/client/TezClient.java:return > getDAGClient(appId, tezConf, new YarnConfiguration(tezConf), frameworkClient, > ugi); > tez-tests/src/test/java/org/apache/tez/test/FaultToleranceTestRunner.java: > tezConf = new TezConfiguration(new YarnConfiguration()); > tez-tests/src/test/java/org/apache/tez/test/FaultToleranceTestRunner.java: >tezConf = new TezConfiguration(new YarnConfiguration(this.conf)); > tez-mapreduce/src/test/java/org/apache/tez/mapreduce/hadoop/TestMRInputHelpers.java: > Configuration testConf = new YarnConfiguration( > tez-mapreduce/src/main/java/org/apache/tez/mapreduce/client/YARNRunner.java: > this(conf, new ResourceMgrDelegate(new YarnConfiguration(conf))); > tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java: > Configuration conf = new Configuration(new YarnConfiguration()); > tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java: > Configuration conf = new Configuration(new YarnConfiguration()); > tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java: > Configuration tezConf = new Configuration(new YarnConfiguration()); > tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java: > Configuration tezConf = new Configuration(new YarnConfiguration()); > tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java: > Configuration tezConf = new Configuration(new YarnConfiguration()); > tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java: > Configuration tezConf = new Configuration(new YarnConfiguration()); > tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java: > Configuration tezConf = new Configuration(new YarnConfiguration()); > tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java: > Configuration tezConf = new Configuration(new YarnConfiguration()); > tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java: > Configuration tezConf = new Configuration(new YarnConfiguration()); > tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java: >