[jira] [Commented] (TEZ-3901) Add hadoop3 profile for upgrade to Jersey 1.19
[ https://issues.apache.org/jira/browse/TEZ-3901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16390760#comment-16390760 ] Eric Wohlstadter commented on TEZ-3901: --- np. I should have pointed it out. Thanks for your help. > Add hadoop3 profile for upgrade to Jersey 1.19 > -- > > Key: TEZ-3901 > URL: https://issues.apache.org/jira/browse/TEZ-3901 > Project: Apache Tez > Issue Type: Improvement >Reporter: Eric Wohlstadter >Assignee: Eric Wohlstadter >Priority: Major > Attachments: TEZ-3901.1.patch, TEZ-3901.2.patch > > > From [~harishjp]: > "DAGAppMaster fails to start when using hadoop3 and ATSv15, because > TimelineWriter has been changed to use jersey-client 1.19 in hadoop3, but tez > packages jersey-client 1.9 with it. There are incompatible changes between > them, so we cannot upgrade to 1.19 for all versions, it should be 1.9 in > older hadoop and 1.19 in hadoop3." > > This patch includes some copy and paste of the hadoop28 profile to a hadoop3 > profile. Maven doesn't include anything like "profile inheritance". -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (TEZ-3901) Add hadoop3 profile for upgrade to Jersey 1.19
[ https://issues.apache.org/jira/browse/TEZ-3901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16390759#comment-16390759 ] Harish Jaiprakash commented on TEZ-3901: Sorry [~ewohlstadter]. I did not see the second patch. LGTM +1. > Add hadoop3 profile for upgrade to Jersey 1.19 > -- > > Key: TEZ-3901 > URL: https://issues.apache.org/jira/browse/TEZ-3901 > Project: Apache Tez > Issue Type: Improvement >Reporter: Eric Wohlstadter >Assignee: Eric Wohlstadter >Priority: Major > Attachments: TEZ-3901.1.patch, TEZ-3901.2.patch > > > From [~harishjp]: > "DAGAppMaster fails to start when using hadoop3 and ATSv15, because > TimelineWriter has been changed to use jersey-client 1.19 in hadoop3, but tez > packages jersey-client 1.9 with it. There are incompatible changes between > them, so we cannot upgrade to 1.19 for all versions, it should be 1.9 in > older hadoop and 1.19 in hadoop3." > > This patch includes some copy and paste of the hadoop28 profile to a hadoop3 > profile. Maven doesn't include anything like "profile inheritance". -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (TEZ-3901) Add hadoop3 profile for upgrade to Jersey 1.19
[ https://issues.apache.org/jira/browse/TEZ-3901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16390757#comment-16390757 ] Eric Wohlstadter commented on TEZ-3901: --- [~harishjp] Are you using the second patch, TEZ-3901.2.patch? That error looks like the one you found in the first patch. > Add hadoop3 profile for upgrade to Jersey 1.19 > -- > > Key: TEZ-3901 > URL: https://issues.apache.org/jira/browse/TEZ-3901 > Project: Apache Tez > Issue Type: Improvement >Reporter: Eric Wohlstadter >Assignee: Eric Wohlstadter >Priority: Major > Attachments: TEZ-3901.1.patch, TEZ-3901.2.patch > > > From [~harishjp]: > "DAGAppMaster fails to start when using hadoop3 and ATSv15, because > TimelineWriter has been changed to use jersey-client 1.19 in hadoop3, but tez > packages jersey-client 1.9 with it. There are incompatible changes between > them, so we cannot upgrade to 1.19 for all versions, it should be 1.9 in > older hadoop and 1.19 in hadoop3." > > This patch includes some copy and paste of the hadoop28 profile to a hadoop3 > profile. Maven doesn't include anything like "profile inheritance". -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (TEZ-3901) Add hadoop3 profile for upgrade to Jersey 1.19
[
https://issues.apache.org/jira/browse/TEZ-3901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16390745#comment-16390745
]
Harish Jaiprakash commented on TEZ-3901:
{code:java}
$ mvn clean install[INFO] Scanning for projects...
[ERROR] [ERROR] Some problems were encountered while processing the POMs:
[ERROR] 'dependencies.dependency.version' for com.sun.jersey:jersey-client:jar
must be a valid version but is '${jersey.version}'. @
org.apache.tez:tez:0.9.2-SNAPSHOT, /Users/harishjp/devel/tez/pom.xml, line 727,
column 17
[ERROR] 'dependencies.dependency.version' for com.sun.jersey:jersey-json:jar
must be a valid version but is '${jersey.version}'. @
org.apache.tez:tez:0.9.2-SNAPSHOT, /Users/harishjp/devel/tez/pom.xml, line 732,
column 18
...{code}
I compile with java 8, which activates jdk1.8 profile and hadoop27 profile gets
deactivated, because of how maven handles activateByDefault and hence the
suggestion to move the property jersey.version to global properties and
override them for hadoop28. With this patch we cannot build with Java 8 and
hadoop 2.7.x.
> Add hadoop3 profile for upgrade to Jersey 1.19
> --
>
> Key: TEZ-3901
> URL: https://issues.apache.org/jira/browse/TEZ-3901
> Project: Apache Tez
> Issue Type: Improvement
>Reporter: Eric Wohlstadter
>Assignee: Eric Wohlstadter
>Priority: Major
> Attachments: TEZ-3901.1.patch, TEZ-3901.2.patch
>
>
> From [~harishjp]:
> "DAGAppMaster fails to start when using hadoop3 and ATSv15, because
> TimelineWriter has been changed to use jersey-client 1.19 in hadoop3, but tez
> packages jersey-client 1.9 with it. There are incompatible changes between
> them, so we cannot upgrade to 1.19 for all versions, it should be 1.9 in
> older hadoop and 1.19 in hadoop3."
>
> This patch includes some copy and paste of the hadoop28 profile to a hadoop3
> profile. Maven doesn't include anything like "profile inheritance".
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
Failed: TEZ-3901 PreCommit Build #2739
Jira: https://issues.apache.org/jira/browse/TEZ-3901
Build: https://builds.apache.org/job/PreCommit-TEZ-Build/2739/
###
## LAST 60 LINES OF THE CONSOLE
###
[...truncated 340.80 KB...]
[INFO] Final Memory: 84M/1327M
[INFO]
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12913481/TEZ-3901.2.patch
against master revision 82d73b3.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/2739//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/2739//console
This message is automatically generated.
==
==
Adding comment to Jira.
==
==
Unable to log in to server:
https://issues.apache.org/jira/rpc/soap/jirasoapservice-v2 with user: tezqa.
Cause: (404)404
Unable to log in to server:
https://issues.apache.org/jira/rpc/soap/jirasoapservice-v2 with user: tezqa.
Cause: (404)404
==
==
Finished build.
==
==
Build step 'Execute shell' marked build as failure
Archiving artifacts
[Fast Archiver] Compressed 3.53 MB of artifacts by 12.4% relative to #2736
[description-setter] Could not determine description.
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
###
## FAILED TESTS (if any)
##
All tests passed
[jira] [Commented] (TEZ-3901) Add hadoop3 profile for upgrade to Jersey 1.19
[
https://issues.apache.org/jira/browse/TEZ-3901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16390550#comment-16390550
]
Eric Wohlstadter commented on TEZ-3901:
---
[~harishjp]
I see the same behavior, you can't explicitly specify -Phadoop27. This is not
related to this patch.
I think it's ok, because BUILDING.txt doesn't say that you should ever specify
-Phadoop27.
FOR 2.7.0
{code}
$ mvn clean install
$ ls tez-dist/target/tez-0.9.2-SNAPSHOT/lib/ | grep jersey
jersey-client-1.9.jar
jersey-core-1.9.jar
jersey-guice-1.9.jar
jersey-json-1.9.jar
{code}
FOR 2.8.2
{code}
$ mvn -Phadoop28 -P\!hadoop27 -Dhadoop.version=2.8.2 clean install
$ ls tez-dist/target/tez-0.9.2-SNAPSHOT/lib/ | grep jersey
jersey-client-1.9.jar
jersey-core-1.9.jar
jersey-guice-1.9.jar
jersey-json-1.9.jar
{code}
FOR 3.0.0
{code}
$ mvn -Phadoop3 -P\!hadoop27 -Dhadoop.version=3.0.0 clean install
$ ls tez-dist/target/tez-0.9.2-SNAPSHOT/lib/ | grep jersey
jersey-client-1.19.jar
jersey-core-1.19.jar
jersey-guice-1.19.jar
jersey-json-1.19.jar
jersey-servlet-1.19.jar
{code}
> Add hadoop3 profile for upgrade to Jersey 1.19
> --
>
> Key: TEZ-3901
> URL: https://issues.apache.org/jira/browse/TEZ-3901
> Project: Apache Tez
> Issue Type: Improvement
>Reporter: Eric Wohlstadter
>Assignee: Eric Wohlstadter
>Priority: Major
> Attachments: TEZ-3901.1.patch, TEZ-3901.2.patch
>
>
> From [~harishjp]:
> "DAGAppMaster fails to start when using hadoop3 and ATSv15, because
> TimelineWriter has been changed to use jersey-client 1.19 in hadoop3, but tez
> packages jersey-client 1.9 with it. There are incompatible changes between
> them, so we cannot upgrade to 1.19 for all versions, it should be 1.9 in
> older hadoop and 1.19 in hadoop3."
>
> This patch includes some copy and paste of the hadoop28 profile to a hadoop3
> profile. Maven doesn't include anything like "profile inheritance".
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (TEZ-3901) Add hadoop3 profile for upgrade to Jersey 1.19
[ https://issues.apache.org/jira/browse/TEZ-3901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Wohlstadter updated TEZ-3901: -- Attachment: TEZ-3901.2.patch > Add hadoop3 profile for upgrade to Jersey 1.19 > -- > > Key: TEZ-3901 > URL: https://issues.apache.org/jira/browse/TEZ-3901 > Project: Apache Tez > Issue Type: Improvement >Reporter: Eric Wohlstadter >Assignee: Eric Wohlstadter >Priority: Major > Attachments: TEZ-3901.1.patch, TEZ-3901.2.patch > > > From [~harishjp]: > "DAGAppMaster fails to start when using hadoop3 and ATSv15, because > TimelineWriter has been changed to use jersey-client 1.19 in hadoop3, but tez > packages jersey-client 1.9 with it. There are incompatible changes between > them, so we cannot upgrade to 1.19 for all versions, it should be 1.9 in > older hadoop and 1.19 in hadoop3." > > This patch includes some copy and paste of the hadoop28 profile to a hadoop3 > profile. Maven doesn't include anything like "profile inheritance". -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (TEZ-3901) Add hadoop3 profile for upgrade to Jersey 1.19
[ https://issues.apache.org/jira/browse/TEZ-3901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16390511#comment-16390511 ] Harish Jaiprakash commented on TEZ-3901: This might explain: https://issues.apache.org/jira/browse/MNG-4917 since jdk18 is getting activated it disables hadoop27. Adding jersey.version=1.9 into the default properties list and override in a profile is working for me. If we pass -Phadoop27, tez-plugins/tez-yarn-timeline-history/pom.xml fails. The profile dependency is wrong there, its an older bug we never hit, since we assumed default build was hadoop27. > Add hadoop3 profile for upgrade to Jersey 1.19 > -- > > Key: TEZ-3901 > URL: https://issues.apache.org/jira/browse/TEZ-3901 > Project: Apache Tez > Issue Type: Improvement >Reporter: Eric Wohlstadter >Assignee: Eric Wohlstadter >Priority: Major > Attachments: TEZ-3901.1.patch > > > From [~harishjp]: > "DAGAppMaster fails to start when using hadoop3 and ATSv15, because > TimelineWriter has been changed to use jersey-client 1.19 in hadoop3, but tez > packages jersey-client 1.9 with it. There are incompatible changes between > them, so we cannot upgrade to 1.19 for all versions, it should be 1.9 in > older hadoop and 1.19 in hadoop3." > > This patch includes some copy and paste of the hadoop28 profile to a hadoop3 > profile. Maven doesn't include anything like "profile inheritance". -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (TEZ-3901) Add hadoop3 profile for upgrade to Jersey 1.19
[ https://issues.apache.org/jira/browse/TEZ-3901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16390480#comment-16390480 ] Eric Wohlstadter commented on TEZ-3901: --- [~harishjp] Thanks. I see what you mean. hadoop28 and hadoop3 work ok, but hadoop27 is not working. I'll look into it. > Add hadoop3 profile for upgrade to Jersey 1.19 > -- > > Key: TEZ-3901 > URL: https://issues.apache.org/jira/browse/TEZ-3901 > Project: Apache Tez > Issue Type: Improvement >Reporter: Eric Wohlstadter >Assignee: Eric Wohlstadter >Priority: Major > Attachments: TEZ-3901.1.patch > > > From [~harishjp]: > "DAGAppMaster fails to start when using hadoop3 and ATSv15, because > TimelineWriter has been changed to use jersey-client 1.19 in hadoop3, but tez > packages jersey-client 1.9 with it. There are incompatible changes between > them, so we cannot upgrade to 1.19 for all versions, it should be 1.9 in > older hadoop and 1.19 in hadoop3." > > This patch includes some copy and paste of the hadoop28 profile to a hadoop3 > profile. Maven doesn't include anything like "profile inheritance". -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (TEZ-3901) Add hadoop3 profile for upgrade to Jersey 1.19
[
https://issues.apache.org/jira/browse/TEZ-3901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16390472#comment-16390472
]
Harish Jaiprakash edited comment on TEZ-3901 at 3/8/18 12:05 AM:
-
mvn clean install fails for me. Not sure why its not reading the jersey.version
property from hadoop27 profile. My maven version is: 3.5.2.
The other 2 profiles are working fine.
{{[ERROR] [ERROR] Some problems were encountered while processing the POMs:}}
{{[ERROR] 'dependencies.dependency.version' for
com.sun.jersey:jersey-client:jar must be a valid version but is
'${jersey.version}'. @ org.apache.tez:tez:0.9.2-SNAPSHOT,
/Users/harishjp/devel/tez/pom.xml, line 727, column 17}}
{{[ERROR] 'dependencies.dependency.version' for com.sun.jersey:jersey-json:jar
must be a valid version but is '${jersey.version}'. @
org.apache.tez:tez:0.9.2-SNAPSHOT, /Users/harishjp/devel/tez/pom.xml, line 732,
column 18}}
was (Author: harishjp):
mvn clean install fails for me. Not sure why its not reading the jersey.version
property from hadoop27 profile. My maven version is: 3.5.2.
The other 2 profiles are working fine.
> Add hadoop3 profile for upgrade to Jersey 1.19
> --
>
> Key: TEZ-3901
> URL: https://issues.apache.org/jira/browse/TEZ-3901
> Project: Apache Tez
> Issue Type: Improvement
>Reporter: Eric Wohlstadter
>Assignee: Eric Wohlstadter
>Priority: Major
> Attachments: TEZ-3901.1.patch
>
>
> From [~harishjp]:
> "DAGAppMaster fails to start when using hadoop3 and ATSv15, because
> TimelineWriter has been changed to use jersey-client 1.19 in hadoop3, but tez
> packages jersey-client 1.9 with it. There are incompatible changes between
> them, so we cannot upgrade to 1.19 for all versions, it should be 1.9 in
> older hadoop and 1.19 in hadoop3."
>
> This patch includes some copy and paste of the hadoop28 profile to a hadoop3
> profile. Maven doesn't include anything like "profile inheritance".
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (TEZ-3901) Add hadoop3 profile for upgrade to Jersey 1.19
[ https://issues.apache.org/jira/browse/TEZ-3901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16390472#comment-16390472 ] Harish Jaiprakash commented on TEZ-3901: mvn clean install fails for me. Not sure why its not reading the jersey.version property from hadoop27 profile. My maven version is: 3.5.2. The other 2 profiles are working fine. > Add hadoop3 profile for upgrade to Jersey 1.19 > -- > > Key: TEZ-3901 > URL: https://issues.apache.org/jira/browse/TEZ-3901 > Project: Apache Tez > Issue Type: Improvement >Reporter: Eric Wohlstadter >Assignee: Eric Wohlstadter >Priority: Major > Attachments: TEZ-3901.1.patch > > > From [~harishjp]: > "DAGAppMaster fails to start when using hadoop3 and ATSv15, because > TimelineWriter has been changed to use jersey-client 1.19 in hadoop3, but tez > packages jersey-client 1.9 with it. There are incompatible changes between > them, so we cannot upgrade to 1.19 for all versions, it should be 1.9 in > older hadoop and 1.19 in hadoop3." > > This patch includes some copy and paste of the hadoop28 profile to a hadoop3 > profile. Maven doesn't include anything like "profile inheritance". -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (TEZ-3903) [Umbrella] 0.9.2 maven deps. changes
Eric Wohlstadter created TEZ-3903: - Summary: [Umbrella] 0.9.2 maven deps. changes Key: TEZ-3903 URL: https://issues.apache.org/jira/browse/TEZ-3903 Project: Apache Tez Issue Type: Improvement Reporter: Eric Wohlstadter Assignee: Eric Wohlstadter Placeholder to organize proposed dependency change tickets for 0.9.2. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (TEZ-3902) Upgrade to netty-3.10.5.Final.jar
Eric Wohlstadter created TEZ-3902: - Summary: Upgrade to netty-3.10.5.Final.jar Key: TEZ-3902 URL: https://issues.apache.org/jira/browse/TEZ-3902 Project: Apache Tez Issue Type: Improvement Reporter: Eric Wohlstadter Hadoop 3 and Hive have upgraded to netty-3.10.5.Final, which is not compatible with current Tez dependency netty-3.6.2.Final. However, org.apache.tez.shufflehandler.ShuffleHandler depends on 3.6.2 specific methods. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (TEZ-3900) upgrade to a recent guava version
[ https://issues.apache.org/jira/browse/TEZ-3900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16390138#comment-16390138 ] Jonathan Eagles commented on TEZ-3900: -- If you are deploying tez according to best practices, then the guava version from hive should be used and not the one from tez. Can you describe the actual failure you see. Also, would you say the failure is intermittent or does it always fail. > upgrade to a recent guava version > - > > Key: TEZ-3900 > URL: https://issues.apache.org/jira/browse/TEZ-3900 > Project: Apache Tez > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: TEZ-3900.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (TEZ-3900) upgrade to a recent guava version
[ https://issues.apache.org/jira/browse/TEZ-3900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16390121#comment-16390121 ] Sergey Shelukhin commented on TEZ-3900: --- Hmm... the problem we actually have is that Hive has recently upgraded guava and we are hitting issues due to the old version coming from Tez. The referenced JIRA hasn't had any progress in almost 2 years, I wonder if it's time to just upgrade it and force Hive/Pig to upgrade too (Hive has already done that, in fact). > upgrade to a recent guava version > - > > Key: TEZ-3900 > URL: https://issues.apache.org/jira/browse/TEZ-3900 > Project: Apache Tez > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: TEZ-3900.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (TEZ-3900) upgrade to a recent guava version
[ https://issues.apache.org/jira/browse/TEZ-3900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16389767#comment-16389767 ] Jonathan Eagles commented on TEZ-3900: -- It is difficult to upgrade guava directly like this, which is why it hasn't been done yet. This will force all users of tez (pig/hive/etc) plus all users of that code to be forced to upgrade accordingly. Instead we have been taking an approach where Tez can easily switch between versions of guava (essentially using a subset of APIs that work across all supported versions of guava). To that point there is another jira that is taking a completely different approach to shade guava in tez and have users rely directly on library provided by pig/hive or possibly hadoop directly. Please have a look at the reference jira TEZ-2164 > upgrade to a recent guava version > - > > Key: TEZ-3900 > URL: https://issues.apache.org/jira/browse/TEZ-3900 > Project: Apache Tez > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: TEZ-3900.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (TEZ-3874) NPE in TezClientUtils when "yarn.resourcemanager.zk-address" is present in Configuration
[
https://issues.apache.org/jira/browse/TEZ-3874?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16389712#comment-16389712
]
Jason Lowe commented on TEZ-3874:
-
Thanks for updating the patch! Looks much better overall, just one nit left.
testNotNullKvpWithValueReplacement is no longer testing Tez code. It creates a
mocked object and verifies the mock behaves properly, but that does not test
anything from Tez. The test along with MockKeyFailureConfiguration should be
removed.
+1 after that change.
> NPE in TezClientUtils when "yarn.resourcemanager.zk-address" is present in
> Configuration
>
>
> Key: TEZ-3874
> URL: https://issues.apache.org/jira/browse/TEZ-3874
> Project: Apache Tez
> Issue Type: Bug
>Affects Versions: 0.9.1
>Reporter: Eric Wohlstadter
>Assignee: Eric Wohlstadter
>Priority: Blocker
> Attachments: TEZ-3874.1.patch, TEZ-3874.3.patch, TEZ-3874.4.patch,
> TEZ-3874.5.patch, TEZ-3874.patch.2
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> "yarn.resourcemanager.zk-address" is deprecated in favor of
> "hadoop.zk.address" for Hadoop 2.9+.
> Configuration base class does't auto-translate the deprecation. Only
> YarnConfiguration applies the translation.
> In TezClientUtils.createFinalConfProtoForApp, a NPE is throw if
> "yarn.resourcemanager.zk-address" is present in the Configuration.
> {code}
> for (Entry entry : amConf) {
> PlanKeyValuePair.Builder kvp = PlanKeyValuePair.newBuilder();
> kvp.setKey(entry.getKey());
> kvp.setValue(amConf.get(entry.getKey()));
> builder.addConfKeyValues(kvp);
> }
> {code}
> Even though Tez is not specifically looking for the deprecated property,
> {{amConf.get(entry.getKey())}} will find it during the iteration, if it is in
> any of the merged xml property resources.
> {{amConf.get(entry.getKey())}} will return null, and {{kvp.setValue(null)}}
> will trigger NPE.
> Suggested solution is to change to:
> {code}
> YarnConfiguration wrappedConf = new YarnConfiguration(amConf);
> for (Entry entry : wrappedConf) {
> PlanKeyValuePair.Builder kvp = PlanKeyValuePair.newBuilder();
> kvp.setKey(entry.getKey());
> kvp.setValue(wrappedConf.get(entry.getKey()));
> builder.addConfKeyValues(kvp);
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
