[
https://issues.apache.org/jira/browse/TEZ-3874?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16283953#comment-16283953
]
Eric Wohlstadter commented on TEZ-3874:
---------------------------------------
[~jlowe]
That makes sense to me.
There is a difference here than with skipping null values. By wrapping with
YarnConfiguration, the deprecationContext correctly handles translating
"yarn.resourcemanager.zk-address".
Although, I don't think Tez uses that parameter anywhere. So skipping the null
values is just as good and more general.
I'll go ahead and change it how you suggested, unless I'm wrong about not
needing "yarn.resourcemanager.zk-address" anywhere in Tez.
> NPE in TezClientUtils when "yarn.resourcemanager.zk-address" is present in
> Configuration
> ----------------------------------------------------------------------------------------
>
> Key: TEZ-3874
> URL: https://issues.apache.org/jira/browse/TEZ-3874
> Project: Apache Tez
> Issue Type: Bug
> Affects Versions: 0.9.1
> Reporter: Eric Wohlstadter
> Priority: Blocker
> Fix For: 0.9.1
>
> Attachments: TEZ-3874.1.patch
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> "yarn.resourcemanager.zk-address" is deprecated in favor of
> "hadoop.zk.address" for Hadoop 2.9+.
> Configuration base class does't auto-translate the deprecation. Only
> YarnConfiguration applies the translation.
> In TezClientUtils.createFinalConfProtoForApp, a NPE is throw if
> "yarn.resourcemanager.zk-address" is present in the Configuration.
> {code}
> for (Entry<String, String> entry : amConf) {
> PlanKeyValuePair.Builder kvp = PlanKeyValuePair.newBuilder();
> kvp.setKey(entry.getKey());
> kvp.setValue(amConf.get(entry.getKey()));
> builder.addConfKeyValues(kvp);
> }
> {code}
> Even though Tez is not specifically looking for the deprecated property,
> {{amConf.get(entry.getKey())}} will find it during the iteration, if it is in
> any of the merged xml property resources.
> {{amConf.get(entry.getKey())}} will return null, and {{kvp.setValue(null)}}
> will trigger NPE.
> Suggested solution is to change to:
> {code}
> YarnConfiguration wrappedConf = new YarnConfiguration(amConf);
> for (Entry<String, String> entry : wrappedConf) {
> PlanKeyValuePair.Builder kvp = PlanKeyValuePair.newBuilder();
> kvp.setKey(entry.getKey());
> kvp.setValue(wrappedConf.get(entry.getKey()));
> builder.addConfKeyValues(kvp);
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
