[jira] [Commented] (TWILL-262) YarnUtils#cloneHaNnCredentials uses DFSUtil#getHaNnRpcAddresses, which is removed from DFSUtils from hadoop-2.8
[
https://issues.apache.org/jira/browse/TWILL-262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16719621#comment-16719621
]
Hongyuan Li commented on TWILL-262:
---
Thanks for your review, Yim
> YarnUtils#cloneHaNnCredentials uses DFSUtil#getHaNnRpcAddresses, which is
> removed from DFSUtils from hadoop-2.8
>
>
> Key: TWILL-262
> URL: https://issues.apache.org/jira/browse/TWILL-262
> Project: Apache Twill
> Issue Type: Bug
> Components: yarn
>Affects Versions: 0.8.0, 0.9.0, 0.11.0, 0.12.0, 0.13.0
>Reporter: Hongyuan Li
>Assignee: Hongyuan Li
>Priority: Major
> Fix For: 0.14.0
>
> Attachments: errors.txt
>
>
> YarnUtils#cloneHaNnCredentials uses DFSUtil#getHaNnRpcAddresses, which is
> removed from DFSUtils from hadoop-2.8
> current code
> {code}
> public static void cloneHaNnCredentials(Configuration config) throws
> IOException {
> ……
> // Loop through all name services. Each name service could have multiple
> name node associated with it.
> for (Map.Entry> entry :
> DFSUtil.getHaNnRpcAddresses(config).entrySet()) {
> String nsId = entry.getKey();
> Map addressesInNN = entry.getValue();
> if (!HAUtil.isHAEnabled(config, nsId) || addressesInNN == null ||
> addressesInNN.isEmpty()) {
> continue;
> }
> ……
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (TWILL-262) YarnUtils#cloneHaNnCredentials uses DFSUtil#getHaNnRpcAddresses, which is removed from DFSUtils from hadoop-2.8
[
https://issues.apache.org/jira/browse/TWILL-262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16719298#comment-16719298
]
ASF GitHub Bot commented on TWILL-262:
--
Github user asfgit closed the pull request at:
https://github.com/apache/twill/pull/71
> YarnUtils#cloneHaNnCredentials uses DFSUtil#getHaNnRpcAddresses, which is
> removed from DFSUtils from hadoop-2.8
>
>
> Key: TWILL-262
> URL: https://issues.apache.org/jira/browse/TWILL-262
> Project: Apache Twill
> Issue Type: Bug
> Components: yarn
>Affects Versions: 0.8.0, 0.9.0, 0.11.0, 0.12.0, 0.13.0
>Reporter: Hongyuan Li
>Priority: Major
> Attachments: errors.txt
>
>
> YarnUtils#cloneHaNnCredentials uses DFSUtil#getHaNnRpcAddresses, which is
> removed from DFSUtils from hadoop-2.8
> current code
> {code}
> public static void cloneHaNnCredentials(Configuration config) throws
> IOException {
> ……
> // Loop through all name services. Each name service could have multiple
> name node associated with it.
> for (Map.Entry> entry :
> DFSUtil.getHaNnRpcAddresses(config).entrySet()) {
> String nsId = entry.getKey();
> Map addressesInNN = entry.getValue();
> if (!HAUtil.isHAEnabled(config, nsId) || addressesInNN == null ||
> addressesInNN.isEmpty()) {
> continue;
> }
> ……
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (TWILL-262) YarnUtils#cloneHaNnCredentials uses DFSUtil#getHaNnRpcAddresses, which is removed from DFSUtils from hadoop-2.8
[
https://issues.apache.org/jira/browse/TWILL-262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16653178#comment-16653178
]
ASF GitHub Bot commented on TWILL-262:
--
Github user piaosama commented on a diff in the pull request:
https://github.com/apache/twill/pull/71#discussion_r225824325
--- Diff:
twill-yarn/src/main/java/org/apache/twill/internal/yarn/YarnUtils.java ---
@@ -202,6 +223,28 @@ public static void cloneHaNnCredentials(Configuration
config) throws IOException
}
}
+ /***
+ * When hadoop_version > 2.8.0, class DFSUtils has no method
getHaNnRpcAddresses(Configuration config)
--- End diff --
https://github.com/apache/hadoop/commit/f02ca4ab158aa2257e839a1f74bc8254e1a3d61b#diff-f42f0d7039ba6a4ce427c67ae74dd7ec
HDFS-8185 Separate client related routines in HAUtil into a new clas also
said this.
> YarnUtils#cloneHaNnCredentials uses DFSUtil#getHaNnRpcAddresses, which is
> removed from DFSUtils from hadoop-2.8
>
>
> Key: TWILL-262
> URL: https://issues.apache.org/jira/browse/TWILL-262
> Project: Apache Twill
> Issue Type: Bug
> Components: yarn
>Affects Versions: 0.8.0, 0.9.0, 0.11.0, 0.12.0, 0.13.0
>Reporter: Hongyuan Li
>Priority: Major
> Attachments: errors.txt
>
>
> YarnUtils#cloneHaNnCredentials uses DFSUtil#getHaNnRpcAddresses, which is
> removed from DFSUtils from hadoop-2.8
> current code
> {code}
> public static void cloneHaNnCredentials(Configuration config) throws
> IOException {
> ……
> // Loop through all name services. Each name service could have multiple
> name node associated with it.
> for (Map.Entry> entry :
> DFSUtil.getHaNnRpcAddresses(config).entrySet()) {
> String nsId = entry.getKey();
> Map addressesInNN = entry.getValue();
> if (!HAUtil.isHAEnabled(config, nsId) || addressesInNN == null ||
> addressesInNN.isEmpty()) {
> continue;
> }
> ……
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (TWILL-262) YarnUtils#cloneHaNnCredentials uses DFSUtil#getHaNnRpcAddresses, which is removed from DFSUtils from hadoop-2.8
[
https://issues.apache.org/jira/browse/TWILL-262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16653160#comment-16653160
]
ASF GitHub Bot commented on TWILL-262:
--
Github user piaosama commented on a diff in the pull request:
https://github.com/apache/twill/pull/71#discussion_r225817747
--- Diff:
twill-yarn/src/main/java/org/apache/twill/internal/yarn/YarnUtils.java ---
@@ -202,6 +223,28 @@ public static void cloneHaNnCredentials(Configuration
config) throws IOException
}
}
+ /***
+ * When hadoop_version > 2.8.0, class DFSUtils has no method
getHaNnRpcAddresses(Configuration config)
--- End diff --
https://github.com/apache/hadoop/commit/d12a0a25b4da1957ba0f658694729d2f2d400c5f#diff-f42f0d7039ba6a4ce427c67ae74dd7ec
HDFS-11538. Move ClientProtocol HA proxies into hadoop-hdfs-client said
this.
> YarnUtils#cloneHaNnCredentials uses DFSUtil#getHaNnRpcAddresses, which is
> removed from DFSUtils from hadoop-2.8
>
>
> Key: TWILL-262
> URL: https://issues.apache.org/jira/browse/TWILL-262
> Project: Apache Twill
> Issue Type: Bug
> Components: yarn
>Affects Versions: 0.8.0, 0.9.0, 0.11.0, 0.12.0, 0.13.0
>Reporter: Hongyuan Li
>Priority: Major
> Attachments: errors.txt
>
>
> YarnUtils#cloneHaNnCredentials uses DFSUtil#getHaNnRpcAddresses, which is
> removed from DFSUtils from hadoop-2.8
> current code
> {code}
> public static void cloneHaNnCredentials(Configuration config) throws
> IOException {
> ……
> // Loop through all name services. Each name service could have multiple
> name node associated with it.
> for (Map.Entry> entry :
> DFSUtil.getHaNnRpcAddresses(config).entrySet()) {
> String nsId = entry.getKey();
> Map addressesInNN = entry.getValue();
> if (!HAUtil.isHAEnabled(config, nsId) || addressesInNN == null ||
> addressesInNN.isEmpty()) {
> continue;
> }
> ……
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (TWILL-262) YarnUtils#cloneHaNnCredentials uses DFSUtil#getHaNnRpcAddresses, which is removed from DFSUtils from hadoop-2.8
[
https://issues.apache.org/jira/browse/TWILL-262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16640720#comment-16640720
]
ASF GitHub Bot commented on TWILL-262:
--
Github user piaosama commented on a diff in the pull request:
https://github.com/apache/twill/pull/71#discussion_r223182267
--- Diff:
twill-yarn/src/main/java/org/apache/twill/internal/yarn/YarnUtils.java ---
@@ -345,6 +388,28 @@ private static FileSystem
getFileSystem(LocationFactory locationFactory) throws
return null;
}
+ private static void handleLogAction(Exception e) {
--- End diff --
add new code, modify the code as you suggested and add the HAutils method
judgement.
> YarnUtils#cloneHaNnCredentials uses DFSUtil#getHaNnRpcAddresses, which is
> removed from DFSUtils from hadoop-2.8
>
>
> Key: TWILL-262
> URL: https://issues.apache.org/jira/browse/TWILL-262
> Project: Apache Twill
> Issue Type: Bug
> Components: yarn
>Affects Versions: 0.8.0, 0.9.0, 0.11.0, 0.12.0, 0.13.0
>Reporter: Hongyuan Li
>Priority: Major
> Attachments: errors.txt
>
>
> YarnUtils#cloneHaNnCredentials uses DFSUtil#getHaNnRpcAddresses, which is
> removed from DFSUtils from hadoop-2.8
> current code
> {code}
> public static void cloneHaNnCredentials(Configuration config) throws
> IOException {
> ……
> // Loop through all name services. Each name service could have multiple
> name node associated with it.
> for (Map.Entry> entry :
> DFSUtil.getHaNnRpcAddresses(config).entrySet()) {
> String nsId = entry.getKey();
> Map addressesInNN = entry.getValue();
> if (!HAUtil.isHAEnabled(config, nsId) || addressesInNN == null ||
> addressesInNN.isEmpty()) {
> continue;
> }
> ……
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (TWILL-262) YarnUtils#cloneHaNnCredentials uses DFSUtil#getHaNnRpcAddresses, which is removed from DFSUtils from hadoop-2.8
[
https://issues.apache.org/jira/browse/TWILL-262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16640721#comment-16640721
]
ASF GitHub Bot commented on TWILL-262:
--
Github user piaosama commented on the issue:
https://github.com/apache/twill/pull/71
add new code, modify the code as you suggested and add the HAutils method
judgement.
> YarnUtils#cloneHaNnCredentials uses DFSUtil#getHaNnRpcAddresses, which is
> removed from DFSUtils from hadoop-2.8
>
>
> Key: TWILL-262
> URL: https://issues.apache.org/jira/browse/TWILL-262
> Project: Apache Twill
> Issue Type: Bug
> Components: yarn
>Affects Versions: 0.8.0, 0.9.0, 0.11.0, 0.12.0, 0.13.0
>Reporter: Hongyuan Li
>Priority: Major
> Attachments: errors.txt
>
>
> YarnUtils#cloneHaNnCredentials uses DFSUtil#getHaNnRpcAddresses, which is
> removed from DFSUtils from hadoop-2.8
> current code
> {code}
> public static void cloneHaNnCredentials(Configuration config) throws
> IOException {
> ……
> // Loop through all name services. Each name service could have multiple
> name node associated with it.
> for (Map.Entry> entry :
> DFSUtil.getHaNnRpcAddresses(config).entrySet()) {
> String nsId = entry.getKey();
> Map addressesInNN = entry.getValue();
> if (!HAUtil.isHAEnabled(config, nsId) || addressesInNN == null ||
> addressesInNN.isEmpty()) {
> continue;
> }
> ……
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (TWILL-262) YarnUtils#cloneHaNnCredentials uses DFSUtil#getHaNnRpcAddresses, which is removed from DFSUtils from hadoop-2.8
[
https://issues.apache.org/jira/browse/TWILL-262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16640715#comment-16640715
]
ASF GitHub Bot commented on TWILL-262:
--
Github user piaosama commented on a diff in the pull request:
https://github.com/apache/twill/pull/71#discussion_r223181503
--- Diff:
twill-yarn/src/main/java/org/apache/twill/internal/yarn/YarnUtils.java ---
@@ -202,6 +223,28 @@ public static void cloneHaNnCredentials(Configuration
config) throws IOException
}
}
+ /***
+ * When hadoop_version > 2.8.0, class DFSUtils has no method
getHaNnRpcAddresses(Configuration config)
+ * @param config
+ * @return
+ */
+ private static Set>>
getEntries(Configuration config) {
--- End diff --
ok
> YarnUtils#cloneHaNnCredentials uses DFSUtil#getHaNnRpcAddresses, which is
> removed from DFSUtils from hadoop-2.8
>
>
> Key: TWILL-262
> URL: https://issues.apache.org/jira/browse/TWILL-262
> Project: Apache Twill
> Issue Type: Bug
> Components: yarn
>Affects Versions: 0.8.0, 0.9.0, 0.11.0, 0.12.0, 0.13.0
>Reporter: Hongyuan Li
>Priority: Major
> Attachments: errors.txt
>
>
> YarnUtils#cloneHaNnCredentials uses DFSUtil#getHaNnRpcAddresses, which is
> removed from DFSUtils from hadoop-2.8
> current code
> {code}
> public static void cloneHaNnCredentials(Configuration config) throws
> IOException {
> ……
> // Loop through all name services. Each name service could have multiple
> name node associated with it.
> for (Map.Entry> entry :
> DFSUtil.getHaNnRpcAddresses(config).entrySet()) {
> String nsId = entry.getKey();
> Map addressesInNN = entry.getValue();
> if (!HAUtil.isHAEnabled(config, nsId) || addressesInNN == null ||
> addressesInNN.isEmpty()) {
> continue;
> }
> ……
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (TWILL-262) YarnUtils#cloneHaNnCredentials uses DFSUtil#getHaNnRpcAddresses, which is removed from DFSUtils from hadoop-2.8
[
https://issues.apache.org/jira/browse/TWILL-262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16640714#comment-16640714
]
ASF GitHub Bot commented on TWILL-262:
--
Github user piaosama commented on a diff in the pull request:
https://github.com/apache/twill/pull/71#discussion_r223181498
--- Diff:
twill-yarn/src/main/java/org/apache/twill/internal/yarn/YarnUtils.java ---
@@ -202,6 +223,28 @@ public static void cloneHaNnCredentials(Configuration
config) throws IOException
}
}
+ /***
+ * When hadoop_version > 2.8.0, class DFSUtils has no method
getHaNnRpcAddresses(Configuration config)
+ * @param config
+ * @return
+ */
+ private static Set>>
getEntries(Configuration config) {
+return iDFSUtilClientExists ? invoke(config) :
+DFSUtil.getHaNnRpcAddresses(config).entrySet();
+ }
+
+ private static Set>>
invoke(Configuration config) {
--- End diff --
ok
> YarnUtils#cloneHaNnCredentials uses DFSUtil#getHaNnRpcAddresses, which is
> removed from DFSUtils from hadoop-2.8
>
>
> Key: TWILL-262
> URL: https://issues.apache.org/jira/browse/TWILL-262
> Project: Apache Twill
> Issue Type: Bug
> Components: yarn
>Affects Versions: 0.8.0, 0.9.0, 0.11.0, 0.12.0, 0.13.0
>Reporter: Hongyuan Li
>Priority: Major
> Attachments: errors.txt
>
>
> YarnUtils#cloneHaNnCredentials uses DFSUtil#getHaNnRpcAddresses, which is
> removed from DFSUtils from hadoop-2.8
> current code
> {code}
> public static void cloneHaNnCredentials(Configuration config) throws
> IOException {
> ……
> // Loop through all name services. Each name service could have multiple
> name node associated with it.
> for (Map.Entry> entry :
> DFSUtil.getHaNnRpcAddresses(config).entrySet()) {
> String nsId = entry.getKey();
> Map addressesInNN = entry.getValue();
> if (!HAUtil.isHAEnabled(config, nsId) || addressesInNN == null ||
> addressesInNN.isEmpty()) {
> continue;
> }
> ……
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (TWILL-262) YarnUtils#cloneHaNnCredentials uses DFSUtil#getHaNnRpcAddresses, which is removed from DFSUtils from hadoop-2.8
[
https://issues.apache.org/jira/browse/TWILL-262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16640713#comment-16640713
]
ASF GitHub Bot commented on TWILL-262:
--
Github user piaosama commented on a diff in the pull request:
https://github.com/apache/twill/pull/71#discussion_r223181489
--- Diff:
twill-yarn/src/main/java/org/apache/twill/internal/yarn/YarnUtils.java ---
@@ -202,6 +223,28 @@ public static void cloneHaNnCredentials(Configuration
config) throws IOException
}
}
+ /***
+ * When hadoop_version > 2.8.0, class DFSUtils has no method
getHaNnRpcAddresses(Configuration config)
+ * @param config
+ * @return
+ */
+ private static Set>>
getEntries(Configuration config) {
+return iDFSUtilClientExists ? invoke(config) :
+DFSUtil.getHaNnRpcAddresses(config).entrySet();
+ }
+
+ private static Set>>
invoke(Configuration config) {
+try {
+ return ((Map) getHaNnRpcAddressesMethod.invoke(null,
config)).entrySet();
--- End diff --
getHaNnRpcAddressesMethod invoke return Map instance.
> YarnUtils#cloneHaNnCredentials uses DFSUtil#getHaNnRpcAddresses, which is
> removed from DFSUtils from hadoop-2.8
>
>
> Key: TWILL-262
> URL: https://issues.apache.org/jira/browse/TWILL-262
> Project: Apache Twill
> Issue Type: Bug
> Components: yarn
>Affects Versions: 0.8.0, 0.9.0, 0.11.0, 0.12.0, 0.13.0
>Reporter: Hongyuan Li
>Priority: Major
> Attachments: errors.txt
>
>
> YarnUtils#cloneHaNnCredentials uses DFSUtil#getHaNnRpcAddresses, which is
> removed from DFSUtils from hadoop-2.8
> current code
> {code}
> public static void cloneHaNnCredentials(Configuration config) throws
> IOException {
> ……
> // Loop through all name services. Each name service could have multiple
> name node associated with it.
> for (Map.Entry> entry :
> DFSUtil.getHaNnRpcAddresses(config).entrySet()) {
> String nsId = entry.getKey();
> Map addressesInNN = entry.getValue();
> if (!HAUtil.isHAEnabled(config, nsId) || addressesInNN == null ||
> addressesInNN.isEmpty()) {
> continue;
> }
> ……
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (TWILL-262) YarnUtils#cloneHaNnCredentials uses DFSUtil#getHaNnRpcAddresses, which is removed from DFSUtils from hadoop-2.8
[
https://issues.apache.org/jira/browse/TWILL-262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16640708#comment-16640708
]
ASF GitHub Bot commented on TWILL-262:
--
Github user piaosama commented on a diff in the pull request:
https://github.com/apache/twill/pull/71#discussion_r223181337
--- Diff:
twill-yarn/src/main/java/org/apache/twill/internal/yarn/YarnUtils.java ---
@@ -72,7 +77,23 @@
HADOOP_26
}
- private static final Logger LOG =
LoggerFactory.getLogger(YarnUtils.class);
+ private static boolean iDFSUtilClientExists = false; // use this to
judge if the hadoop version is above 2.8
--- End diff --
ok， suddenly find HAUtil class does not exists as the DFSUtils class. i
will add the two judge into the code and make the code more nice as you
commented.
> YarnUtils#cloneHaNnCredentials uses DFSUtil#getHaNnRpcAddresses, which is
> removed from DFSUtils from hadoop-2.8
>
>
> Key: TWILL-262
> URL: https://issues.apache.org/jira/browse/TWILL-262
> Project: Apache Twill
> Issue Type: Bug
> Components: yarn
>Affects Versions: 0.8.0, 0.9.0, 0.11.0, 0.12.0, 0.13.0
>Reporter: Hongyuan Li
>Priority: Major
> Attachments: errors.txt
>
>
> YarnUtils#cloneHaNnCredentials uses DFSUtil#getHaNnRpcAddresses, which is
> removed from DFSUtils from hadoop-2.8
> current code
> {code}
> public static void cloneHaNnCredentials(Configuration config) throws
> IOException {
> ……
> // Loop through all name services. Each name service could have multiple
> name node associated with it.
> for (Map.Entry> entry :
> DFSUtil.getHaNnRpcAddresses(config).entrySet()) {
> String nsId = entry.getKey();
> Map addressesInNN = entry.getValue();
> if (!HAUtil.isHAEnabled(config, nsId) || addressesInNN == null ||
> addressesInNN.isEmpty()) {
> continue;
> }
> ……
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (TWILL-262) YarnUtils#cloneHaNnCredentials uses DFSUtil#getHaNnRpcAddresses, which is removed from DFSUtils from hadoop-2.8
[
https://issues.apache.org/jira/browse/TWILL-262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16634233#comment-16634233
]
ASF GitHub Bot commented on TWILL-262:
--
Github user chtyim commented on a diff in the pull request:
https://github.com/apache/twill/pull/71#discussion_r221661448
--- Diff:
twill-yarn/src/main/java/org/apache/twill/internal/yarn/YarnUtils.java ---
@@ -202,6 +223,28 @@ public static void cloneHaNnCredentials(Configuration
config) throws IOException
}
}
+ /***
+ * When hadoop_version > 2.8.0, class DFSUtils has no method
getHaNnRpcAddresses(Configuration config)
--- End diff --
This is not a very clear javadoc that describe what this method does (it
should have the same javadoc as the `DFSUtil.getHaNnRpcAddresses` method).
> YarnUtils#cloneHaNnCredentials uses DFSUtil#getHaNnRpcAddresses, which is
> removed from DFSUtils from hadoop-2.8
>
>
> Key: TWILL-262
> URL: https://issues.apache.org/jira/browse/TWILL-262
> Project: Apache Twill
> Issue Type: Bug
> Components: yarn
>Affects Versions: 0.8.0, 0.9.0, 0.11.0, 0.12.0, 0.13.0
>Reporter: Hongyuan Li
>Priority: Major
> Attachments: errors.txt
>
>
> YarnUtils#cloneHaNnCredentials uses DFSUtil#getHaNnRpcAddresses, which is
> removed from DFSUtils from hadoop-2.8
> current code
> {code}
> public static void cloneHaNnCredentials(Configuration config) throws
> IOException {
> ……
> // Loop through all name services. Each name service could have multiple
> name node associated with it.
> for (Map.Entry> entry :
> DFSUtil.getHaNnRpcAddresses(config).entrySet()) {
> String nsId = entry.getKey();
> Map addressesInNN = entry.getValue();
> if (!HAUtil.isHAEnabled(config, nsId) || addressesInNN == null ||
> addressesInNN.isEmpty()) {
> continue;
> }
> ……
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (TWILL-262) YarnUtils#cloneHaNnCredentials uses DFSUtil#getHaNnRpcAddresses, which is removed from DFSUtils from hadoop-2.8
[
https://issues.apache.org/jira/browse/TWILL-262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16634235#comment-16634235
]
ASF GitHub Bot commented on TWILL-262:
--
Github user chtyim commented on a diff in the pull request:
https://github.com/apache/twill/pull/71#discussion_r221660130
--- Diff:
twill-yarn/src/main/java/org/apache/twill/internal/yarn/YarnUtils.java ---
@@ -202,6 +223,28 @@ public static void cloneHaNnCredentials(Configuration
config) throws IOException
}
}
+ /***
+ * When hadoop_version > 2.8.0, class DFSUtils has no method
getHaNnRpcAddresses(Configuration config)
+ * @param config
+ * @return
+ */
+ private static Set>>
getEntries(Configuration config) {
+return iDFSUtilClientExists ? invoke(config) :
+DFSUtil.getHaNnRpcAddresses(config).entrySet();
+ }
+
+ private static Set>>
invoke(Configuration config) {
--- End diff --
Please name this method with a more appropriate name, rather than a generic
name `invoke`.
> YarnUtils#cloneHaNnCredentials uses DFSUtil#getHaNnRpcAddresses, which is
> removed from DFSUtils from hadoop-2.8
>
>
> Key: TWILL-262
> URL: https://issues.apache.org/jira/browse/TWILL-262
> Project: Apache Twill
> Issue Type: Bug
> Components: yarn
>Affects Versions: 0.8.0, 0.9.0, 0.11.0, 0.12.0, 0.13.0
>Reporter: Hongyuan Li
>Priority: Major
> Attachments: errors.txt
>
>
> YarnUtils#cloneHaNnCredentials uses DFSUtil#getHaNnRpcAddresses, which is
> removed from DFSUtils from hadoop-2.8
> current code
> {code}
> public static void cloneHaNnCredentials(Configuration config) throws
> IOException {
> ……
> // Loop through all name services. Each name service could have multiple
> name node associated with it.
> for (Map.Entry> entry :
> DFSUtil.getHaNnRpcAddresses(config).entrySet()) {
> String nsId = entry.getKey();
> Map addressesInNN = entry.getValue();
> if (!HAUtil.isHAEnabled(config, nsId) || addressesInNN == null ||
> addressesInNN.isEmpty()) {
> continue;
> }
> ……
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (TWILL-262) YarnUtils#cloneHaNnCredentials uses DFSUtil#getHaNnRpcAddresses, which is removed from DFSUtils from hadoop-2.8
[
https://issues.apache.org/jira/browse/TWILL-262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16634234#comment-16634234
]
ASF GitHub Bot commented on TWILL-262:
--
Github user chtyim commented on a diff in the pull request:
https://github.com/apache/twill/pull/71#discussion_r221660462
--- Diff:
twill-yarn/src/main/java/org/apache/twill/internal/yarn/YarnUtils.java ---
@@ -202,6 +223,28 @@ public static void cloneHaNnCredentials(Configuration
config) throws IOException
}
}
+ /***
--- End diff --
Use two asterisks instead of three `/**`
> YarnUtils#cloneHaNnCredentials uses DFSUtil#getHaNnRpcAddresses, which is
> removed from DFSUtils from hadoop-2.8
>
>
> Key: TWILL-262
> URL: https://issues.apache.org/jira/browse/TWILL-262
> Project: Apache Twill
> Issue Type: Bug
> Components: yarn
>Affects Versions: 0.8.0, 0.9.0, 0.11.0, 0.12.0, 0.13.0
>Reporter: Hongyuan Li
>Priority: Major
> Attachments: errors.txt
>
>
> YarnUtils#cloneHaNnCredentials uses DFSUtil#getHaNnRpcAddresses, which is
> removed from DFSUtils from hadoop-2.8
> current code
> {code}
> public static void cloneHaNnCredentials(Configuration config) throws
> IOException {
> ……
> // Loop through all name services. Each name service could have multiple
> name node associated with it.
> for (Map.Entry> entry :
> DFSUtil.getHaNnRpcAddresses(config).entrySet()) {
> String nsId = entry.getKey();
> Map addressesInNN = entry.getValue();
> if (!HAUtil.isHAEnabled(config, nsId) || addressesInNN == null ||
> addressesInNN.isEmpty()) {
> continue;
> }
> ……
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (TWILL-262) YarnUtils#cloneHaNnCredentials uses DFSUtil#getHaNnRpcAddresses, which is removed from DFSUtils from hadoop-2.8
[
https://issues.apache.org/jira/browse/TWILL-262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16634238#comment-16634238
]
ASF GitHub Bot commented on TWILL-262:
--
Github user chtyim commented on a diff in the pull request:
https://github.com/apache/twill/pull/71#discussion_r221661663
--- Diff:
twill-yarn/src/main/java/org/apache/twill/internal/yarn/YarnUtils.java ---
@@ -345,6 +388,28 @@ private static FileSystem
getFileSystem(LocationFactory locationFactory) throws
return null;
}
+ private static void handleLogAction(Exception e) {
--- End diff --
This method is unnecessary, better log it in place. E.g. in the static
initializer above:
```java
static {
try {
Class dfsUtilsClientClazz =
Class.forName("org.apache.hadoop.hdfs.DFSUtilClient");
getHaNnRpcAddressesMethod =
dfsUtilsClientClazz.getMethod("getHaNnRpcAddresses",
Configuration.class);
hasDFSUtilClient = true;
} catch (ClassNotFoundException e) {
// Expected for Hadoop version < 2.8, hence log it as debug only to no
polluting the logs
LOG.debug("No DFSUtilClient found", e);
} catch (NoSuchMethodException e) {
// This is unexpected for not founding the getHaNnRpcAddresses method
if the DFSUtilClient class exists
LOG.warn("No DFSUtilClient.getHaNnRpcAddresses method found. Getting HA
NameNode address might fail.", e);
}
}
```
> YarnUtils#cloneHaNnCredentials uses DFSUtil#getHaNnRpcAddresses, which is
> removed from DFSUtils from hadoop-2.8
>
>
> Key: TWILL-262
> URL: https://issues.apache.org/jira/browse/TWILL-262
> Project: Apache Twill
> Issue Type: Bug
> Components: yarn
>Affects Versions: 0.8.0, 0.9.0, 0.11.0, 0.12.0, 0.13.0
>Reporter: Hongyuan Li
>Priority: Major
> Attachments: errors.txt
>
>
> YarnUtils#cloneHaNnCredentials uses DFSUtil#getHaNnRpcAddresses, which is
> removed from DFSUtils from hadoop-2.8
> current code
> {code}
> public static void cloneHaNnCredentials(Configuration config) throws
> IOException {
> ……
> // Loop through all name services. Each name service could have multiple
> name node associated with it.
> for (Map.Entry> entry :
> DFSUtil.getHaNnRpcAddresses(config).entrySet()) {
> String nsId = entry.getKey();
> Map addressesInNN = entry.getValue();
> if (!HAUtil.isHAEnabled(config, nsId) || addressesInNN == null ||
> addressesInNN.isEmpty()) {
> continue;
> }
> ……
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (TWILL-262) YarnUtils#cloneHaNnCredentials uses DFSUtil#getHaNnRpcAddresses, which is removed from DFSUtils from hadoop-2.8
[
https://issues.apache.org/jira/browse/TWILL-262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16634237#comment-16634237
]
ASF GitHub Bot commented on TWILL-262:
--
Github user chtyim commented on a diff in the pull request:
https://github.com/apache/twill/pull/71#discussion_r221663573
--- Diff:
twill-yarn/src/main/java/org/apache/twill/internal/yarn/YarnUtils.java ---
@@ -72,7 +77,23 @@
HADOOP_26
}
- private static final Logger LOG =
LoggerFactory.getLogger(YarnUtils.class);
+ private static boolean iDFSUtilClientExists = false; // use this to
judge if the hadoop version is above 2.8
--- End diff --
What does the `i` prefix mean? Better just call this `hasDFSUtilClient`.
> YarnUtils#cloneHaNnCredentials uses DFSUtil#getHaNnRpcAddresses, which is
> removed from DFSUtils from hadoop-2.8
>
>
> Key: TWILL-262
> URL: https://issues.apache.org/jira/browse/TWILL-262
> Project: Apache Twill
> Issue Type: Bug
> Components: yarn
>Affects Versions: 0.8.0, 0.9.0, 0.11.0, 0.12.0, 0.13.0
>Reporter: Hongyuan Li
>Priority: Major
> Attachments: errors.txt
>
>
> YarnUtils#cloneHaNnCredentials uses DFSUtil#getHaNnRpcAddresses, which is
> removed from DFSUtils from hadoop-2.8
> current code
> {code}
> public static void cloneHaNnCredentials(Configuration config) throws
> IOException {
> ……
> // Loop through all name services. Each name service could have multiple
> name node associated with it.
> for (Map.Entry> entry :
> DFSUtil.getHaNnRpcAddresses(config).entrySet()) {
> String nsId = entry.getKey();
> Map addressesInNN = entry.getValue();
> if (!HAUtil.isHAEnabled(config, nsId) || addressesInNN == null ||
> addressesInNN.isEmpty()) {
> continue;
> }
> ……
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (TWILL-262) YarnUtils#cloneHaNnCredentials uses DFSUtil#getHaNnRpcAddresses, which is removed from DFSUtils from hadoop-2.8
[
https://issues.apache.org/jira/browse/TWILL-262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16634239#comment-16634239
]
ASF GitHub Bot commented on TWILL-262:
--
Github user chtyim commented on a diff in the pull request:
https://github.com/apache/twill/pull/71#discussion_r221660324
--- Diff:
twill-yarn/src/main/java/org/apache/twill/internal/yarn/YarnUtils.java ---
@@ -202,6 +223,28 @@ public static void cloneHaNnCredentials(Configuration
config) throws IOException
}
}
+ /***
+ * When hadoop_version > 2.8.0, class DFSUtils has no method
getHaNnRpcAddresses(Configuration config)
+ * @param config
+ * @return
+ */
+ private static Set>>
getEntries(Configuration config) {
--- End diff --
Please name this method with a more specific name, such as
`getHaNnRpcAddress`.
> YarnUtils#cloneHaNnCredentials uses DFSUtil#getHaNnRpcAddresses, which is
> removed from DFSUtils from hadoop-2.8
>
>
> Key: TWILL-262
> URL: https://issues.apache.org/jira/browse/TWILL-262
> Project: Apache Twill
> Issue Type: Bug
> Components: yarn
>Affects Versions: 0.8.0, 0.9.0, 0.11.0, 0.12.0, 0.13.0
>Reporter: Hongyuan Li
>Priority: Major
> Attachments: errors.txt
>
>
> YarnUtils#cloneHaNnCredentials uses DFSUtil#getHaNnRpcAddresses, which is
> removed from DFSUtils from hadoop-2.8
> current code
> {code}
> public static void cloneHaNnCredentials(Configuration config) throws
> IOException {
> ……
> // Loop through all name services. Each name service could have multiple
> name node associated with it.
> for (Map.Entry> entry :
> DFSUtil.getHaNnRpcAddresses(config).entrySet()) {
> String nsId = entry.getKey();
> Map addressesInNN = entry.getValue();
> if (!HAUtil.isHAEnabled(config, nsId) || addressesInNN == null ||
> addressesInNN.isEmpty()) {
> continue;
> }
> ……
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (TWILL-262) YarnUtils#cloneHaNnCredentials uses DFSUtil#getHaNnRpcAddresses, which is removed from DFSUtils from hadoop-2.8
[
https://issues.apache.org/jira/browse/TWILL-262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16634236#comment-16634236
]
ASF GitHub Bot commented on TWILL-262:
--
Github user chtyim commented on a diff in the pull request:
https://github.com/apache/twill/pull/71#discussion_r221661290
--- Diff:
twill-yarn/src/main/java/org/apache/twill/internal/yarn/YarnUtils.java ---
@@ -202,6 +223,28 @@ public static void cloneHaNnCredentials(Configuration
config) throws IOException
}
}
+ /***
+ * When hadoop_version > 2.8.0, class DFSUtils has no method
getHaNnRpcAddresses(Configuration config)
+ * @param config
+ * @return
+ */
+ private static Set>>
getEntries(Configuration config) {
+return iDFSUtilClientExists ? invoke(config) :
+DFSUtil.getHaNnRpcAddresses(config).entrySet();
+ }
+
+ private static Set>>
invoke(Configuration config) {
+try {
+ return ((Map) getHaNnRpcAddressesMethod.invoke(null,
config)).entrySet();
--- End diff --
Why casting to `Map` while the return type of this method is `Set`??
> YarnUtils#cloneHaNnCredentials uses DFSUtil#getHaNnRpcAddresses, which is
> removed from DFSUtils from hadoop-2.8
>
>
> Key: TWILL-262
> URL: https://issues.apache.org/jira/browse/TWILL-262
> Project: Apache Twill
> Issue Type: Bug
> Components: yarn
>Affects Versions: 0.8.0, 0.9.0, 0.11.0, 0.12.0, 0.13.0
>Reporter: Hongyuan Li
>Priority: Major
> Attachments: errors.txt
>
>
> YarnUtils#cloneHaNnCredentials uses DFSUtil#getHaNnRpcAddresses, which is
> removed from DFSUtils from hadoop-2.8
> current code
> {code}
> public static void cloneHaNnCredentials(Configuration config) throws
> IOException {
> ……
> // Loop through all name services. Each name service could have multiple
> name node associated with it.
> for (Map.Entry> entry :
> DFSUtil.getHaNnRpcAddresses(config).entrySet()) {
> String nsId = entry.getKey();
> Map addressesInNN = entry.getValue();
> if (!HAUtil.isHAEnabled(config, nsId) || addressesInNN == null ||
> addressesInNN.isEmpty()) {
> continue;
> }
> ……
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (TWILL-262) YarnUtils#cloneHaNnCredentials uses DFSUtil#getHaNnRpcAddresses, which is removed from DFSUtils from hadoop-2.8
[
https://issues.apache.org/jira/browse/TWILL-262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16633305#comment-16633305
]
Hongyuan Li commented on TWILL-262:
---
seems other errorss follows.
2018-09-30T10:35:37,439Z ERROR o.a.t.i.y.AbstractYarnTwillService [node3]
[message-callback]
AbstractYarnTwillService:handleSecureStoreUpdate(AbstractYarnTwillService.java:110)
- Failed to update secure store.
java.lang.NoSuchMethodError:
org.apache.hadoop.hdfs.HAUtil.cloneDelegationTokenForLogicalUri(Lorg/apache/hadoop/security/UserGroupInformation;Ljava/net/URI;Ljava/util/Collection;)V
at
org.apache.twill.internal.yarn.YarnUtils.cloneHaNnCredentials(YarnUtils.java:222)
at
org.apache.twill.internal.yarn.AbstractYarnTwillService.handleSecureStoreUpdate(AbstractYarnTwillService.java:104)
at
org.apache.twill.internal.container.TwillContainerService.onReceived(TwillContainerService.java:107)
at
org.apache.twill.internal.AbstractTwillService.handleMessage(AbstractTwillService.java:333)
at
org.apache.twill.internal.AbstractTwillService.access$900(AbstractTwillService.java:85)
at
org.apache.twill.internal.AbstractTwillService$4.onSuccess(AbstractTwillService.java:284)
at
org.apache.twill.internal.AbstractTwillService$4.onSuccess(AbstractTwillService.java:264)
at com.google.common.util.concurrent.Futures$6.run(Futures.java:799)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
> YarnUtils#cloneHaNnCredentials uses DFSUtil#getHaNnRpcAddresses, which is
> removed from DFSUtils from hadoop-2.8
>
>
> Key: TWILL-262
> URL: https://issues.apache.org/jira/browse/TWILL-262
> Project: Apache Twill
> Issue Type: Bug
> Components: yarn
>Affects Versions: 0.8.0, 0.9.0, 0.11.0, 0.12.0, 0.13.0
>Reporter: Hongyuan Li
>Priority: Major
> Attachments: errors.txt
>
>
> YarnUtils#cloneHaNnCredentials uses DFSUtil#getHaNnRpcAddresses, which is
> removed from DFSUtils from hadoop-2.8
> current code
> {code}
> public static void cloneHaNnCredentials(Configuration config) throws
> IOException {
> ……
> // Loop through all name services. Each name service could have multiple
> name node associated with it.
> for (Map.Entry> entry :
> DFSUtil.getHaNnRpcAddresses(config).entrySet()) {
> String nsId = entry.getKey();
> Map addressesInNN = entry.getValue();
> if (!HAUtil.isHAEnabled(config, nsId) || addressesInNN == null ||
> addressesInNN.isEmpty()) {
> continue;
> }
> ……
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (TWILL-262) YarnUtils#cloneHaNnCredentials uses DFSUtil#getHaNnRpcAddresses, which is removed from DFSUtils from hadoop-2.8
[
https://issues.apache.org/jira/browse/TWILL-262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16632853#comment-16632853
]
Hongyuan Li commented on TWILL-262:
---
add errors information as attachment
> YarnUtils#cloneHaNnCredentials uses DFSUtil#getHaNnRpcAddresses, which is
> removed from DFSUtils from hadoop-2.8
>
>
> Key: TWILL-262
> URL: https://issues.apache.org/jira/browse/TWILL-262
> Project: Apache Twill
> Issue Type: Bug
> Components: yarn
>Affects Versions: 0.8.0, 0.9.0, 0.11.0, 0.12.0, 0.13.0
>Reporter: Hongyuan Li
>Priority: Major
> Attachments: errors.txt
>
>
> YarnUtils#cloneHaNnCredentials uses DFSUtil#getHaNnRpcAddresses, which is
> removed from DFSUtils from hadoop-2.8
> current code
> {code}
> public static void cloneHaNnCredentials(Configuration config) throws
> IOException {
> ……
> // Loop through all name services. Each name service could have multiple
> name node associated with it.
> for (Map.Entry> entry :
> DFSUtil.getHaNnRpcAddresses(config).entrySet()) {
> String nsId = entry.getKey();
> Map addressesInNN = entry.getValue();
> if (!HAUtil.isHAEnabled(config, nsId) || addressesInNN == null ||
> addressesInNN.isEmpty()) {
> continue;
> }
> ……
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (TWILL-262) YarnUtils#cloneHaNnCredentials uses DFSUtil#getHaNnRpcAddresses, which is removed from DFSUtils from hadoop-2.8
[
https://issues.apache.org/jira/browse/TWILL-262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16632850#comment-16632850
]
Hongyuan Li commented on TWILL-262:
---
[~chtyim], can you give me a code review ? seems build failure is not caused by
what i cmmited.
> YarnUtils#cloneHaNnCredentials uses DFSUtil#getHaNnRpcAddresses, which is
> removed from DFSUtils from hadoop-2.8
>
>
> Key: TWILL-262
> URL: https://issues.apache.org/jira/browse/TWILL-262
> Project: Apache Twill
> Issue Type: Bug
> Components: yarn
>Affects Versions: 0.8.0, 0.9.0, 0.11.0, 0.12.0, 0.13.0
>Reporter: Hongyuan Li
>Priority: Major
>
> YarnUtils#cloneHaNnCredentials uses DFSUtil#getHaNnRpcAddresses, which is
> removed from DFSUtils from hadoop-2.8
> current code
> {code}
> public static void cloneHaNnCredentials(Configuration config) throws
> IOException {
> ……
> // Loop through all name services. Each name service could have multiple
> name node associated with it.
> for (Map.Entry> entry :
> DFSUtil.getHaNnRpcAddresses(config).entrySet()) {
> String nsId = entry.getKey();
> Map addressesInNN = entry.getValue();
> if (!HAUtil.isHAEnabled(config, nsId) || addressesInNN == null ||
> addressesInNN.isEmpty()) {
> continue;
> }
> ……
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (TWILL-262) YarnUtils#cloneHaNnCredentials uses DFSUtil#getHaNnRpcAddresses, which is removed from DFSUtils from hadoop-2.8
[
https://issues.apache.org/jira/browse/TWILL-262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16632842#comment-16632842
]
ASF GitHub Bot commented on TWILL-262:
--
Github user piaosama commented on the issue:
https://github.com/apache/twill/pull/70
seems code style check failed,will try to open a new pull request
> YarnUtils#cloneHaNnCredentials uses DFSUtil#getHaNnRpcAddresses, which is
> removed from DFSUtils from hadoop-2.8
>
>
> Key: TWILL-262
> URL: https://issues.apache.org/jira/browse/TWILL-262
> Project: Apache Twill
> Issue Type: Bug
> Components: yarn
>Affects Versions: 0.8.0, 0.9.0, 0.11.0, 0.12.0, 0.13.0
>Reporter: Hongyuan Li
>Priority: Major
>
> YarnUtils#cloneHaNnCredentials uses DFSUtil#getHaNnRpcAddresses, which is
> removed from DFSUtils from hadoop-2.8
> current code
> {code}
> public static void cloneHaNnCredentials(Configuration config) throws
> IOException {
> ……
> // Loop through all name services. Each name service could have multiple
> name node associated with it.
> for (Map.Entry> entry :
> DFSUtil.getHaNnRpcAddresses(config).entrySet()) {
> String nsId = entry.getKey();
> Map addressesInNN = entry.getValue();
> if (!HAUtil.isHAEnabled(config, nsId) || addressesInNN == null ||
> addressesInNN.isEmpty()) {
> continue;
> }
> ……
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (TWILL-262) YarnUtils#cloneHaNnCredentials uses DFSUtil#getHaNnRpcAddresses, which is removed from DFSUtils from hadoop-2.8
[
https://issues.apache.org/jira/browse/TWILL-262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16632843#comment-16632843
]
ASF GitHub Bot commented on TWILL-262:
--
Github user piaosama closed the pull request at:
https://github.com/apache/twill/pull/70
> YarnUtils#cloneHaNnCredentials uses DFSUtil#getHaNnRpcAddresses, which is
> removed from DFSUtils from hadoop-2.8
>
>
> Key: TWILL-262
> URL: https://issues.apache.org/jira/browse/TWILL-262
> Project: Apache Twill
> Issue Type: Bug
> Components: yarn
>Affects Versions: 0.8.0, 0.9.0, 0.11.0, 0.12.0, 0.13.0
>Reporter: Hongyuan Li
>Priority: Major
>
> YarnUtils#cloneHaNnCredentials uses DFSUtil#getHaNnRpcAddresses, which is
> removed from DFSUtils from hadoop-2.8
> current code
> {code}
> public static void cloneHaNnCredentials(Configuration config) throws
> IOException {
> ……
> // Loop through all name services. Each name service could have multiple
> name node associated with it.
> for (Map.Entry> entry :
> DFSUtil.getHaNnRpcAddresses(config).entrySet()) {
> String nsId = entry.getKey();
> Map addressesInNN = entry.getValue();
> if (!HAUtil.isHAEnabled(config, nsId) || addressesInNN == null ||
> addressesInNN.isEmpty()) {
> continue;
> }
> ……
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (TWILL-262) YarnUtils#cloneHaNnCredentials uses DFSUtil#getHaNnRpcAddresses, which is removed from DFSUtils from hadoop-2.8
[
https://issues.apache.org/jira/browse/TWILL-262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16632840#comment-16632840
]
ASF GitHub Bot commented on TWILL-262:
--
GitHub user piaosama opened a pull request:
https://github.com/apache/twill/pull/70
TWILL-262 YarnUtils#cloneHaNnCredentials uses DFSUtil#getHaNnRpcAddresses,
which is removed from DFSUtils from hadoop-2.8
TWILL-262 YarnUtils#cloneHaNnCredentials uses DFSUtil#getHaNnRpcAddresses,
which is removed from DFSUtils from hadoop-2.8
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/piaosama/twill master
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/twill/pull/70.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #70
commit 892fdc172522aa67d1581b324503fddc912e839c
Author: lihongyuan
Date: 2018-09-29T06:33:05Z
TWILL-262 YarnUtils#cloneHaNnCredentials uses DFSUtil#getHaNnRpcAddresses,
which is removed from DFSUtils from hadoop-2.8
> YarnUtils#cloneHaNnCredentials uses DFSUtil#getHaNnRpcAddresses, which is
> removed from DFSUtils from hadoop-2.8
>
>
> Key: TWILL-262
> URL: https://issues.apache.org/jira/browse/TWILL-262
> Project: Apache Twill
> Issue Type: Bug
> Components: yarn
>Affects Versions: 0.8.0, 0.9.0, 0.11.0, 0.12.0, 0.13.0
>Reporter: Hongyuan Li
>Priority: Major
>
> YarnUtils#cloneHaNnCredentials uses DFSUtil#getHaNnRpcAddresses, which is
> removed from DFSUtils from hadoop-2.8
> current code
> {code}
> public static void cloneHaNnCredentials(Configuration config) throws
> IOException {
> ……
> // Loop through all name services. Each name service could have multiple
> name node associated with it.
> for (Map.Entry> entry :
> DFSUtil.getHaNnRpcAddresses(config).entrySet()) {
> String nsId = entry.getKey();
> Map addressesInNN = entry.getValue();
> if (!HAUtil.isHAEnabled(config, nsId) || addressesInNN == null ||
> addressesInNN.isEmpty()) {
> continue;
> }
> ……
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
