[jira] [Commented] (TWILL-262) YarnUtils#cloneHaNnCredentials uses DFSUtil#getHaNnRpcAddresses, which is removed from DFSUtils from hadoop-2.8
[
https://issues.apache.org/jira/browse/TWILL-262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16640720#comment-16640720
]
ASF GitHub Bot commented on TWILL-262:
--
Github user piaosama commented on a diff in the pull request:
https://github.com/apache/twill/pull/71#discussion_r223182267
--- Diff:
twill-yarn/src/main/java/org/apache/twill/internal/yarn/YarnUtils.java ---
@@ -345,6 +388,28 @@ private static FileSystem
getFileSystem(LocationFactory locationFactory) throws
return null;
}
+ private static void handleLogAction(Exception e) {
--- End diff --
add new code, modify the code as you suggested and add the HAutils method
judgement.
> YarnUtils#cloneHaNnCredentials uses DFSUtil#getHaNnRpcAddresses, which is
> removed from DFSUtils from hadoop-2.8
>
>
> Key: TWILL-262
> URL: https://issues.apache.org/jira/browse/TWILL-262
> Project: Apache Twill
> Issue Type: Bug
> Components: yarn
>Affects Versions: 0.8.0, 0.9.0, 0.11.0, 0.12.0, 0.13.0
>Reporter: Hongyuan Li
>Priority: Major
> Attachments: errors.txt
>
>
> YarnUtils#cloneHaNnCredentials uses DFSUtil#getHaNnRpcAddresses, which is
> removed from DFSUtils from hadoop-2.8
> current code
> {code}
> public static void cloneHaNnCredentials(Configuration config) throws
> IOException {
> ……
> // Loop through all name services. Each name service could have multiple
> name node associated with it.
> for (Map.Entry> entry :
> DFSUtil.getHaNnRpcAddresses(config).entrySet()) {
> String nsId = entry.getKey();
> Map addressesInNN = entry.getValue();
> if (!HAUtil.isHAEnabled(config, nsId) || addressesInNN == null ||
> addressesInNN.isEmpty()) {
> continue;
> }
> ……
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (TWILL-262) YarnUtils#cloneHaNnCredentials uses DFSUtil#getHaNnRpcAddresses, which is removed from DFSUtils from hadoop-2.8
[
https://issues.apache.org/jira/browse/TWILL-262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16640721#comment-16640721
]
ASF GitHub Bot commented on TWILL-262:
--
Github user piaosama commented on the issue:
https://github.com/apache/twill/pull/71
add new code, modify the code as you suggested and add the HAutils method
judgement.
> YarnUtils#cloneHaNnCredentials uses DFSUtil#getHaNnRpcAddresses, which is
> removed from DFSUtils from hadoop-2.8
>
>
> Key: TWILL-262
> URL: https://issues.apache.org/jira/browse/TWILL-262
> Project: Apache Twill
> Issue Type: Bug
> Components: yarn
>Affects Versions: 0.8.0, 0.9.0, 0.11.0, 0.12.0, 0.13.0
>Reporter: Hongyuan Li
>Priority: Major
> Attachments: errors.txt
>
>
> YarnUtils#cloneHaNnCredentials uses DFSUtil#getHaNnRpcAddresses, which is
> removed from DFSUtils from hadoop-2.8
> current code
> {code}
> public static void cloneHaNnCredentials(Configuration config) throws
> IOException {
> ……
> // Loop through all name services. Each name service could have multiple
> name node associated with it.
> for (Map.Entry> entry :
> DFSUtil.getHaNnRpcAddresses(config).entrySet()) {
> String nsId = entry.getKey();
> Map addressesInNN = entry.getValue();
> if (!HAUtil.isHAEnabled(config, nsId) || addressesInNN == null ||
> addressesInNN.isEmpty()) {
> continue;
> }
> ……
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] twill issue #71: TWILL-262 YarnUtils#cloneHaNnCredentials uses DFSUtil#getHa...
Github user piaosama commented on the issue: https://github.com/apache/twill/pull/71 add new code, modify the code as you suggested and add the HAutils method judgement. ---
[GitHub] twill pull request #71: TWILL-262 YarnUtils#cloneHaNnCredentials uses DFSUti...
Github user piaosama commented on a diff in the pull request:
https://github.com/apache/twill/pull/71#discussion_r223182267
--- Diff:
twill-yarn/src/main/java/org/apache/twill/internal/yarn/YarnUtils.java ---
@@ -345,6 +388,28 @@ private static FileSystem
getFileSystem(LocationFactory locationFactory) throws
return null;
}
+ private static void handleLogAction(Exception e) {
--- End diff --
add new code, modify the code as you suggested and add the HAutils method
judgement.
---
[jira] [Commented] (TWILL-262) YarnUtils#cloneHaNnCredentials uses DFSUtil#getHaNnRpcAddresses, which is removed from DFSUtils from hadoop-2.8
[
https://issues.apache.org/jira/browse/TWILL-262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16640715#comment-16640715
]
ASF GitHub Bot commented on TWILL-262:
--
Github user piaosama commented on a diff in the pull request:
https://github.com/apache/twill/pull/71#discussion_r223181503
--- Diff:
twill-yarn/src/main/java/org/apache/twill/internal/yarn/YarnUtils.java ---
@@ -202,6 +223,28 @@ public static void cloneHaNnCredentials(Configuration
config) throws IOException
}
}
+ /***
+ * When hadoop_version > 2.8.0, class DFSUtils has no method
getHaNnRpcAddresses(Configuration config)
+ * @param config
+ * @return
+ */
+ private static Set>>
getEntries(Configuration config) {
--- End diff --
ok
> YarnUtils#cloneHaNnCredentials uses DFSUtil#getHaNnRpcAddresses, which is
> removed from DFSUtils from hadoop-2.8
>
>
> Key: TWILL-262
> URL: https://issues.apache.org/jira/browse/TWILL-262
> Project: Apache Twill
> Issue Type: Bug
> Components: yarn
>Affects Versions: 0.8.0, 0.9.0, 0.11.0, 0.12.0, 0.13.0
>Reporter: Hongyuan Li
>Priority: Major
> Attachments: errors.txt
>
>
> YarnUtils#cloneHaNnCredentials uses DFSUtil#getHaNnRpcAddresses, which is
> removed from DFSUtils from hadoop-2.8
> current code
> {code}
> public static void cloneHaNnCredentials(Configuration config) throws
> IOException {
> ……
> // Loop through all name services. Each name service could have multiple
> name node associated with it.
> for (Map.Entry> entry :
> DFSUtil.getHaNnRpcAddresses(config).entrySet()) {
> String nsId = entry.getKey();
> Map addressesInNN = entry.getValue();
> if (!HAUtil.isHAEnabled(config, nsId) || addressesInNN == null ||
> addressesInNN.isEmpty()) {
> continue;
> }
> ……
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (TWILL-262) YarnUtils#cloneHaNnCredentials uses DFSUtil#getHaNnRpcAddresses, which is removed from DFSUtils from hadoop-2.8
[
https://issues.apache.org/jira/browse/TWILL-262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16640714#comment-16640714
]
ASF GitHub Bot commented on TWILL-262:
--
Github user piaosama commented on a diff in the pull request:
https://github.com/apache/twill/pull/71#discussion_r223181498
--- Diff:
twill-yarn/src/main/java/org/apache/twill/internal/yarn/YarnUtils.java ---
@@ -202,6 +223,28 @@ public static void cloneHaNnCredentials(Configuration
config) throws IOException
}
}
+ /***
+ * When hadoop_version > 2.8.0, class DFSUtils has no method
getHaNnRpcAddresses(Configuration config)
+ * @param config
+ * @return
+ */
+ private static Set>>
getEntries(Configuration config) {
+return iDFSUtilClientExists ? invoke(config) :
+DFSUtil.getHaNnRpcAddresses(config).entrySet();
+ }
+
+ private static Set>>
invoke(Configuration config) {
--- End diff --
ok
> YarnUtils#cloneHaNnCredentials uses DFSUtil#getHaNnRpcAddresses, which is
> removed from DFSUtils from hadoop-2.8
>
>
> Key: TWILL-262
> URL: https://issues.apache.org/jira/browse/TWILL-262
> Project: Apache Twill
> Issue Type: Bug
> Components: yarn
>Affects Versions: 0.8.0, 0.9.0, 0.11.0, 0.12.0, 0.13.0
>Reporter: Hongyuan Li
>Priority: Major
> Attachments: errors.txt
>
>
> YarnUtils#cloneHaNnCredentials uses DFSUtil#getHaNnRpcAddresses, which is
> removed from DFSUtils from hadoop-2.8
> current code
> {code}
> public static void cloneHaNnCredentials(Configuration config) throws
> IOException {
> ……
> // Loop through all name services. Each name service could have multiple
> name node associated with it.
> for (Map.Entry> entry :
> DFSUtil.getHaNnRpcAddresses(config).entrySet()) {
> String nsId = entry.getKey();
> Map addressesInNN = entry.getValue();
> if (!HAUtil.isHAEnabled(config, nsId) || addressesInNN == null ||
> addressesInNN.isEmpty()) {
> continue;
> }
> ……
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] twill pull request #71: TWILL-262 YarnUtils#cloneHaNnCredentials uses DFSUti...
Github user piaosama commented on a diff in the pull request:
https://github.com/apache/twill/pull/71#discussion_r223181503
--- Diff:
twill-yarn/src/main/java/org/apache/twill/internal/yarn/YarnUtils.java ---
@@ -202,6 +223,28 @@ public static void cloneHaNnCredentials(Configuration
config) throws IOException
}
}
+ /***
+ * When hadoop_version > 2.8.0, class DFSUtils has no method
getHaNnRpcAddresses(Configuration config)
+ * @param config
+ * @return
+ */
+ private static Set>>
getEntries(Configuration config) {
--- End diff --
ok
---
[jira] [Commented] (TWILL-262) YarnUtils#cloneHaNnCredentials uses DFSUtil#getHaNnRpcAddresses, which is removed from DFSUtils from hadoop-2.8
[
https://issues.apache.org/jira/browse/TWILL-262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16640713#comment-16640713
]
ASF GitHub Bot commented on TWILL-262:
--
Github user piaosama commented on a diff in the pull request:
https://github.com/apache/twill/pull/71#discussion_r223181489
--- Diff:
twill-yarn/src/main/java/org/apache/twill/internal/yarn/YarnUtils.java ---
@@ -202,6 +223,28 @@ public static void cloneHaNnCredentials(Configuration
config) throws IOException
}
}
+ /***
+ * When hadoop_version > 2.8.0, class DFSUtils has no method
getHaNnRpcAddresses(Configuration config)
+ * @param config
+ * @return
+ */
+ private static Set>>
getEntries(Configuration config) {
+return iDFSUtilClientExists ? invoke(config) :
+DFSUtil.getHaNnRpcAddresses(config).entrySet();
+ }
+
+ private static Set>>
invoke(Configuration config) {
+try {
+ return ((Map) getHaNnRpcAddressesMethod.invoke(null,
config)).entrySet();
--- End diff --
getHaNnRpcAddressesMethod invoke return Map instance.
> YarnUtils#cloneHaNnCredentials uses DFSUtil#getHaNnRpcAddresses, which is
> removed from DFSUtils from hadoop-2.8
>
>
> Key: TWILL-262
> URL: https://issues.apache.org/jira/browse/TWILL-262
> Project: Apache Twill
> Issue Type: Bug
> Components: yarn
>Affects Versions: 0.8.0, 0.9.0, 0.11.0, 0.12.0, 0.13.0
>Reporter: Hongyuan Li
>Priority: Major
> Attachments: errors.txt
>
>
> YarnUtils#cloneHaNnCredentials uses DFSUtil#getHaNnRpcAddresses, which is
> removed from DFSUtils from hadoop-2.8
> current code
> {code}
> public static void cloneHaNnCredentials(Configuration config) throws
> IOException {
> ……
> // Loop through all name services. Each name service could have multiple
> name node associated with it.
> for (Map.Entry> entry :
> DFSUtil.getHaNnRpcAddresses(config).entrySet()) {
> String nsId = entry.getKey();
> Map addressesInNN = entry.getValue();
> if (!HAUtil.isHAEnabled(config, nsId) || addressesInNN == null ||
> addressesInNN.isEmpty()) {
> continue;
> }
> ……
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] twill pull request #71: TWILL-262 YarnUtils#cloneHaNnCredentials uses DFSUti...
Github user piaosama commented on a diff in the pull request:
https://github.com/apache/twill/pull/71#discussion_r223181498
--- Diff:
twill-yarn/src/main/java/org/apache/twill/internal/yarn/YarnUtils.java ---
@@ -202,6 +223,28 @@ public static void cloneHaNnCredentials(Configuration
config) throws IOException
}
}
+ /***
+ * When hadoop_version > 2.8.0, class DFSUtils has no method
getHaNnRpcAddresses(Configuration config)
+ * @param config
+ * @return
+ */
+ private static Set>>
getEntries(Configuration config) {
+return iDFSUtilClientExists ? invoke(config) :
+DFSUtil.getHaNnRpcAddresses(config).entrySet();
+ }
+
+ private static Set>>
invoke(Configuration config) {
--- End diff --
ok
---
[GitHub] twill pull request #71: TWILL-262 YarnUtils#cloneHaNnCredentials uses DFSUti...
Github user piaosama commented on a diff in the pull request:
https://github.com/apache/twill/pull/71#discussion_r223181489
--- Diff:
twill-yarn/src/main/java/org/apache/twill/internal/yarn/YarnUtils.java ---
@@ -202,6 +223,28 @@ public static void cloneHaNnCredentials(Configuration
config) throws IOException
}
}
+ /***
+ * When hadoop_version > 2.8.0, class DFSUtils has no method
getHaNnRpcAddresses(Configuration config)
+ * @param config
+ * @return
+ */
+ private static Set>>
getEntries(Configuration config) {
+return iDFSUtilClientExists ? invoke(config) :
+DFSUtil.getHaNnRpcAddresses(config).entrySet();
+ }
+
+ private static Set>>
invoke(Configuration config) {
+try {
+ return ((Map) getHaNnRpcAddressesMethod.invoke(null,
config)).entrySet();
--- End diff --
getHaNnRpcAddressesMethod invoke return Map instance.
---
[jira] [Commented] (TWILL-262) YarnUtils#cloneHaNnCredentials uses DFSUtil#getHaNnRpcAddresses, which is removed from DFSUtils from hadoop-2.8
[
https://issues.apache.org/jira/browse/TWILL-262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16640708#comment-16640708
]
ASF GitHub Bot commented on TWILL-262:
--
Github user piaosama commented on a diff in the pull request:
https://github.com/apache/twill/pull/71#discussion_r223181337
--- Diff:
twill-yarn/src/main/java/org/apache/twill/internal/yarn/YarnUtils.java ---
@@ -72,7 +77,23 @@
HADOOP_26
}
- private static final Logger LOG =
LoggerFactory.getLogger(YarnUtils.class);
+ private static boolean iDFSUtilClientExists = false; // use this to
judge if the hadoop version is above 2.8
--- End diff --
ok， suddenly find HAUtil class does not exists as the DFSUtils class. i
will add the two judge into the code and make the code more nice as you
commented.
> YarnUtils#cloneHaNnCredentials uses DFSUtil#getHaNnRpcAddresses, which is
> removed from DFSUtils from hadoop-2.8
>
>
> Key: TWILL-262
> URL: https://issues.apache.org/jira/browse/TWILL-262
> Project: Apache Twill
> Issue Type: Bug
> Components: yarn
>Affects Versions: 0.8.0, 0.9.0, 0.11.0, 0.12.0, 0.13.0
>Reporter: Hongyuan Li
>Priority: Major
> Attachments: errors.txt
>
>
> YarnUtils#cloneHaNnCredentials uses DFSUtil#getHaNnRpcAddresses, which is
> removed from DFSUtils from hadoop-2.8
> current code
> {code}
> public static void cloneHaNnCredentials(Configuration config) throws
> IOException {
> ……
> // Loop through all name services. Each name service could have multiple
> name node associated with it.
> for (Map.Entry> entry :
> DFSUtil.getHaNnRpcAddresses(config).entrySet()) {
> String nsId = entry.getKey();
> Map addressesInNN = entry.getValue();
> if (!HAUtil.isHAEnabled(config, nsId) || addressesInNN == null ||
> addressesInNN.isEmpty()) {
> continue;
> }
> ……
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] twill pull request #71: TWILL-262 YarnUtils#cloneHaNnCredentials uses DFSUti...
Github user piaosama commented on a diff in the pull request: https://github.com/apache/twill/pull/71#discussion_r223181337 --- Diff: twill-yarn/src/main/java/org/apache/twill/internal/yarn/YarnUtils.java --- @@ -72,7 +77,23 @@ HADOOP_26 } - private static final Logger LOG = LoggerFactory.getLogger(YarnUtils.class); + private static boolean iDFSUtilClientExists = false; // use this to judge if the hadoop version is above 2.8 --- End diff -- okï¼ suddenly find HAUtil class does not exists as the DFSUtils class. i will add the two judge into the code and make the code more nice as you commented. ---
