[jira] [Commented] (YARN-4496) Improve HA ResourceManager Failover detection on the client
[
https://issues.apache.org/jira/browse/YARN-4496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15113476#comment-15113476
]
Xuan Gong commented on YARN-4496:
-
+1 lgtm. Checking this in
> Improve HA ResourceManager Failover detection on the client
> ---
>
> Key: YARN-4496
> URL: https://issues.apache.org/jira/browse/YARN-4496
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: client, resourcemanager
>Reporter: Arun Suresh
>Assignee: Jian He
> Attachments: YARN-4496.1.patch, YARN-4496.2.patch, YARN-4496.3.patch,
> YARN-4496.4.patch, YARN-4496.5.patch, YARN-4496.6.patch
>
>
> HDFS deployments can currently use the {{RequestHedgingProxyProvider}} to
> improve Namenode failover detection in the client. It does this by
> concurrently trying all namenodes and picks the namenode that returns the
> fastest with a successful response as the active node.
> It would be useful to have a similar ProxyProvider for the Yarn RM (it can
> possibly be done by converging some the class hierarchies to use the same
> ProxyProvider)
> This would especially be useful for large YARN deployments with multiple
> standby RMs where clients will be able to pick the active RM without having
> to traverse a list of configured RMs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4496) Improve HA ResourceManager Failover detection on the client
[
https://issues.apache.org/jira/browse/YARN-4496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15113481#comment-15113481
]
Xuan Gong commented on YARN-4496:
-
Committed into trunk/branch-2. Thanks, Jian for working on this. And Thanks,
Arun for the review.
> Improve HA ResourceManager Failover detection on the client
> ---
>
> Key: YARN-4496
> URL: https://issues.apache.org/jira/browse/YARN-4496
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: client, resourcemanager
>Reporter: Arun Suresh
>Assignee: Jian He
> Fix For: 2.9.0
>
> Attachments: YARN-4496.1.patch, YARN-4496.2.patch, YARN-4496.3.patch,
> YARN-4496.4.patch, YARN-4496.5.patch, YARN-4496.6.patch
>
>
> HDFS deployments can currently use the {{RequestHedgingProxyProvider}} to
> improve Namenode failover detection in the client. It does this by
> concurrently trying all namenodes and picks the namenode that returns the
> fastest with a successful response as the active node.
> It would be useful to have a similar ProxyProvider for the Yarn RM (it can
> possibly be done by converging some the class hierarchies to use the same
> ProxyProvider)
> This would especially be useful for large YARN deployments with multiple
> standby RMs where clients will be able to pick the active RM without having
> to traverse a list of configured RMs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4496) Improve HA ResourceManager Failover detection on the client
[
https://issues.apache.org/jira/browse/YARN-4496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15113495#comment-15113495
]
Hudson commented on YARN-4496:
--
FAILURE: Integrated in Hadoop-trunk-Commit #9172 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/9172/])
YARN-4496. Improve HA ResourceManager Failover detection on the client. (xgong:
rev 618bfd6ac2a5b62d39e9bed80f75362bafc0ef28)
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/src/test/java/org/apache/hadoop/yarn/server/MiniYARNCluster.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestNodeStatusUpdater.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/client/ConfiguredRMFailoverProxyProvider.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/client/RequestHedgingRMFailoverProxyProvider.java
* hadoop-yarn-project/CHANGES.txt
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/TestHedgingRequestRMFailoverProxyProvider.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/client/RMProxy.java
> Improve HA ResourceManager Failover detection on the client
> ---
>
> Key: YARN-4496
> URL: https://issues.apache.org/jira/browse/YARN-4496
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: client, resourcemanager
>Reporter: Arun Suresh
>Assignee: Jian He
> Fix For: 2.9.0
>
> Attachments: YARN-4496.1.patch, YARN-4496.2.patch, YARN-4496.3.patch,
> YARN-4496.4.patch, YARN-4496.5.patch, YARN-4496.6.patch
>
>
> HDFS deployments can currently use the {{RequestHedgingProxyProvider}} to
> improve Namenode failover detection in the client. It does this by
> concurrently trying all namenodes and picks the namenode that returns the
> fastest with a successful response as the active node.
> It would be useful to have a similar ProxyProvider for the Yarn RM (it can
> possibly be done by converging some the class hierarchies to use the same
> ProxyProvider)
> This would especially be useful for large YARN deployments with multiple
> standby RMs where clients will be able to pick the active RM without having
> to traverse a list of configured RMs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4496) Improve HA ResourceManager Failover detection on the client
[
https://issues.apache.org/jira/browse/YARN-4496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15110464#comment-15110464
]
Hadoop QA commented on YARN-4496:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 59s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
39s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 47s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 7s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
27s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 47s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
52s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
15s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 11s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 28s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 33s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
24s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 45s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 3m 43s {color}
| {color:red} hadoop-yarn-project_hadoop-yarn-jdk1.8.0_66 with JDK v1.8.0_66
generated 1 new + 9 unchanged - 0 fixed = 10 total (was 9) {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 45s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 4s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 5m 48s {color}
| {color:red} hadoop-yarn-project_hadoop-yarn-jdk1.7.0_91 with JDK v1.7.0_91
generated 1 new + 10 unchanged - 0 fixed = 11 total (was 10) {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 4s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 27s
{color} | {color:red} hadoop-yarn-project/hadoop-yarn: patch generated 3 new +
11 unchanged - 0 fixed = 14 total (was 11) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 38s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
47s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
56s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 6s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 21s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 53s
{color} | {color:green} hadoop-yarn-common in the patch passed with JDK
v1.8.0_66. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 8m 33s {color}
| {color:red} hadoop-yarn-server-nodemanager in the patch failed with JDK
v1.8.0_66. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 6m 36s {color}
| {color:red} hadoop-yarn-server-tests in
[jira] [Commented] (YARN-4496) Improve HA ResourceManager Failover detection on the client
[
https://issues.apache.org/jira/browse/YARN-4496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15111577#comment-15111577
]
Hadoop QA commented on YARN-4496:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 48s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
47s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 56s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 8s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
28s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 44s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
53s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
11s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 12s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 27s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 33s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
23s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 41s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 3m 37s {color}
| {color:red} hadoop-yarn-project_hadoop-yarn-jdk1.8.0_66 with JDK v1.8.0_66
generated 1 new + 9 unchanged - 0 fixed = 10 total (was 9) {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 41s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 2s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 5m 39s {color}
| {color:red} hadoop-yarn-project_hadoop-yarn-jdk1.7.0_91 with JDK v1.7.0_91
generated 1 new + 10 unchanged - 0 fixed = 11 total (was 10) {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 2s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 26s
{color} | {color:red} hadoop-yarn-project/hadoop-yarn: patch generated 3 new +
11 unchanged - 0 fixed = 14 total (was 11) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 36s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
45s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
51s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 4s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 20s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 50s
{color} | {color:green} hadoop-yarn-common in the patch passed with JDK
v1.8.0_66. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 8m 21s {color}
| {color:red} hadoop-yarn-server-nodemanager in the patch failed with JDK
v1.8.0_66. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 6m 36s {color}
| {color:red} hadoop-yarn-server-tests in
[jira] [Commented] (YARN-4496) Improve HA ResourceManager Failover detection on the client
[
https://issues.apache.org/jira/browse/YARN-4496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15111907#comment-15111907
]
Hadoop QA commented on YARN-4496:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 45s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
15s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 9s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 25s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
31s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 52s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
54s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
31s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 20s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 34s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 35s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
35s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 8s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 8s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 25s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 25s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 32s
{color} | {color:red} hadoop-yarn-project/hadoop-yarn: patch generated 3 new +
11 unchanged - 0 fixed = 14 total (was 11) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 50s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
51s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
12s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 15s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 24s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 3s
{color} | {color:green} hadoop-yarn-common in the patch passed with JDK
v1.8.0_66. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 8m 45s {color}
| {color:red} hadoop-yarn-server-nodemanager in the patch failed with JDK
v1.8.0_66. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 6m 38s {color}
| {color:red} hadoop-yarn-server-tests in the patch failed with JDK v1.8.0_66.
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 64m 38s {color}
| {color:red} hadoop-yarn-client in the patch failed with JDK v1.8.0_66.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 9s
{color} | {color:green} hadoop-yarn-common in the patch passed with JDK
v1.7.0_91. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 8m 56s
[jira] [Commented] (YARN-4496) Improve HA ResourceManager Failover detection on the client
[
https://issues.apache.org/jira/browse/YARN-4496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15109235#comment-15109235
]
Jian He commented on YARN-4496:
---
yeah, I actually tried to make them common when I started writing the patch
initially. but because of some other logic divergence, making them common
requires nontrival refactoring on RM and maybe hdfs too. I'd like to keep them
as is and open a jira separately
> Improve HA ResourceManager Failover detection on the client
> ---
>
> Key: YARN-4496
> URL: https://issues.apache.org/jira/browse/YARN-4496
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: client, resourcemanager
>Reporter: Arun Suresh
>Assignee: Jian He
> Attachments: YARN-4496.1.patch, YARN-4496.2.patch
>
>
> HDFS deployments can currently use the {{RequestHedgingProxyProvider}} to
> improve Namenode failover detection in the client. It does this by
> concurrently trying all namenodes and picks the namenode that returns the
> fastest with a successful response as the active node.
> It would be useful to have a similar ProxyProvider for the Yarn RM (it can
> possibly be done by converging some the class hierarchies to use the same
> ProxyProvider)
> This would especially be useful for large YARN deployments with multiple
> standby RMs where clients will be able to pick the active RM without having
> to traverse a list of configured RMs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4496) Improve HA ResourceManager Failover detection on the client
[
https://issues.apache.org/jira/browse/YARN-4496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15109212#comment-15109212
]
Arun Suresh commented on YARN-4496:
---
Thanks for the patch [~jianhe] !!
It looks good generally, although, my initial thought process was to actually
merge both the Namenode and RM proxy hierarchies. Currently, it looks like the
common ancestor is the {{FailoverProxyProvider}} class. Unfortunately, for
YARN, we have a {{RMFailoverProxyProvider}} which extends it and adds an
{{init()}} method, which in the NN hierarchy, everything is done in the
constructor. I was wondering if we could possibly avoid using the init method
and somehow make {{ConfiguredFailoverProxyProvider}} and
{{RequestHedgingProxyProvider}} common enough so that it can be used by both RM
and NN.. and avoid some code duplication too.
Thoughts ?
> Improve HA ResourceManager Failover detection on the client
> ---
>
> Key: YARN-4496
> URL: https://issues.apache.org/jira/browse/YARN-4496
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: client, resourcemanager
>Reporter: Arun Suresh
>Assignee: Jian He
> Attachments: YARN-4496.1.patch, YARN-4496.2.patch
>
>
> HDFS deployments can currently use the {{RequestHedgingProxyProvider}} to
> improve Namenode failover detection in the client. It does this by
> concurrently trying all namenodes and picks the namenode that returns the
> fastest with a successful response as the active node.
> It would be useful to have a similar ProxyProvider for the Yarn RM (it can
> possibly be done by converging some the class hierarchies to use the same
> ProxyProvider)
> This would especially be useful for large YARN deployments with multiple
> standby RMs where clients will be able to pick the active RM without having
> to traverse a list of configured RMs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4496) Improve HA ResourceManager Failover detection on the client
[
https://issues.apache.org/jira/browse/YARN-4496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15109286#comment-15109286
]
Arun Suresh commented on YARN-4496:
---
Agreed.. Couple of minor comments:
# I noticed that you had changed the {{RMProxy.createRetryPolicy}} method to
allow setting a boolean flag for HA. Do we need that change ? since I see that
you are calling {{HAUtils.isHAEnabled()}} whenever you are calling
{{createRetryPolicy}} anyway.
# Can we kill the RM threads before exiting the test case ?
minor nit :
# the test case looks good, but am worried it might be a bit flaky due to
reliance on the clock and timing.. anyway it can be refactored ?
> Improve HA ResourceManager Failover detection on the client
> ---
>
> Key: YARN-4496
> URL: https://issues.apache.org/jira/browse/YARN-4496
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: client, resourcemanager
>Reporter: Arun Suresh
>Assignee: Jian He
> Attachments: YARN-4496.1.patch, YARN-4496.2.patch
>
>
> HDFS deployments can currently use the {{RequestHedgingProxyProvider}} to
> improve Namenode failover detection in the client. It does this by
> concurrently trying all namenodes and picks the namenode that returns the
> fastest with a successful response as the active node.
> It would be useful to have a similar ProxyProvider for the Yarn RM (it can
> possibly be done by converging some the class hierarchies to use the same
> ProxyProvider)
> This would especially be useful for large YARN deployments with multiple
> standby RMs where clients will be able to pick the active RM without having
> to traverse a list of configured RMs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4496) Improve HA ResourceManager Failover detection on the client
[
https://issues.apache.org/jira/browse/YARN-4496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15109416#comment-15109416
]
Jian He commented on YARN-4496:
---
bq. Do we need that change ?
This method actually uses this flag
{code}
public static RetryPolicy createRetryPolicy(Configuration conf,
boolean isHAEnabled) {
{code}
bq. a bit flaky due to reliance on the clock and timing
The goal is to test client can discover active rm fast. by fast, I need to test
the time interval is reasonably small.. I'll remove the 6 seconds sleep in
startRMInThread which should help this situation. I think 10 seconds overall
should be well enough. of course, i'm open to any suggestion
bq. Can we kill the RM threads before exiting the test case ?
Do you mean the startRMInThread in the test case? that thread just call a
single method.
> Improve HA ResourceManager Failover detection on the client
> ---
>
> Key: YARN-4496
> URL: https://issues.apache.org/jira/browse/YARN-4496
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: client, resourcemanager
>Reporter: Arun Suresh
>Assignee: Jian He
> Attachments: YARN-4496.1.patch, YARN-4496.2.patch
>
>
> HDFS deployments can currently use the {{RequestHedgingProxyProvider}} to
> improve Namenode failover detection in the client. It does this by
> concurrently trying all namenodes and picks the namenode that returns the
> fastest with a successful response as the active node.
> It would be useful to have a similar ProxyProvider for the Yarn RM (it can
> possibly be done by converging some the class hierarchies to use the same
> ProxyProvider)
> This would especially be useful for large YARN deployments with multiple
> standby RMs where clients will be able to pick the active RM without having
> to traverse a list of configured RMs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4496) Improve HA ResourceManager Failover detection on the client
[
https://issues.apache.org/jira/browse/YARN-4496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15109726#comment-15109726
]
Xuan Gong commented on YARN-4496:
-
The patch looks good.
But can we separate these changes from this jira ? These changes are not
related.
{code}
public static final long DEFAULT_RESOURCEMANAGER_CONNECT_RETRY_INTERVAL_MS
- = 30 * 1000;
+ = 5 * 1000;
{code}
> Improve HA ResourceManager Failover detection on the client
> ---
>
> Key: YARN-4496
> URL: https://issues.apache.org/jira/browse/YARN-4496
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: client, resourcemanager
>Reporter: Arun Suresh
>Assignee: Jian He
> Attachments: YARN-4496.1.patch, YARN-4496.2.patch, YARN-4496.3.patch
>
>
> HDFS deployments can currently use the {{RequestHedgingProxyProvider}} to
> improve Namenode failover detection in the client. It does this by
> concurrently trying all namenodes and picks the namenode that returns the
> fastest with a successful response as the active node.
> It would be useful to have a similar ProxyProvider for the Yarn RM (it can
> possibly be done by converging some the class hierarchies to use the same
> ProxyProvider)
> This would especially be useful for large YARN deployments with multiple
> standby RMs where clients will be able to pick the active RM without having
> to traverse a list of configured RMs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4496) Improve HA ResourceManager Failover detection on the client
[
https://issues.apache.org/jira/browse/YARN-4496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15109786#comment-15109786
]
Jian He commented on YARN-4496:
---
bq. when it is possible to evaluate it using HAUtil.isHAEnabled(conf) (like it
was done before).
Because {{RetryPolicy retryPolicy = RMProxy.createRetryPolicy(conf, false);}}
this line code in RequestHedgingRMFailoverProxyProvider needs to call this
method with false, but under the context of HAUtil.isHAEnabled == true.
> Improve HA ResourceManager Failover detection on the client
> ---
>
> Key: YARN-4496
> URL: https://issues.apache.org/jira/browse/YARN-4496
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: client, resourcemanager
>Reporter: Arun Suresh
>Assignee: Jian He
> Attachments: YARN-4496.1.patch, YARN-4496.2.patch, YARN-4496.3.patch,
> YARN-4496.4.patch
>
>
> HDFS deployments can currently use the {{RequestHedgingProxyProvider}} to
> improve Namenode failover detection in the client. It does this by
> concurrently trying all namenodes and picks the namenode that returns the
> fastest with a successful response as the active node.
> It would be useful to have a similar ProxyProvider for the Yarn RM (it can
> possibly be done by converging some the class hierarchies to use the same
> ProxyProvider)
> This would especially be useful for large YARN deployments with multiple
> standby RMs where clients will be able to pick the active RM without having
> to traverse a list of configured RMs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4496) Improve HA ResourceManager Failover detection on the client
[
https://issues.apache.org/jira/browse/YARN-4496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15109793#comment-15109793
]
Jian He commented on YARN-4496:
---
The v4 patch fixed comments from Xuan and Arun.
thank you both for the review !
> Improve HA ResourceManager Failover detection on the client
> ---
>
> Key: YARN-4496
> URL: https://issues.apache.org/jira/browse/YARN-4496
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: client, resourcemanager
>Reporter: Arun Suresh
>Assignee: Jian He
> Attachments: YARN-4496.1.patch, YARN-4496.2.patch, YARN-4496.3.patch,
> YARN-4496.4.patch
>
>
> HDFS deployments can currently use the {{RequestHedgingProxyProvider}} to
> improve Namenode failover detection in the client. It does this by
> concurrently trying all namenodes and picks the namenode that returns the
> fastest with a successful response as the active node.
> It would be useful to have a similar ProxyProvider for the Yarn RM (it can
> possibly be done by converging some the class hierarchies to use the same
> ProxyProvider)
> This would especially be useful for large YARN deployments with multiple
> standby RMs where clients will be able to pick the active RM without having
> to traverse a list of configured RMs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4496) Improve HA ResourceManager Failover detection on the client
[
https://issues.apache.org/jira/browse/YARN-4496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15109805#comment-15109805
]
Arun Suresh commented on YARN-4496:
---
Got it... missed that... thanks for clarifying
+1, pending jenkins
> Improve HA ResourceManager Failover detection on the client
> ---
>
> Key: YARN-4496
> URL: https://issues.apache.org/jira/browse/YARN-4496
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: client, resourcemanager
>Reporter: Arun Suresh
>Assignee: Jian He
> Attachments: YARN-4496.1.patch, YARN-4496.2.patch, YARN-4496.3.patch,
> YARN-4496.4.patch
>
>
> HDFS deployments can currently use the {{RequestHedgingProxyProvider}} to
> improve Namenode failover detection in the client. It does this by
> concurrently trying all namenodes and picks the namenode that returns the
> fastest with a successful response as the active node.
> It would be useful to have a similar ProxyProvider for the Yarn RM (it can
> possibly be done by converging some the class hierarchies to use the same
> ProxyProvider)
> This would especially be useful for large YARN deployments with multiple
> standby RMs where clients will be able to pick the active RM without having
> to traverse a list of configured RMs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4496) Improve HA ResourceManager Failover detection on the client
[
https://issues.apache.org/jira/browse/YARN-4496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15109770#comment-15109770
]
Jian He commented on YARN-4496:
---
ok , removed that change. will track separately
> Improve HA ResourceManager Failover detection on the client
> ---
>
> Key: YARN-4496
> URL: https://issues.apache.org/jira/browse/YARN-4496
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: client, resourcemanager
>Reporter: Arun Suresh
>Assignee: Jian He
> Attachments: YARN-4496.1.patch, YARN-4496.2.patch, YARN-4496.3.patch,
> YARN-4496.4.patch
>
>
> HDFS deployments can currently use the {{RequestHedgingProxyProvider}} to
> improve Namenode failover detection in the client. It does this by
> concurrently trying all namenodes and picks the namenode that returns the
> fastest with a successful response as the active node.
> It would be useful to have a similar ProxyProvider for the Yarn RM (it can
> possibly be done by converging some the class hierarchies to use the same
> ProxyProvider)
> This would especially be useful for large YARN deployments with multiple
> standby RMs where clients will be able to pick the active RM without having
> to traverse a list of configured RMs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4496) Improve HA ResourceManager Failover detection on the client
[
https://issues.apache.org/jira/browse/YARN-4496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15109780#comment-15109780
]
Arun Suresh commented on YARN-4496:
---
bq. This method actually uses this flag
Correct.. but what I meant to say was, why you would want to add an extra
boolean attribute to the {{createRetryPolicy}} function, when it is possible to
evaluate it using {{HAUtil.isHAEnabled(conf)}} (like it was done before).
bq. Do you mean the startRMInThread in the test case? that thread just call a
single method..
Got it... I thought you had actually started an RM in a thread (maybe, just
rename it to makeRMActive)
> Improve HA ResourceManager Failover detection on the client
> ---
>
> Key: YARN-4496
> URL: https://issues.apache.org/jira/browse/YARN-4496
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: client, resourcemanager
>Reporter: Arun Suresh
>Assignee: Jian He
> Attachments: YARN-4496.1.patch, YARN-4496.2.patch, YARN-4496.3.patch,
> YARN-4496.4.patch
>
>
> HDFS deployments can currently use the {{RequestHedgingProxyProvider}} to
> improve Namenode failover detection in the client. It does this by
> concurrently trying all namenodes and picks the namenode that returns the
> fastest with a successful response as the active node.
> It would be useful to have a similar ProxyProvider for the Yarn RM (it can
> possibly be done by converging some the class hierarchies to use the same
> ProxyProvider)
> This would especially be useful for large YARN deployments with multiple
> standby RMs where clients will be able to pick the active RM without having
> to traverse a list of configured RMs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4496) Improve HA ResourceManager Failover detection on the client
[
https://issues.apache.org/jira/browse/YARN-4496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15109781#comment-15109781
]
Hadoop QA commented on YARN-4496:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 44s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
6s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 1s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 18s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
31s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 19s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
7s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
48s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 59s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 4m 23s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 44s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
57s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 26s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 5m 7s {color}
| {color:red} hadoop-yarn-project_hadoop-yarn-jdk1.8.0_66 with JDK v1.8.0_66
generated 1 new + 9 unchanged - 0 fixed = 10 total (was 9) {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 26s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 22s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 7m 29s {color}
| {color:red} hadoop-yarn-project_hadoop-yarn-jdk1.7.0_91 with JDK v1.7.0_91
generated 1 new + 10 unchanged - 0 fixed = 11 total (was 10) {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 22s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 33s
{color} | {color:red} hadoop-yarn-project/hadoop-yarn: patch generated 3 new +
221 unchanged - 0 fixed = 224 total (was 221) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 36s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
5s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 1s
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m
31s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 47s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 4m 37s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 23s
{color} | {color:green} hadoop-yarn-api in the patch passed with JDK v1.8.0_66.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 3s
{color} | {color:green} hadoop-yarn-common in the patch passed with JDK
[jira] [Commented] (YARN-4496) Improve HA ResourceManager Failover detection on the client
[
https://issues.apache.org/jira/browse/YARN-4496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15108356#comment-15108356
]
Hadoop QA commented on YARN-4496:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 54s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
35s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 54s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 9s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
29s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 16s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
5s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
36s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 53s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 4m 16s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 42s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
50s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 51s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 4m 24s {color}
| {color:red} hadoop-yarn-project_hadoop-yarn-jdk1.8.0_66 with JDK v1.8.0_66
generated 1 new + 9 unchanged - 0 fixed = 10 total (was 9) {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 51s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 14s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 6m 38s {color}
| {color:red} hadoop-yarn-project_hadoop-yarn-jdk1.7.0_91 with JDK v1.7.0_91
generated 1 new + 10 unchanged - 0 fixed = 11 total (was 10) {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 14s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 35s
{color} | {color:red} hadoop-yarn-project/hadoop-yarn: patch generated 3 new +
221 unchanged - 0 fixed = 224 total (was 221) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 22s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
1s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 0s
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m 2s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 7s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 4m 27s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 24s
{color} | {color:green} hadoop-yarn-api in the patch passed with JDK v1.8.0_66.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 7s
{color} | {color:green} hadoop-yarn-common in the patch passed with JDK
[jira] [Commented] (YARN-4496) Improve HA ResourceManager Failover detection on the client
[
https://issues.apache.org/jira/browse/YARN-4496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15107685#comment-15107685
]
Jian He commented on YARN-4496:
---
Uploaded a patch:
- added a new RequestHedgingRMFailoverProxyProvider. When client tries to
failover, it uses separate proxy object to talk to each RM simultaneously ,
each proxy retries the RM until the first one receives a response from the
active RM. All the other requests are then cancelled.
- changed the default rm-retry-interval to be 5 seconds, 30 seconds interval I
think is too long.
> Improve HA ResourceManager Failover detection on the client
> ---
>
> Key: YARN-4496
> URL: https://issues.apache.org/jira/browse/YARN-4496
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: client, resourcemanager
>Reporter: Arun Suresh
>Assignee: Jian He
> Attachments: YARN-4496.1.patch
>
>
> HDFS deployments can currently use the {{RequestHedgingProxyProvider}} to
> improve Namenode failover detection in the client. It does this by
> concurrently trying all namenodes and picks the namenode that returns the
> fastest with a successful response as the active node.
> It would be useful to have a similar ProxyProvider for the Yarn RM (it can
> possibly be done by converging some the class hierarchies to use the same
> ProxyProvider)
> This would especially be useful for large YARN deployments with multiple
> standby RMs where clients will be able to pick the active RM without having
> to traverse a list of configured RMs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4496) Improve HA ResourceManager Failover detection on the client
[
https://issues.apache.org/jira/browse/YARN-4496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15108014#comment-15108014
]
Hadoop QA commented on YARN-4496:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 57s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
47s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 47s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 9s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
30s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 11s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
4s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
43s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 1s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 4m 21s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 42s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
49s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 48s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 4m 20s {color}
| {color:red} hadoop-yarn-project_hadoop-yarn-jdk1.8.0_66 with JDK v1.8.0_66
generated 1 new + 9 unchanged - 0 fixed = 10 total (was 9) {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 48s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 8s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 6m 29s {color}
| {color:red} hadoop-yarn-project_hadoop-yarn-jdk1.7.0_91 with JDK v1.7.0_91
generated 1 new + 10 unchanged - 0 fixed = 11 total (was 10) {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 8s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 29s
{color} | {color:red} hadoop-yarn-project/hadoop-yarn: patch generated 5 new +
221 unchanged - 0 fixed = 226 total (was 221) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 2s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
55s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 0s
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
29s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 43s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 4m 8s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 21s
{color} | {color:green} hadoop-yarn-api in the patch passed with JDK v1.8.0_66.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 3s
{color} | {color:green} hadoop-yarn-common in the patch passed with JDK
[jira] [Commented] (YARN-4496) Improve HA ResourceManager Failover detection on the client
[
https://issues.apache.org/jira/browse/YARN-4496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15101053#comment-15101053
]
Jian He commented on YARN-4496:
---
taking over, thanks !
> Improve HA ResourceManager Failover detection on the client
> ---
>
> Key: YARN-4496
> URL: https://issues.apache.org/jira/browse/YARN-4496
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: client, resourcemanager
>Reporter: Arun Suresh
>Assignee: Jian He
>
> HDFS deployments can currently use the {{RequestHedgingProxyProvider}} to
> improve Namenode failover detection in the client. It does this by
> concurrently trying all namenodes and picks the namenode that returns the
> fastest with a successful response as the active node.
> It would be useful to have a similar ProxyProvider for the Yarn RM (it can
> possibly be done by converging some the class hierarchies to use the same
> ProxyProvider)
> This would especially be useful for large YARN deployments with multiple
> standby RMs where clients will be able to pick the active RM without having
> to traverse a list of configured RMs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4496) Improve HA ResourceManager Failover detection on the client
[
https://issues.apache.org/jira/browse/YARN-4496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15088332#comment-15088332
]
Arun Suresh commented on YARN-4496:
---
Hey [~jianhe], Feel free to take it up.. I can help with the reviews..
> Improve HA ResourceManager Failover detection on the client
> ---
>
> Key: YARN-4496
> URL: https://issues.apache.org/jira/browse/YARN-4496
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: client, resourcemanager
>Reporter: Arun Suresh
>Assignee: Arun Suresh
>
> HDFS deployments can currently use the {{RequestHedgingProxyProvider}} to
> improve Namenode failover detection in the client. It does this by
> concurrently trying all namenodes and picks the namenode that returns the
> fastest with a successful response as the active node.
> It would be useful to have a similar ProxyProvider for the Yarn RM (it can
> possibly be done by converging some the class hierarchies to use the same
> ProxyProvider)
> This would especially be useful for large YARN deployments with multiple
> standby RMs where clients will be able to pick the active RM without having
> to traverse a list of configured RMs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4496) Improve HA ResourceManager Failover detection on the client
[
https://issues.apache.org/jira/browse/YARN-4496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15088251#comment-15088251
]
Jian He commented on YARN-4496:
---
+1, [~asuresh], are you working on this ?
> Improve HA ResourceManager Failover detection on the client
> ---
>
> Key: YARN-4496
> URL: https://issues.apache.org/jira/browse/YARN-4496
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: client, resourcemanager
>Reporter: Arun Suresh
>Assignee: Arun Suresh
>
> HDFS deployments can currently use the {{RequestHedgingProxyProvider}} to
> improve Namenode failover detection in the client. It does this by
> concurrently trying all namenodes and picks the namenode that returns the
> fastest with a successful response as the active node.
> It would be useful to have a similar ProxyProvider for the Yarn RM (it can
> possibly be done by converging some the class hierarchies to use the same
> ProxyProvider)
> This would especially be useful for large YARN deployments with multiple
> standby RMs where clients will be able to pick the active RM without having
> to traverse a list of configured RMs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4496) Improve HA ResourceManager Failover detection on the client
[
https://issues.apache.org/jira/browse/YARN-4496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15070140#comment-15070140
]
Subru Krishnan commented on YARN-4496:
--
+1. Thanks [~asuresh] for initiating this. To add more context, our deployments
are fairly large with multiple secondaries which is resulting in considerable
connection latencies based on the current failover proxy.
> Improve HA ResourceManager Failover detection on the client
> ---
>
> Key: YARN-4496
> URL: https://issues.apache.org/jira/browse/YARN-4496
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: client, resourcemanager
>Reporter: Arun Suresh
>Assignee: Arun Suresh
>
> HDFS deployments can currently use the {{RequestHedgingProxyProvider}} to
> improve Namenode failover detection in the client. It does this by
> concurrently trying all namenodes and picks the namenode that returns the
> fastest with a successful response as the active node.
> It would be useful to have a similar ProxyProvider for the Yarn RM (it can
> possibly be done by converging some the class hierarchies to use the same
> ProxyProvider)
> This would especially be useful for large YARN deployments with multiple
> standby RMs where clients will be able to pick the active RM without having
> to traverse a list of configured RMs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4496) Improve HA ResourceManager Failover detection on the client
[
https://issues.apache.org/jira/browse/YARN-4496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15069043#comment-15069043
]
Arun Suresh commented on YARN-4496:
---
Had a discussion with [~subru], and we think this would also be useful for YARN
federation (YARN-2915) as well
> Improve HA ResourceManager Failover detection on the client
> ---
>
> Key: YARN-4496
> URL: https://issues.apache.org/jira/browse/YARN-4496
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: client, resourcemanager
>Reporter: Arun Suresh
>Assignee: Arun Suresh
>
> HDFS deployments can currently use the {{RequestHedgingProxyProvider}} to
> improve Namenode failover detection in the client. It does this by
> concurrently trying all namenodes and picks the namenode that returns the
> fastest with a successful response as the active node.
> It would be useful to have a similar ProxyProvider for the Yarn RM (it can
> possibly be done by converging some the class hierarchies to use the same
> ProxyProvider)
> This would especially be useful for large YARN deployments with multiple
> standby RMs where clients will be able to pick the active RM without having
> to traverse a list of configured RMs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
