[jira] [Comment Edited] (HDFS-13388) RequestHedgingProxyProvider calls multiple configured NNs all the time
[ https://issues.apache.org/jira/browse/HDFS-13388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16470293#comment-16470293 ] Bibin A Chundatt edited comment on HDFS-13388 at 5/10/18 12:05 PM: --- Since HDFS-12813 was closed long back we might need a backport jira. I am ok with you filing jira too. was (Author: bibinchundatt): Since HDFS-12813 is long back closed might need a backport jira . I am ok with you filing jira too. > RequestHedgingProxyProvider calls multiple configured NNs all the time > -- > > Key: HDFS-13388 > URL: https://issues.apache.org/jira/browse/HDFS-13388 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs-client >Reporter: Jinglun >Assignee: Jinglun >Priority: Major > Fix For: 3.2.0, 3.1.1, 3.0.3 > > Attachments: HADOOP-13388.0001.patch, HADOOP-13388.0002.patch, > HADOOP-13388.0003.patch, HADOOP-13388.0004.patch, HADOOP-13388.0005.patch, > HADOOP-13388.0006.patch, HADOOP-13388.0007.patch, HADOOP-13388.0008.patch, > HADOOP-13388.0009.patch, HADOOP-13388.0010.patch, HADOOP-13388.0011.patch, > HADOOP-13388.0012.patch, HADOOP-13388.0013.patch, HADOOP-13388.0014.patch > > > In HDFS-7858 RequestHedgingProxyProvider was designed to "first > simultaneously call multiple configured NNs to decide which is the active > Namenode and then for subsequent calls it will invoke the previously > successful NN ." But the current code call multiple configured NNs every time > even when we already got the successful NN. > That's because in RetryInvocationHandler.java, ProxyDescriptor's member > proxyInfo is assigned only when it is constructed or when failover occurs. > RequestHedgingProxyProvider.currentUsedProxy is null in both cases, so the > only proxy we can get is always a dynamic proxy handled by > RequestHedgingInvocationHandler.class. RequestHedgingInvocationHandler.class > handles invoked method by calling multiple configured NNs. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-13388) RequestHedgingProxyProvider calls multiple configured NNs all the time
[
https://issues.apache.org/jira/browse/HDFS-13388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16469901#comment-16469901
]
Bibin A Chundatt edited comment on HDFS-13388 at 5/10/18 6:26 AM:
--
[~elgoiri]
branch-3 compilation is broken. Backport of HDFS-12813 should fix compilation.
{code:java}
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-compiler-plugin:3.1:compile (default-compile) on
project hadoop-hdfs-client: Compilation failure: Compilation failure:
[ERROR]
/opt/apacheprojects/hadoop/FORCOMMIT/branch3/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/RequestHedgingProxyProvider.java:[79,11]
cannot find symbol
[ERROR] symbol: variable currentUsedProxy
[ERROR] location: class
org.apache.hadoop.hdfs.server.namenode.ha.RequestHedgingProxyProvider.RequestHedgingInvocationHandler
[ERROR]
/opt/apacheprojects/hadoop/FORCOMMIT/branch3/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/RequestHedgingProxyProvider.java:[81,41]
cannot find symbol
[ERROR] symbol: variable currentUsedProxy
[ERROR] location: class
org.apache.hadoop.hdfs.server.namenode.ha.RequestHedgingProxyProvider.RequestHedgingInvocationHandler
[ERROR]
/opt/apacheprojects/hadoop/FORCOMMIT/branch3/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/RequestHedgingProxyProvider.java:[83,15]
cannot find symbol
[ERROR] symbol: variable currentUsedProxy
[ERROR] location: class
org.apache.hadoop.hdfs.server.namenode.ha.RequestHedgingProxyProvider.RequestHedgingInvocationHandler
[ERROR]
/opt/apacheprojects/hadoop/FORCOMMIT/branch3/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/RequestHedgingProxyProvider.java:[85,18]
cannot find symbol
[ERROR] symbol: class InvocationTargetException
[ERROR] location: class
org.apache.hadoop.hdfs.server.namenode.ha.RequestHedgingProxyProvider.RequestHedgingInvocationHandler
[ERROR]
/opt/apacheprojects/hadoop/FORCOMMIT/branch3/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/RequestHedgingProxyProvider.java:[87,49]
cannot find symbol
[ERROR] symbol: variable currentUsedProxy
[ERROR] location: class
org.apache.hadoop.hdfs.server.namenode.ha.RequestHedgingProxyProvider.RequestHedgingInvocationHandler
[ERROR]
/opt/apacheprojects/hadoop/FORCOMMIT/branch3/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/RequestHedgingProxyProvider.java:[89,15]
cannot find symbol
[ERROR] symbol: variable currentUsedProxy
[ERROR] location: class
org.apache.hadoop.hdfs.server.namenode.ha.RequestHedgingProxyProvider.RequestHedgingInvocationHandler
[ERROR] -> [Help 1]
{code}
was (Author: bibinchundatt):
[~yzhangal]
branch-3 compilation is broken
{code}
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-compiler-plugin:3.1:compile (default-compile) on
project hadoop-hdfs-client: Compilation failure: Compilation failure:
[ERROR]
/opt/apacheprojects/hadoop/FORCOMMIT/branch3/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/RequestHedgingProxyProvider.java:[79,11]
cannot find symbol
[ERROR] symbol: variable currentUsedProxy
[ERROR] location: class
org.apache.hadoop.hdfs.server.namenode.ha.RequestHedgingProxyProvider.RequestHedgingInvocationHandler
[ERROR]
/opt/apacheprojects/hadoop/FORCOMMIT/branch3/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/RequestHedgingProxyProvider.java:[81,41]
cannot find symbol
[ERROR] symbol: variable currentUsedProxy
[ERROR] location: class
org.apache.hadoop.hdfs.server.namenode.ha.RequestHedgingProxyProvider.RequestHedgingInvocationHandler
[ERROR]
/opt/apacheprojects/hadoop/FORCOMMIT/branch3/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/RequestHedgingProxyProvider.java:[83,15]
cannot find symbol
[ERROR] symbol: variable currentUsedProxy
[ERROR] location: class
org.apache.hadoop.hdfs.server.namenode.ha.RequestHedgingProxyProvider.RequestHedgingInvocationHandler
[ERROR]
/opt/apacheprojects/hadoop/FORCOMMIT/branch3/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/RequestHedgingProxyProvider.java:[85,18]
cannot find symbol
[ERROR] symbol: class InvocationTargetException
[ERROR] location: class
org.apache.hadoop.hdfs.server.namenode.ha.RequestHedgingProxyProvider.RequestHedgingInvocationHandler
[ERROR]
/opt/apacheprojects/hadoop/FORCOMMIT/branch3/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/RequestHedgingProxyProvider.java:[87,49]
cannot find symbol
[ERROR] symbol: variable currentUsedProxy
[jira] [Comment Edited] (HDFS-13388) RequestHedgingProxyProvider calls multiple configured NNs all the time
[ https://issues.apache.org/jira/browse/HDFS-13388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16469299#comment-16469299 ] Yongjun Zhang edited comment on HDFS-13388 at 5/9/18 10:10 PM: --- Hi [~elgoiri], This Jira was released in 3.0.2 then get reverted from branch-3.0. Seems reasonable to get it to branch-3.0, which is now targeting for 3.0.3. Would you please do so? Thanks. was (Author: yzhangal): Hi [~elgoiri], This Jira was released in 3.0.2 then get reverted from branch-3.0. Seems reasonable to get it to branch-3.0, which is now targeting for 3.0.3. Would you please do so? Thanks. > RequestHedgingProxyProvider calls multiple configured NNs all the time > -- > > Key: HDFS-13388 > URL: https://issues.apache.org/jira/browse/HDFS-13388 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs-client >Reporter: Jinglun >Assignee: Jinglun >Priority: Major > Fix For: 3.2.0, 3.1.1, 3.0.3 > > Attachments: HADOOP-13388.0001.patch, HADOOP-13388.0002.patch, > HADOOP-13388.0003.patch, HADOOP-13388.0004.patch, HADOOP-13388.0005.patch, > HADOOP-13388.0006.patch, HADOOP-13388.0007.patch, HADOOP-13388.0008.patch, > HADOOP-13388.0009.patch, HADOOP-13388.0010.patch, HADOOP-13388.0011.patch, > HADOOP-13388.0012.patch, HADOOP-13388.0013.patch, HADOOP-13388.0014.patch > > > In HDFS-7858 RequestHedgingProxyProvider was designed to "first > simultaneously call multiple configured NNs to decide which is the active > Namenode and then for subsequent calls it will invoke the previously > successful NN ." But the current code call multiple configured NNs every time > even when we already got the successful NN. > That's because in RetryInvocationHandler.java, ProxyDescriptor's member > proxyInfo is assigned only when it is constructed or when failover occurs. > RequestHedgingProxyProvider.currentUsedProxy is null in both cases, so the > only proxy we can get is always a dynamic proxy handled by > RequestHedgingInvocationHandler.class. RequestHedgingInvocationHandler.class > handles invoked method by calling multiple configured NNs. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-13388) RequestHedgingProxyProvider calls multiple configured NNs all the time
[ https://issues.apache.org/jira/browse/HDFS-13388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16434817#comment-16434817 ] Jinglun edited comment on HDFS-13388 at 4/12/18 3:42 AM: - Hi [~elgoiri], patch 007 is an addendum for what we committed. How about we go with the first option? I upload a new patch 009 which includes all the changes. I also add a new test case to test whether RequestHedgingInvocationHandler throws the right Exception. I will submit the patch when the current commit is reverted from trunk, nor jenkins won't let me pass. I'm also ok with the other 2 options, so if you think other option is better, I could give a new patch or open a new jira. was (Author: lijinglun): hi [~elgoiri], patch 007 is an addendum for what we committed. I prefer the first option, and patch 008 includes all the changes. > RequestHedgingProxyProvider calls multiple configured NNs all the time > -- > > Key: HDFS-13388 > URL: https://issues.apache.org/jira/browse/HDFS-13388 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs-client >Reporter: Jinglun >Assignee: Jinglun >Priority: Major > Fix For: 2.10.0, 3.2.0, 3.1.1 > > Attachments: HADOOP-13388.0001.patch, HADOOP-13388.0002.patch, > HADOOP-13388.0003.patch, HADOOP-13388.0004.patch, HADOOP-13388.0005.patch, > HADOOP-13388.0006.patch, HADOOP-13388.0007.patch, HADOOP-13388.0008.patch, > HADOOP-13388.0009.patch > > > In HDFS-7858 RequestHedgingProxyProvider was designed to "first > simultaneously call multiple configured NNs to decide which is the active > Namenode and then for subsequent calls it will invoke the previously > successful NN ." But the current code call multiple configured NNs every time > even when we already got the successful NN. > That's because in RetryInvocationHandler.java, ProxyDescriptor's member > proxyInfo is assigned only when it is constructed or when failover occurs. > RequestHedgingProxyProvider.currentUsedProxy is null in both cases, so the > only proxy we can get is always a dynamic proxy handled by > RequestHedgingInvocationHandler.class. RequestHedgingInvocationHandler.class > handles invoked method by calling multiple configured NNs. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-13388) RequestHedgingProxyProvider calls multiple configured NNs all the time
[ https://issues.apache.org/jira/browse/HDFS-13388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16433855#comment-16433855 ] Jinglun edited comment on HDFS-13388 at 4/11/18 12:41 PM: -- Hi [~elgoiri], I find a bug of 0006.patch: the InvocationTargetException isn't handled well so we can't get the correct Exception information when the proxy fails. My bad:(. The fix is in 0007.patch. was (Author: lijinglun): Hi [~elgoiri], I find a bug of 0006.patch: the InvocationTargetException isn't handled well so we can't get the correct Exception information when the proxy fails. My bad:(. > RequestHedgingProxyProvider calls multiple configured NNs all the time > -- > > Key: HDFS-13388 > URL: https://issues.apache.org/jira/browse/HDFS-13388 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs-client >Reporter: Jinglun >Assignee: Jinglun >Priority: Major > Fix For: 2.10.0, 3.2.0, 3.1.1 > > Attachments: HADOOP-13388.0001.patch, HADOOP-13388.0002.patch, > HADOOP-13388.0003.patch, HADOOP-13388.0004.patch, HADOOP-13388.0005.patch, > HADOOP-13388.0006.patch, HADOOP-13388.0007.patch > > > In HDFS-7858 RequestHedgingProxyProvider was designed to "first > simultaneously call multiple configured NNs to decide which is the active > Namenode and then for subsequent calls it will invoke the previously > successful NN ." But the current code call multiple configured NNs every time > even when we already got the successful NN. > That's because in RetryInvocationHandler.java, ProxyDescriptor's member > proxyInfo is assigned only when it is constructed or when failover occurs. > RequestHedgingProxyProvider.currentUsedProxy is null in both cases, so the > only proxy we can get is always a dynamic proxy handled by > RequestHedgingInvocationHandler.class. RequestHedgingInvocationHandler.class > handles invoked method by calling multiple configured NNs. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-13388) RequestHedgingProxyProvider calls multiple configured NNs all the time
[ https://issues.apache.org/jira/browse/HDFS-13388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16432503#comment-16432503 ] Íñigo Goiri edited comment on HDFS-13388 at 4/10/18 3:52 PM: - Actually, branch-2 is fine as the rename of the variable is already there. [~LiJinglun], post a branch-2.9 if you want it there. Sorry [~Sammi] for the inconvenience. was (Author: elgoiri): Actually, branch-2 is fine as the rename of the variable is already there. [~LiJinglun], post a branch-2.9 if you want it there. > RequestHedgingProxyProvider calls multiple configured NNs all the time > -- > > Key: HDFS-13388 > URL: https://issues.apache.org/jira/browse/HDFS-13388 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs-client >Reporter: Jinglun >Assignee: Jinglun >Priority: Major > Fix For: 2.10.0, 3.2.0, 3.1.1, 3.0.4 > > Attachments: HADOOP-13388.0001.patch, HADOOP-13388.0002.patch, > HADOOP-13388.0003.patch, HADOOP-13388.0004.patch, HADOOP-13388.0005.patch, > HADOOP-13388.0006.patch > > > In HDFS-7858 RequestHedgingProxyProvider was designed to "first > simultaneously call multiple configured NNs to decide which is the active > Namenode and then for subsequent calls it will invoke the previously > successful NN ." But the current code call multiple configured NNs every time > even when we already got the successful NN. > That's because in RetryInvocationHandler.java, ProxyDescriptor's member > proxyInfo is assigned only when it is constructed or when failover occurs. > RequestHedgingProxyProvider.currentUsedProxy is null in both cases, so the > only proxy we can get is always a dynamic proxy handled by > RequestHedgingInvocationHandler.class. RequestHedgingInvocationHandler.class > handles invoked method by calling multiple configured NNs. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-13388) RequestHedgingProxyProvider calls multiple configured NNs all the time
[ https://issues.apache.org/jira/browse/HDFS-13388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16432185#comment-16432185 ] Jinglun edited comment on HDFS-13388 at 4/10/18 12:30 PM: -- Hi [~elgoiri], [~Sammi] , the problem is the variable represents the successful proxy is named successfulProxy in branch-2.9 while named currentUsedProxy in trunk. What could I do to fix this? Should i submit a new patch for branch-2.9？ The variable is named currentUsedProxy in branch-2 so it should be ok. was (Author: lijinglun): Hi [~elgoiri], [~Sammi] , the problem is the variable represents the successful proxy is named successfulProxy in branch-2.9 while named currentUsedProxy in trunk. What could I do to fix this? Should i submit a new patch for branch-2.9？ branch-2 is named currentUsedProxy so it should be ok. > RequestHedgingProxyProvider calls multiple configured NNs all the time > -- > > Key: HDFS-13388 > URL: https://issues.apache.org/jira/browse/HDFS-13388 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs-client >Reporter: Jinglun >Assignee: Jinglun >Priority: Major > Fix For: 2.10.0, 3.2.0, 3.1.1, 2.9.2, 3.0.4 > > Attachments: HADOOP-13388.0001.patch, HADOOP-13388.0002.patch, > HADOOP-13388.0003.patch, HADOOP-13388.0004.patch, HADOOP-13388.0005.patch, > HADOOP-13388.0006.patch > > > In HDFS-7858 RequestHedgingProxyProvider was designed to "first > simultaneously call multiple configured NNs to decide which is the active > Namenode and then for subsequent calls it will invoke the previously > successful NN ." But the current code call multiple configured NNs every time > even when we already got the successful NN. > That's because in RetryInvocationHandler.java, ProxyDescriptor's member > proxyInfo is assigned only when it is constructed or when failover occurs. > RequestHedgingProxyProvider.currentUsedProxy is null in both cases, so the > only proxy we can get is always a dynamic proxy handled by > RequestHedgingInvocationHandler.class. RequestHedgingInvocationHandler.class > handles invoked method by calling multiple configured NNs. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-13388) RequestHedgingProxyProvider calls multiple configured NNs all the time
[ https://issues.apache.org/jira/browse/HDFS-13388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16431908#comment-16431908 ] SammiChen edited comment on HDFS-13388 at 4/10/18 8:22 AM: --- Hi [~LiJinglun] and [~elgoiri], branch-2.9 suffers build failure with this commit. Would you please double check it? also check the branch-2. was (Author: sammi): Hi [~LiJinglun] and [~elgoiri], branch-2.9 suffers build failure with this commit. Would you please double check it? > RequestHedgingProxyProvider calls multiple configured NNs all the time > -- > > Key: HDFS-13388 > URL: https://issues.apache.org/jira/browse/HDFS-13388 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs-client >Reporter: Jinglun >Assignee: Jinglun >Priority: Major > Fix For: 2.10.0, 3.2.0, 3.1.1, 2.9.2, 3.0.4 > > Attachments: HADOOP-13388.0001.patch, HADOOP-13388.0002.patch, > HADOOP-13388.0003.patch, HADOOP-13388.0004.patch, HADOOP-13388.0005.patch, > HADOOP-13388.0006.patch > > > In HDFS-7858 RequestHedgingProxyProvider was designed to "first > simultaneously call multiple configured NNs to decide which is the active > Namenode and then for subsequent calls it will invoke the previously > successful NN ." But the current code call multiple configured NNs every time > even when we already got the successful NN. > That's because in RetryInvocationHandler.java, ProxyDescriptor's member > proxyInfo is assigned only when it is constructed or when failover occurs. > RequestHedgingProxyProvider.currentUsedProxy is null in both cases, so the > only proxy we can get is always a dynamic proxy handled by > RequestHedgingInvocationHandler.class. RequestHedgingInvocationHandler.class > handles invoked method by calling multiple configured NNs. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
