[
https://issues.apache.org/jira/browse/HADOOP-14982?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16226913#comment-16226913
]
Hadoop QA commented on HADOOP-14982:
------------------------------------
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
10s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 13m
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 42s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
50s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 10m
42s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 37s{color} | {color:orange} hadoop-common-project/hadoop-common: The patch
generated 12 new + 104 unchanged - 0 fixed = 116 total (was 104) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 52s{color} | {color:green} patch has no errors when building and testing our
client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 8m 3s{color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
29s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 80m 32s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.ha.TestZKFailoverController |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HADOOP-14982 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12894978/HADOOP-14982-003.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 7b1811e3b477 3.13.0-117-generic #164-Ubuntu SMP Fri Apr 7
11:05:26 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / c02d2ba |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_131 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13604/artifact/out/diff-checkstyle-hadoop-common-project_hadoop-common.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13604/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13604/testReport/ |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13604/console |
| Powered by | Apache Yetus 0.7.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Clients using FailoverOnNetworkExceptionRetry can go into a loop if they're
> used without authenticating with kerberos in HA env
> -------------------------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-14982
> URL: https://issues.apache.org/jira/browse/HADOOP-14982
> Project: Hadoop Common
> Issue Type: Bug
> Components: common
> Reporter: Peter Bacsko
> Assignee: Peter Bacsko
> Attachments: HADOOP-14892-001.patch, HADOOP-14892-002.patch,
> HADOOP-14982-003.patch
>
>
> If HA is configured for the Resource Manager in a secure environment, using
> the mapred client goes into a loop if the user is not authenticated with
> Kerberos.
> {noformat}
> [root@pb6sec-1 ~]# mapred job -list
> 17/10/25 06:37:43 INFO client.ConfiguredRMFailoverProxyProvider: Failing over
> to rm36
> 17/10/25 06:37:43 WARN ipc.Client: Exception encountered while connecting to
> the server : javax.security.sasl.SaslException: GSS initiate failed [Caused
> by GSSException: No valid credentials provided (Mechanism level: Failed to
> find any Kerberos tgt)]
> 17/10/25 06:37:43 INFO retry.RetryInvocationHandler: java.io.IOException:
> Failed on local exception: java.io.IOException:
> javax.security.sasl.SaslException: GSS initiate failed [Caused by
> GSSException: No valid credentials provided (Mechanism level: Failed to find
> any Kerberos tgt)]; Host Details : local host is:
> "host_redacted/IP_redacted"; destination host is: "com.host2.redacted:8032; ,
> while invoking ApplicationClientProtocolPBClientImpl.getApplications over
> rm36 after 1 failover attempts. Trying to failover after sleeping for 160ms.
> 17/10/25 06:37:43 INFO client.ConfiguredRMFailoverProxyProvider: Failing over
> to rm25
> 17/10/25 06:37:43 INFO retry.RetryInvocationHandler:
> java.net.ConnectException: Call From host_redacted/IP_redacted to
> com.host.redacted:8032 failed on connection exception:
> java.net.ConnectException: Connection refused; For more details see:
> http://wiki.apache.org/hadoop/ConnectionRefused, while invoking
> ApplicationClientProtocolPBClientImpl.getApplications over rm25 after 2
> failover attempts. Trying to failover after sleeping for 582ms.
> 17/10/25 06:37:44 INFO client.ConfiguredRMFailoverProxyProvider: Failing over
> to rm36
> 17/10/25 06:37:44 WARN ipc.Client: Exception encountered while connecting to
> the server : javax.security.sasl.SaslException: GSS initiate failed [Caused
> by GSSException: No valid credentials provided (Mechanism level: Failed to
> find any Kerberos tgt)]
> 17/10/25 06:37:44 INFO retry.RetryInvocationHandler: java.io.IOException:
> Failed on local exception: java.io.IOException:
> javax.security.sasl.SaslException: GSS initiate failed [Caused by
> GSSException: No valid credentials provided (Mechanism level: Failed to find
> any Kerberos tgt)]; Host Details : local host is:
> "host_redacted/IP_redacted"; destination host is: "com.host2.redacted:8032; ,
> while invoking ApplicationClientProtocolPBClientImpl.getApplications over
> rm36 after 3 failover attempts. Trying to failover after sleeping for 977ms.
> 17/10/25 06:37:45 INFO client.ConfiguredRMFailoverProxyProvider: Failing over
> to rm25
> 17/10/25 06:37:45 INFO retry.RetryInvocationHandler:
> java.net.ConnectException: Call From host_redacted/IP_redacted to
> com.host.redacted:8032 failed on connection exception:
> java.net.ConnectException: Connection refused; For more details see:
> http://wiki.apache.org/hadoop/ConnectionRefused, while invoking
> ApplicationClientProtocolPBClientImpl.getApplications over rm25 after 4
> failover attempts. Trying to failover after sleeping for 1667ms.
> 17/10/25 06:37:46 INFO client.ConfiguredRMFailoverProxyProvider: Failing over
> to rm36
> 17/10/25 06:37:46 WARN ipc.Client: Exception encountered while connecting to
> the server : javax.security.sasl.SaslException: GSS initiate failed [Caused
> by GSSException: No valid credentials provided (Mechanism level: Failed to
> find any Kerberos tgt)]
> 17/10/25 06:37:46 INFO retry.RetryInvocationHandler: java.io.IOException:
> Failed on local exception: java.io.IOException:
> javax.security.sasl.SaslException: GSS initiate failed [Caused by
> GSSException: No valid credentials provided (Mechanism level: Failed to find
> any Kerberos tgt)]; Host Details : local host is:
> "host_redacted/IP_redacted"; destination host is: "com.host2.redacted:8032; ,
> while invoking ApplicationClientProtocolPBClientImpl.getApplications over
> rm36 after 5 failover attempts. Trying to failover after sleeping for 2776ms.
> 17/10/25 06:37:49 INFO client.ConfiguredRMFailoverProxyProvider: Failing over
> to rm25
> 17/10/25 06:37:49 INFO retry.RetryInvocationHandler:
> java.net.ConnectException: Call From host_redacted/IP_redacted to
> com.host.redacted:8032 failed on connection exception:
> java.net.ConnectException: Connection refused; For more details see:
> http://wiki.apache.org/hadoop/ConnectionRefused, while invoking
> ApplicationClientProtocolPBClientImpl.getApplications over rm25 after 6
> failover attempts. Trying to failover after sleeping for 1055ms.
> 17/10/25 06:37:50 INFO client.ConfiguredRMFailoverProxyProvider: Failing over
> to rm36
> 17/10/25 06:37:50 WARN ipc.Client: Exception encountered while connecting to
> the server : javax.security.sasl.SaslException: GSS initiate failed [Caused
> by GSSException: No valid credentials provided (Mechanism level: Failed to
> find any Kerberos tgt)]
> 17/10/25 06:37:50 INFO retry.RetryInvocationHandler: java.io.IOException:
> Failed on local exception: java.io.IOException:
> javax.security.sasl.SaslException: GSS initiate failed [Caused by
> GSSException: No valid credentials provided (Mechanism level: Failed to find
> any Kerberos tgt)]; Host Details : local host is:
> "host_redacted/IP_redacted"; destination host is: "com.host2.redacted:8032; ,
> while invoking ApplicationClientProtocolPBClientImpl.getApplications over
> rm36 after 7 failover attempts. Trying to failover after sleeping for 2608ms.
> ...
> {noformat}
> The reason is that the retry handler sees a {{ConnectException}}, then fails
> over to the inactive RM. This obviously doesn't work, so it comes back to the
> active and whole process starts again. The RetryHandler should examine if the
> {{ConnectException}} is actually caused by a {{GSSException}} (and probably
> check the "No valid credentials provided" message) and if so, it should not
> perform a failover.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
