[jira] [Commented] (HADOOP-9969) TGT expiration doesn't trigger Kerberos relogin
[ https://issues.apache.org/jira/browse/HADOOP-9969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16396727#comment-16396727 ] caixiaofeng commented on HADOOP-9969: - and the code in 2.7.2 is the same as already add the patch > TGT expiration doesn't trigger Kerberos relogin > --- > > Key: HADOOP-9969 > URL: https://issues.apache.org/jira/browse/HADOOP-9969 > Project: Hadoop Common > Issue Type: Bug > Components: ipc, security >Affects Versions: 2.1.0-beta, 2.5.0, 2.5.2, 2.6.0, 2.6.1, 2.8.0, 2.7.1, > 2.6.2, 2.6.3 > Environment: IBM JDK7 >Reporter: Yu Gao >Priority: Major > Attachments: HADOOP-9969.patch, JobTracker.log > > > In HADOOP-9698 & HADOOP-9850, RPC client and Sasl client have been changed to > respect the auth method advertised from server, instead of blindly attempting > the configured one at client side. However, when TGT has expired, an > exception will be thrown from SaslRpcClient#createSaslClient(SaslAuth > authType), and at this time the authMethod still holds the initial value > which is SIMPLE and never has a chance to be updated with the expected one > requested by server, so kerberos relogin will not happen. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-9969) TGT expiration doesn't trigger Kerberos relogin
[ https://issues.apache.org/jira/browse/HADOOP-9969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16396722#comment-16396722 ] caixiaofeng commented on HADOOP-9969: - any update? meet this with ibmjdk-1.7.0 SR4 hadoop2.7.2 > TGT expiration doesn't trigger Kerberos relogin > --- > > Key: HADOOP-9969 > URL: https://issues.apache.org/jira/browse/HADOOP-9969 > Project: Hadoop Common > Issue Type: Bug > Components: ipc, security >Affects Versions: 2.1.0-beta, 2.5.0, 2.5.2, 2.6.0, 2.6.1, 2.8.0, 2.7.1, > 2.6.2, 2.6.3 > Environment: IBM JDK7 >Reporter: Yu Gao >Priority: Major > Attachments: HADOOP-9969.patch, JobTracker.log > > > In HADOOP-9698 & HADOOP-9850, RPC client and Sasl client have been changed to > respect the auth method advertised from server, instead of blindly attempting > the configured one at client side. However, when TGT has expired, an > exception will be thrown from SaslRpcClient#createSaslClient(SaslAuth > authType), and at this time the authMethod still holds the initial value > which is SIMPLE and never has a chance to be updated with the expected one > requested by server, so kerberos relogin will not happen. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-9969) TGT expiration doesn't trigger Kerberos relogin
[ https://issues.apache.org/jira/browse/HADOOP-9969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15944053#comment-15944053 ] Daryn Sharp commented on HADOOP-9969: - Please attach a current stack trace. Glancing at the code, it should be retrying... > TGT expiration doesn't trigger Kerberos relogin > --- > > Key: HADOOP-9969 > URL: https://issues.apache.org/jira/browse/HADOOP-9969 > Project: Hadoop Common > Issue Type: Bug > Components: ipc, security >Affects Versions: 2.1.0-beta, 2.5.0, 2.5.2, 2.6.0, 2.6.1, 2.8.0, 2.7.1, > 2.6.2, 2.6.3 > Environment: IBM JDK7 >Reporter: Yu Gao > Attachments: HADOOP-9969.patch, JobTracker.log > > > In HADOOP-9698 & HADOOP-9850, RPC client and Sasl client have been changed to > respect the auth method advertised from server, instead of blindly attempting > the configured one at client side. However, when TGT has expired, an > exception will be thrown from SaslRpcClient#createSaslClient(SaslAuth > authType), and at this time the authMethod still holds the initial value > which is SIMPLE and never has a chance to be updated with the expected one > requested by server, so kerberos relogin will not happen. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-9969) TGT expiration doesn't trigger Kerberos relogin
[ https://issues.apache.org/jira/browse/HADOOP-9969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15871297#comment-15871297 ] Wen Yuan Chen commented on HADOOP-9969: --- Any update on this issue? I meet the same issue on Hadoop 2.7.3 with IBM JDK 1.8 > TGT expiration doesn't trigger Kerberos relogin > --- > > Key: HADOOP-9969 > URL: https://issues.apache.org/jira/browse/HADOOP-9969 > Project: Hadoop Common > Issue Type: Bug > Components: ipc, security >Affects Versions: 2.1.0-beta, 2.5.0, 2.5.2, 2.6.0, 2.6.1, 2.8.0, 2.7.1, > 2.6.2, 2.6.3 > Environment: IBM JDK7 >Reporter: Yu Gao > Attachments: HADOOP-9969.patch, JobTracker.log > > > In HADOOP-9698 & HADOOP-9850, RPC client and Sasl client have been changed to > respect the auth method advertised from server, instead of blindly attempting > the configured one at client side. However, when TGT has expired, an > exception will be thrown from SaslRpcClient#createSaslClient(SaslAuth > authType), and at this time the authMethod still holds the initial value > which is SIMPLE and never has a chance to be updated with the expected one > requested by server, so kerberos relogin will not happen. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-9969) TGT expiration doesn't trigger Kerberos relogin
[ https://issues.apache.org/jira/browse/HADOOP-9969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15143360#comment-15143360 ] Greg Senia commented on HADOOP-9969: [~acmurthy] can we have a quick discussion on this JIRA to find out what is going on with it.. I think Dan or Beth will work to set something up.. > TGT expiration doesn't trigger Kerberos relogin > --- > > Key: HADOOP-9969 > URL: https://issues.apache.org/jira/browse/HADOOP-9969 > Project: Hadoop Common > Issue Type: Bug > Components: ipc, security >Affects Versions: 2.1.0-beta, 2.5.0, 2.5.2, 2.6.0, 2.6.1, 2.8.0, 2.7.1, > 2.6.2, 2.6.3 > Environment: IBM JDK7 >Reporter: Yu Gao > Attachments: HADOOP-9969.patch, JobTracker.log > > > In HADOOP-9698 & HADOOP-9850, RPC client and Sasl client have been changed to > respect the auth method advertised from server, instead of blindly attempting > the configured one at client side. However, when TGT has expired, an > exception will be thrown from SaslRpcClient#createSaslClient(SaslAuth > authType), and at this time the authMethod still holds the initial value > which is SIMPLE and never has a chance to be updated with the expected one > requested by server, so kerberos relogin will not happen. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-9969) TGT expiration doesn't trigger Kerberos relogin
[ https://issues.apache.org/jira/browse/HADOOP-9969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15139596#comment-15139596 ] Greg Senia commented on HADOOP-9969: [~daryn] I have reached out to IBM JDK Security team to try to get info on if IBM is doing it correctly.. I patched my HDP build from HWX and it seems to solve the issues.. But waiting to hear from IBM JDK folks... Any other info on plans to integrate this into the Core Hadoop build would be great.. thanks > TGT expiration doesn't trigger Kerberos relogin > --- > > Key: HADOOP-9969 > URL: https://issues.apache.org/jira/browse/HADOOP-9969 > Project: Hadoop Common > Issue Type: Bug > Components: ipc, security >Affects Versions: 2.1.0-beta, 2.5.0, 2.5.2, 2.6.0, 2.6.1, 2.8.0, 2.7.1, > 2.6.2, 2.6.3 > Environment: IBM JDK7 >Reporter: Yu Gao > Attachments: HADOOP-9969.patch, JobTracker.log > > > In HADOOP-9698 & HADOOP-9850, RPC client and Sasl client have been changed to > respect the auth method advertised from server, instead of blindly attempting > the configured one at client side. However, when TGT has expired, an > exception will be thrown from SaslRpcClient#createSaslClient(SaslAuth > authType), and at this time the authMethod still holds the initial value > which is SIMPLE and never has a chance to be updated with the expected one > requested by server, so kerberos relogin will not happen. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-9969) TGT expiration doesn't trigger Kerberos relogin
[ https://issues.apache.org/jira/browse/HADOOP-9969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15137124#comment-15137124 ] Greg Senia commented on HADOOP-9969: This also affects IBM JDK8... > TGT expiration doesn't trigger Kerberos relogin > --- > > Key: HADOOP-9969 > URL: https://issues.apache.org/jira/browse/HADOOP-9969 > Project: Hadoop Common > Issue Type: Bug > Components: ipc, security >Affects Versions: 2.1.0-beta > Environment: IBM JDK7 >Reporter: Yu Gao > Attachments: HADOOP-9969.patch, JobTracker.log > > > In HADOOP-9698 & HADOOP-9850, RPC client and Sasl client have been changed to > respect the auth method advertised from server, instead of blindly attempting > the configured one at client side. However, when TGT has expired, an > exception will be thrown from SaslRpcClient#createSaslClient(SaslAuth > authType), and at this time the authMethod still holds the initial value > which is SIMPLE and never has a chance to be updated with the expected one > requested by server, so kerberos relogin will not happen. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-9969) TGT expiration doesn't trigger Kerberos relogin
[ https://issues.apache.org/jira/browse/HADOOP-9969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15137486#comment-15137486 ] Greg Senia commented on HADOOP-9969: [~crystal_gaoyu] and [~xinwei] I noticed it's stated that there are some other side-effects? Please advise. > TGT expiration doesn't trigger Kerberos relogin > --- > > Key: HADOOP-9969 > URL: https://issues.apache.org/jira/browse/HADOOP-9969 > Project: Hadoop Common > Issue Type: Bug > Components: ipc, security >Affects Versions: 2.1.0-beta, 2.6.0, 2.6.1, 2.8.0, 2.7.2, 2.6.3 > Environment: IBM JDK7 >Reporter: Yu Gao > Attachments: HADOOP-9969.patch, JobTracker.log > > > In HADOOP-9698 & HADOOP-9850, RPC client and Sasl client have been changed to > respect the auth method advertised from server, instead of blindly attempting > the configured one at client side. However, when TGT has expired, an > exception will be thrown from SaslRpcClient#createSaslClient(SaslAuth > authType), and at this time the authMethod still holds the initial value > which is SIMPLE and never has a chance to be updated with the expected one > requested by server, so kerberos relogin will not happen. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-9969) TGT expiration doesn't trigger Kerberos relogin
[
https://issues.apache.org/jira/browse/HADOOP-9969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14906333#comment-14906333
]
Xinwei Qin commented on HADOOP-9969:
-
Hi, [~crystal_gaoyu], [~daryn],
This bug still exists with IBM JDK7, but will not happen with Oracle JDK. The
exceptions thrown by IBM JDK and Oracle JDK are different:
IBM JDK (*Failure to initialize security context [Caused by
org.ietf.jgss.GSSException*):
{code}
2015-06-01 17:55:40,448 DEBUG security.SaslRpcClient
(SaslRpcClient.java:createSaslClient(247)) - Creating SASL GSSAPI(KERBEROS)
client to authenticate to service at hadoop.hadoop.com
2015-06-01 17:55:40,470 DEBUG security.UserGroupInformation
(UserGroupInformation.java:doAs(1645)) - PrivilegedActionException
as:nsbig...@hadoop.com (auth:KERBEROS) cause:javax.security.sasl.SaslException:
Failure to initialize security context [Caused by org.ietf.jgss.GSSException，
major code: 8, minor code: 0
major string: Credential expired
minor string: Kerberos credential has expired]
2015-06-01 17:55:40,472 DEBUG security.UserGroupInformation
(UserGroupInformation.java:logPrivilegedAction(1665)) - PrivilegedAction
as:nsbig...@hadoop.com (auth:KERBEROS)
from:org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:654)
2015-06-01 17:55:40,472 DEBUG ipc.Client
(Client.java:shouldAuthenticateOverKrb(551)) - this.authMethod !=
SaslRpcServer.AuthMethod.KERBEROS
2015-06-01 17:55:40,473 WARN ipc.Client (Client.java:run(686)) - Exception
encountered while connecting to the server :
{color:red}javax.security.sasl.SaslException: Failure to initialize security
context [Caused by org.ietf.jgss.GSSException{color}, major code: 8, minor
code: 0
major string: Credential expired
minor string: Kerberos credential has expired]]
{code}
Oracle JDK (*GSS initiate failed [Caused by GSSException: No valid credentials
provided*):
{code}
2015-06-01 18:31:24,441 DEBUG [main]: PrivilegedActionException
as:nsbig...@hadoop.com (auth:KERBEROS) cause:javax.security.sasl.SaslException:
GSS initiate failed [Caused by GSSException: No valid credentials provided
(Mechanism level: Failed to find any Kerberos tgt)]
org.apache.hadoop.security.UserGroupInformation(1645)
2015-06-01 18:31:24,442 DEBUG [main]: PrivilegedAction as:nsbig...@hadoop.com
(auth:KERBEROS)
from:org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:654)
org.apache.hadoop.security.UserGroupInformation(1665)
2015-06-01 18:31:24,442 DEBUG [main]: this.authMethod == AuthMethod.KERBEROS
org.apache.hadoop.ipc.Client(535)
{code}
The reason is:
when TGT expired, {{javax.security.sasl.Sasl#createSaslClient()}} of IBM JDK
will throw {{SaslException}}, but the method of Oracle JDK will not. The
{{SaslException}} was thrown by {{saslClient#evaluateChallenge()}} with Oracle
JDK. So, the client can handle the failure with Oracle JDK but cannot with IBM
JDK.
I am confused with the reason why the exceptions are different between IBM JDK
and Oracle JDK. Any thought about it? Can we make
{{javax.security.sasl.Sasl#createSaslClient()}} of IBM JDK return success?
The HADOOP-9969.patch can fix this bug, but also has some other side-effects.
> TGT expiration doesn't trigger Kerberos relogin
> ---
>
> Key: HADOOP-9969
> URL: https://issues.apache.org/jira/browse/HADOOP-9969
> Project: Hadoop Common
> Issue Type: Bug
> Components: ipc, security
>Affects Versions: 2.1.0-beta
>Reporter: Yu Gao
> Attachments: HADOOP-9969.patch, JobTracker.log
>
>
> In HADOOP-9698 & HADOOP-9850, RPC client and Sasl client have been changed to
> respect the auth method advertised from server, instead of blindly attempting
> the configured one at client side. However, when TGT has expired, an
> exception will be thrown from SaslRpcClient#createSaslClient(SaslAuth
> authType), and at this time the authMethod still holds the initial value
> which is SIMPLE and never has a chance to be updated with the expected one
> requested by server, so kerberos relogin will not happen.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-9969) TGT expiration doesn't trigger Kerberos relogin
[ https://issues.apache.org/jira/browse/HADOOP-9969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14907372#comment-14907372 ] Yu Gao commented on HADOOP-9969: This is because IBM JDK behaves differently when initializing SaslClient in Sasl.createSaslClient, which requires valid kerberos credentials in place, even before the server and client start the negotiation. While Oracle JDK seems not checking credentials until evaluateChallenge is called. > TGT expiration doesn't trigger Kerberos relogin > --- > > Key: HADOOP-9969 > URL: https://issues.apache.org/jira/browse/HADOOP-9969 > Project: Hadoop Common > Issue Type: Bug > Components: ipc, security >Affects Versions: 2.1.0-beta >Reporter: Yu Gao > Attachments: HADOOP-9969.patch, JobTracker.log > > > In HADOOP-9698 & HADOOP-9850, RPC client and Sasl client have been changed to > respect the auth method advertised from server, instead of blindly attempting > the configured one at client side. However, when TGT has expired, an > exception will be thrown from SaslRpcClient#createSaslClient(SaslAuth > authType), and at this time the authMethod still holds the initial value > which is SIMPLE and never has a chance to be updated with the expected one > requested by server, so kerberos relogin will not happen. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-9969) TGT expiration doesn't trigger Kerberos relogin
[ https://issues.apache.org/jira/browse/HADOOP-9969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13773289#comment-13773289 ] Yu Gao commented on HADOOP-9969: Sure. Attaching a relevant sub-section of JobTracker log throwing the expiration exception. I'm using Hadoop 2.1.0-beta + MapReduce 1.1.1, with IBM JDK 6. TGT expiration doesn't trigger Kerberos relogin --- Key: HADOOP-9969 URL: https://issues.apache.org/jira/browse/HADOOP-9969 Project: Hadoop Common Issue Type: Bug Components: ipc, security Affects Versions: 2.1.0-beta Reporter: Yu Gao Attachments: HADOOP-9969.patch In HADOOP-9698 HADOOP-9850, RPC client and Sasl client have been changed to respect the auth method advertised from server, instead of blindly attempting the configured one at client side. However, when TGT has expired, an exception will be thrown from SaslRpcClient#createSaslClient(SaslAuth authType), and at this time the authMethod still holds the initial value which is SIMPLE and never has a chance to be updated with the expected one requested by server, so kerberos relogin will not happen. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9969) TGT expiration doesn't trigger Kerberos relogin
[ https://issues.apache.org/jira/browse/HADOOP-9969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13769539#comment-13769539 ] Daryn Sharp commented on HADOOP-9969: - HADOOP-9850 already records the auth being attempted so the sasl failure loop can tell if kerberos is being attempted. We saw this issue internally and 9850 did indeed fix the issue for us. Would you please attach (please don't post inline) a log with client debugging enabled? TGT expiration doesn't trigger Kerberos relogin --- Key: HADOOP-9969 URL: https://issues.apache.org/jira/browse/HADOOP-9969 Project: Hadoop Common Issue Type: Bug Components: ipc, security Affects Versions: 2.1.0-beta Reporter: Yu Gao Attachments: HADOOP-9969.patch In HADOOP-9698 HADOOP-9850, RPC client and Sasl client have been changed to respect the auth method advertised from server, instead of blindly attempting the configured one at client side. However, when TGT has expired, an exception will be thrown from SaslRpcClient#createSaslClient(SaslAuth authType), and at this time the authMethod still holds the initial value which is SIMPLE and never has a chance to be updated with the expected one requested by server, so kerberos relogin will not happen. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9969) TGT expiration doesn't trigger Kerberos relogin
[ https://issues.apache.org/jira/browse/HADOOP-9969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13769020#comment-13769020 ] Yu Gao commented on HADOOP-9969: When TGT expired, client trying to access NameNode got this error: WARN org.apache.hadoop.ipc.Client: Exception encountered while connecting to the server : javax.security.sasl.SaslException: Failure to initialize security context [Caused by org.ietf.jgss.GSSException, major code: 8, minor code: 0 major string: Credential expired minor string: Kerberos credential has expired] And method org.apache.hadoop.ipc.Client.Connection.shouldAuthenticateOverKrb()returned false since the authMethod got from sasl client was SIMPLE, so relogin never happened TGT expiration doesn't trigger Kerberos relogin --- Key: HADOOP-9969 URL: https://issues.apache.org/jira/browse/HADOOP-9969 Project: Hadoop Common Issue Type: Bug Components: ipc, security Affects Versions: 2.1.0-beta Reporter: Yu Gao In HADOOP 9698 HADOOP 9850, RPC client and Sasl client have been changed to respect the auth method advertised from server, instead of blindly attempting the configured one at client side. However, when TGT has expired, an exception will be thrown from SaslRpcClient#createSaslClient(SaslAuth authType), and at this time the authMethod still holds the initial value which is SIMPLE and never has a chance to be updated with the expected one requested by server, so kerberos relogin will not happen. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9969) TGT expiration doesn't trigger Kerberos relogin
[ https://issues.apache.org/jira/browse/HADOOP-9969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13769034#comment-13769034 ] Yu Gao commented on HADOOP-9969: A quick fix would be updating field authMethod of class SaslRpcClient with the current auth type being attempted in the given list, before calling SaslRpcClient#createSaslClient(SaslAuth authType) in method SaslRpcClient#selectSaslClient(ListSaslAuth authTypes). Attaching the patch. TGT expiration doesn't trigger Kerberos relogin --- Key: HADOOP-9969 URL: https://issues.apache.org/jira/browse/HADOOP-9969 Project: Hadoop Common Issue Type: Bug Components: ipc, security Affects Versions: 2.1.0-beta Reporter: Yu Gao Attachments: HADOOP-9969.patch In HADOOP-9698 HADOOP-9850, RPC client and Sasl client have been changed to respect the auth method advertised from server, instead of blindly attempting the configured one at client side. However, when TGT has expired, an exception will be thrown from SaslRpcClient#createSaslClient(SaslAuth authType), and at this time the authMethod still holds the initial value which is SIMPLE and never has a chance to be updated with the expected one requested by server, so kerberos relogin will not happen. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
