[
https://issues.apache.org/jira/browse/YARN-4813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15196386#comment-15196386
]
Daniel Templeton commented on YARN-4813:
----------------------------------------
The root cause is a replay error in GSS:
{noformat}
2016-03-14 16:39:07,982 DEBUG [1511189171@qtp-1707515013-2 -
/ws/v1/cluster/delegation-token] server.AuthenticationFilter
(AuthenticationFilter.java:doFilter(628)) - Authentication exception:
GSSException: Failure unspecified at GSS-API level (Mechanism level: Request is
a replay (34))
org.apache.hadoop.security.authentication.client.AuthenticationException:
GSSException: Failure unspecified at GSS-API level (Mechanism level: Request is
a replay (34))
at
org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler.authenticate(KerberosAuthenticationHandler.java:400)
at
org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticationHandler.authenticate(DelegationTokenAuthenticationHandler.java:348)
at
org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:562)
at
org.apache.hadoop.yarn.server.security.http.RMAuthenticationFilter.doFilter(RMAuthenticationFilter.java:82)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at
org.apache.hadoop.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1249)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at
org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
at
org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at
org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
at
org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
at
org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
at
org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:322)
at
org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
at
org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:945)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:756)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
at
org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
at
org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
Caused by: GSSException: Failure unspecified at GSS-API level (Mechanism level:
Request is a replay (34))
at
sun.security.jgss.krb5.Krb5Context.acceptSecContext(Krb5Context.java:788)
at
sun.security.jgss.GSSContextImpl.acceptSecContext(GSSContextImpl.java:342)
at
sun.security.jgss.GSSContextImpl.acceptSecContext(GSSContextImpl.java:285)
at
sun.security.jgss.spnego.SpNegoContext.GSS_acceptSecContext(SpNegoContext.java:871)
at
sun.security.jgss.spnego.SpNegoContext.acceptSecContext(SpNegoContext.java:544)
at
sun.security.jgss.GSSContextImpl.acceptSecContext(GSSContextImpl.java:342)
at
sun.security.jgss.GSSContextImpl.acceptSecContext(GSSContextImpl.java:285)
at
org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler$2.run(KerberosAuthenticationHandler.java:367)
at
org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler$2.run(KerberosAuthenticationHandler.java:349)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler.authenticate(KerberosAuthenticationHandler.java:349)
... 25 more
Caused by: KrbException: Request is a replay (34)
at sun.security.krb5.KrbApReq.authenticate(KrbApReq.java:308)
at sun.security.krb5.KrbApReq.<init>(KrbApReq.java:144)
at
sun.security.jgss.krb5.InitSecContextToken.<init>(InitSecContextToken.java:108)
at
sun.security.jgss.krb5.Krb5Context.acceptSecContext(Krb5Context.java:771)
{noformat}
My best guess is that the test suite happens to run fast enough that
consecutive requests end up looking like replays. I don't know why else this
error would be intermittent.
> TestRMWebServicesDelegationTokenAuthentication.testDoAs fails intermittently
> ----------------------------------------------------------------------------
>
> Key: YARN-4813
> URL: https://issues.apache.org/jira/browse/YARN-4813
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
> Affects Versions: 2.9.0
> Reporter: Daniel Templeton
>
> {noformat}
> -------------------------------------------------------
> T E S T S
> -------------------------------------------------------
> Running
> org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesDelegationTokenAuthentication
> Tests run: 8, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 11.627 sec
> <<< FAILURE! - in
> org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesDelegationTokenAuthentication
> testDoAs[0](org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesDelegationTokenAuthentication)
> Time elapsed: 0.208 sec <<< ERROR!
> java.io.IOException: Server returned HTTP response code: 403 for URL:
> http://localhost:8088/ws/v1/cluster/delegation-token
> at
> sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1626)
> at
> org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesDelegationTokenAuthentication$3.call(TestRMWebServicesDelegationTokenAuthentication.java:407)
> at
> org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesDelegationTokenAuthentication$3.call(TestRMWebServicesDelegationTokenAuthentication.java:398)
> at
> org.apache.hadoop.security.authentication.KerberosTestUtils$1.run(KerberosTestUtils.java:120)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.authentication.KerberosTestUtils.doAs(KerberosTestUtils.java:117)
> at
> org.apache.hadoop.security.authentication.KerberosTestUtils.doAsClient(KerberosTestUtils.java:133)
> at
> org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesDelegationTokenAuthentication.getDelegationToken(TestRMWebServicesDelegationTokenAuthentication.java:398)
> at
> org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesDelegationTokenAuthentication.testDoAs(TestRMWebServicesDelegationTokenAuthentication.java:357)
> Results :
> Tests in error:
>
> TestRMWebServicesDelegationTokenAuthentication.testDoAs:357->getDelegationToken:398
> ยป IO
> Tests run: 8, Failures: 0, Errors: 1, Skipped: 0
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
