[
https://issues.apache.org/jira/browse/AMBARI-8840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14875851#comment-14875851
]
Diana Prichici commented on AMBARI-8840:
----------------------------------------
Similiar situation on a cluster upgraded from Ambari 1.7 to Ambari 2.3 and HDP
2.2.x to HDP 2.3 with security already enabled. Services started and everything
appeared to work all right until keytabs were regenerated, whereupon we
observed failures in ZKFC/Namenode communication due to missing encryption type
AES256. Checking the nn.service.keytab, AES256 is not listed. JCE policy files
were already installed on all nodes and Ambari server had since been restarted.
Recovery steps were to run: ambari-server setup-security > Setup Ambari
kerberos JAAS configuration
> Keytabs need to be created to include the encryption type of AES256 CTS mode
> with HMAC SHA1-96
> ----------------------------------------------------------------------------------------------
>
> Key: AMBARI-8840
> URL: https://issues.apache.org/jira/browse/AMBARI-8840
> Project: Ambari
> Issue Type: Bug
> Components: ambari-server
> Affects Versions: 2.0.0, 2.1.0
> Environment: Red Hat Enterprise Linux Server release 6.6 (Santiago)
> [root@hdtest253 etc]# java -version
> java version "1.7.0_79"
> OpenJDK Runtime Environment (rhel-2.5.5.3.el6_6-x86_64 u79-b14)
> OpenJDK 64-Bit Server VM (build 24.79-b02, mixed mode)
> Reporter: Robert Levas
> Assignee: Robert Levas
> Priority: Critical
> Labels: kerberos, keytabs
> Fix For: 2.1.2
>
> Attachments: hadoop-hdfs-journalnode-hdtest253.svl.ibm.com.log
>
>
> During automated keytab generation, an entry with the following encryption
> type must be added else certain services will fail to start up or properly
> when Kerberos is enabled:
> {code}AES256 CTS mode with HMAC SHA1-96{code}
> For example, NAMENODE will fail with the following errors:
> {code}
> 2014-12-19 21:45:56,101 WARN server.AuthenticationFilter
> (AuthenticationFilter.java:doFilter(551)) - Authentication exception:
> GSSException: Failure unspecified at GSS-API level (Mechanism level: Invalid
> argument (400) - Cannot find key of appropriate type to decrypt AP REP -
> AES256 CTS mode with HMAC SHA1-96)
> org.apache.hadoop.security.authentication.client.AuthenticationException:
> GSSException: Failure unspecified at GSS-API level (Mechanism level: Invalid
> argument (400) - Cannot find key of appropriate type to decrypt AP REP -
> AES256 CTS mode with HMAC SHA1-96)
> at
> org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler.authenticate(KerberosAuthenticationHandler.java:399)
> at
> org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:507)
> at
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
> at
> org.apache.hadoop.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1224)
> at
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
> at org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45)
> at
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
> at org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45)
> at
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
> at
> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
> at
> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
> at
> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
> at
> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
> at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
> at
> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
> at
> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
> at org.mortbay.jetty.Server.handle(Server.java:326)
> at
> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
> at
> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
> at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
> at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
> at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
> at
> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
> at
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> Caused by: GSSException: Failure unspecified at GSS-API level (Mechanism
> level: Invalid argument (400) - Cannot find key of appropriate type to
> decrypt AP REP - AES256 CTS mode with HMAC SHA1-96)
> at
> sun.security.jgss.krb5.Krb5Context.acceptSecContext(Krb5Context.java:788)
> at
> sun.security.jgss.GSSContextImpl.acceptSecContext(GSSContextImpl.java:342)
> at
> sun.security.jgss.GSSContextImpl.acceptSecContext(GSSContextImpl.java:285)
> at
> sun.security.jgss.spnego.SpNegoContext.GSS_acceptSecContext(SpNegoContext.java:875)
> at
> sun.security.jgss.spnego.SpNegoContext.acceptSecContext(SpNegoContext.java:548)
> at
> sun.security.jgss.GSSContextImpl.acceptSecContext(GSSContextImpl.java:342)
> at
> sun.security.jgss.GSSContextImpl.acceptSecContext(GSSContextImpl.java:285)
> at
> org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler$2.run(KerberosAuthenticationHandler.java:366)
> at
> org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler$2.run(KerberosAuthenticationHandler.java:348)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler.authenticate(KerberosAuthenticationHandler.java:348)
> ... 23 more
> Caused by: KrbException: Invalid argument (400) - Cannot find key of
> appropriate type to decrypt AP REP - AES256 CTS mode with HMAC SHA1-96
> at sun.security.krb5.KrbApReq.authenticate(KrbApReq.java:273)
> at sun.security.krb5.KrbApReq.<init>(KrbApReq.java:144)
> at
> sun.security.jgss.krb5.InitSecContextToken.<init>(InitSecContextToken.java:108)
> at
> sun.security.jgss.krb5.Krb5Context.acceptSecContext(Krb5Context.java:771)
> ... 34 more
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
