[jira] [Commented] (HADOOP-15487) ConcurrentModificationException resulting in Kerberos authentication error.
[ https://issues.apache.org/jira/browse/HADOOP-15487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16580170#comment-16580170 ] Daryn Sharp commented on HADOOP-15487: -- Is this still a valid issue? If it is: bq. We saw a few GSSException in the NN log, but only one threw the ConcurrentModificationException. What were the pre-cursor exceptions and related log messages, ie. (re)login/logout messages. bq. This NN had a failover, which is caused by ZKFC having GSSException too. Suspect it's related issue. I suspect curator/ZK is logging out the UGI's subject instead of the one it created. > ConcurrentModificationException resulting in Kerberos authentication error. > --- > > Key: HADOOP-15487 > URL: https://issues.apache.org/jira/browse/HADOOP-15487 > Project: Hadoop Common > Issue Type: Bug > Environment: CDH 5.13.3. Kerberized, Hadoop-HA, jdk1.8.0_152 >Reporter: Wei-Chiu Chuang >Priority: Major > > We found the following exception message in a NameNode log. It seems the > ConcurrentModificationException caused Kerberos authentication error. > It appears to be a JDK bug, similar to HADOOP-13433 (Race in > UGI.reloginFromKeytab) but the version of Hadoop (CDH5.13.3) already patched > HADOOP-13433. (The stacktrace also differs) This cluster runs on JDK > 1.8.0_152. > {noformat} > 2018-05-19 04:00:00,182 WARN org.apache.hadoop.security.UserGroupInformation: > PriviledgedActionException as:hdfs/no...@example.com (auth:KERBEROS) > cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by > GSSException: No valid credentials provided (Mechanism level: Failed to find > any Kerberos tgt)] > 2018-05-19 04:00:00,183 INFO org.apache.hadoop.ipc.Server: Socket Reader #1 > for port 8020: readAndProcess from client 10.16.20.122 threw exception > [java.util.ConcurrentModificationException] > java.util.ConcurrentModificationException > at > java.util.LinkedList$ListItr.checkForComodification(LinkedList.java:966) > at java.util.LinkedList$ListItr.next(LinkedList.java:888) > at javax.security.auth.Subject$SecureSet$1.next(Subject.java:1070) > at javax.security.auth.Subject$ClassSet$1.run(Subject.java:1401) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject$ClassSet.populateSet(Subject.java:1399) > at javax.security.auth.Subject$ClassSet.(Subject.java:1372) > at javax.security.auth.Subject.getPrivateCredentials(Subject.java:767) > at > sun.security.jgss.krb5.SubjectComber.findAux(SubjectComber.java:127) > at > sun.security.jgss.krb5.SubjectComber.findMany(SubjectComber.java:69) > at > sun.security.jgss.krb5.ServiceCreds.getInstance(ServiceCreds.java:96) > at sun.security.jgss.krb5.Krb5Util.getServiceCreds(Krb5Util.java:203) > at > sun.security.jgss.krb5.Krb5AcceptCredential$1.run(Krb5AcceptCredential.java:74) > at > sun.security.jgss.krb5.Krb5AcceptCredential$1.run(Krb5AcceptCredential.java:72) > at java.security.AccessController.doPrivileged(Native Method) > at > sun.security.jgss.krb5.Krb5AcceptCredential.getInstance(Krb5AcceptCredential.java:71) > at > sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:127) > at > sun.security.jgss.GSSManagerImpl.getCredentialElement(GSSManagerImpl.java:193) > at sun.security.jgss.GSSCredentialImpl.add(GSSCredentialImpl.java:427) > at > sun.security.jgss.GSSCredentialImpl.(GSSCredentialImpl.java:62) > at > sun.security.jgss.GSSManagerImpl.createCredential(GSSManagerImpl.java:154) > at > com.sun.security.sasl.gsskerb.GssKrb5Server.(GssKrb5Server.java:108) > at > com.sun.security.sasl.gsskerb.FactoryImpl.createSaslServer(FactoryImpl.java:85) > at > org.apache.hadoop.security.SaslRpcServer$FastSaslServerFactory.createSaslServer(SaslRpcServer.java:398) > at > org.apache.hadoop.security.SaslRpcServer$1.run(SaslRpcServer.java:164) > at > org.apache.hadoop.security.SaslRpcServer$1.run(SaslRpcServer.java:161) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920) > at > org.apache.hadoop.security.SaslRpcServer.create(SaslRpcServer.java:160) > at > org.apache.hadoop.ipc.Server$Connection.createSaslServer(Server.java:1742) > at > org.apache.hadoop.ipc.Server$Connection.processSaslMessage(Server.java:1522) > at > org.apache.hadoop.ipc.Server$Connection.saslProcess(Server.java:1433) > at >
[jira] [Commented] (HADOOP-15487) ConcurrentModificationException resulting in Kerberos authentication error.
[ https://issues.apache.org/jira/browse/HADOOP-15487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16514352#comment-16514352 ] Wei-Chiu Chuang commented on HADOOP-15487: -- FYI I saw yet another Kerberos exception. It may or may not be related. This one was seen on a JN on CDH6.0.0-beta (Apache Hadoop 3.0.0) cluster on JDK 8u121 2018-06-07 11:19:41,111 INFO org.apache.hadoop.ipc.Server: IPC Server handler 2 on 8485 caught an exception java.lang.NullPointerException at com.sun.security.sasl.gsskerb.GssKrb5Base.wrap(GssKrb5Base.java:103) at org.apache.hadoop.ipc.Server.wrapWithSasl(Server.java:2560) at org.apache.hadoop.ipc.Server.access$2400(Server.java:136) at org.apache.hadoop.ipc.Server$Responder.doRespond(Server.java:1178) at org.apache.hadoop.ipc.Server$Connection.sendResponse(Server.java:2132) at org.apache.hadoop.ipc.Server$Connection.access$400(Server.java:1251) at org.apache.hadoop.ipc.Server$Call.sendResponse(Server.java:642) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2268) According to this post https://community.cloudera.com/t5/Storage-Random-Access-HDFS/Name-nodes-crash-due-to-journal-timeouts/m-p/47107 {quote} The NPE in GssKrb5Base seems to be just a sign that the connection was closed. I suspect this is just a consequence of the NN shutting down without closing a connection. {quote} > ConcurrentModificationException resulting in Kerberos authentication error. > --- > > Key: HADOOP-15487 > URL: https://issues.apache.org/jira/browse/HADOOP-15487 > Project: Hadoop Common > Issue Type: Bug > Environment: CDH 5.13.3. Kerberized, Hadoop-HA, jdk1.8.0_152 >Reporter: Wei-Chiu Chuang >Priority: Major > > We found the following exception message in a NameNode log. It seems the > ConcurrentModificationException caused Kerberos authentication error. > It appears to be a JDK bug, similar to HADOOP-13433 (Race in > UGI.reloginFromKeytab) but the version of Hadoop (CDH5.13.3) already patched > HADOOP-13433. (The stacktrace also differs) This cluster runs on JDK > 1.8.0_152. > {noformat} > 2018-05-19 04:00:00,182 WARN org.apache.hadoop.security.UserGroupInformation: > PriviledgedActionException as:hdfs/no...@example.com (auth:KERBEROS) > cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by > GSSException: No valid credentials provided (Mechanism level: Failed to find > any Kerberos tgt)] > 2018-05-19 04:00:00,183 INFO org.apache.hadoop.ipc.Server: Socket Reader #1 > for port 8020: readAndProcess from client 10.16.20.122 threw exception > [java.util.ConcurrentModificationException] > java.util.ConcurrentModificationException > at > java.util.LinkedList$ListItr.checkForComodification(LinkedList.java:966) > at java.util.LinkedList$ListItr.next(LinkedList.java:888) > at javax.security.auth.Subject$SecureSet$1.next(Subject.java:1070) > at javax.security.auth.Subject$ClassSet$1.run(Subject.java:1401) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject$ClassSet.populateSet(Subject.java:1399) > at javax.security.auth.Subject$ClassSet.(Subject.java:1372) > at javax.security.auth.Subject.getPrivateCredentials(Subject.java:767) > at > sun.security.jgss.krb5.SubjectComber.findAux(SubjectComber.java:127) > at > sun.security.jgss.krb5.SubjectComber.findMany(SubjectComber.java:69) > at > sun.security.jgss.krb5.ServiceCreds.getInstance(ServiceCreds.java:96) > at sun.security.jgss.krb5.Krb5Util.getServiceCreds(Krb5Util.java:203) > at > sun.security.jgss.krb5.Krb5AcceptCredential$1.run(Krb5AcceptCredential.java:74) > at > sun.security.jgss.krb5.Krb5AcceptCredential$1.run(Krb5AcceptCredential.java:72) > at java.security.AccessController.doPrivileged(Native Method) > at > sun.security.jgss.krb5.Krb5AcceptCredential.getInstance(Krb5AcceptCredential.java:71) > at > sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:127) > at > sun.security.jgss.GSSManagerImpl.getCredentialElement(GSSManagerImpl.java:193) > at sun.security.jgss.GSSCredentialImpl.add(GSSCredentialImpl.java:427) > at > sun.security.jgss.GSSCredentialImpl.(GSSCredentialImpl.java:62) > at > sun.security.jgss.GSSManagerImpl.createCredential(GSSManagerImpl.java:154) > at > com.sun.security.sasl.gsskerb.GssKrb5Server.(GssKrb5Server.java:108) > at > com.sun.security.sasl.gsskerb.FactoryImpl.createSaslServer(FactoryImpl.java:85) > at > org.apache.hadoop.security.SaslRpcServer$FastSaslServerFactory.createSaslServer(SaslRpcServer.java:398) > at >
[jira] [Commented] (HADOOP-15487) ConcurrentModificationException resulting in Kerberos authentication error.
[ https://issues.apache.org/jira/browse/HADOOP-15487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16514346#comment-16514346 ] Wei-Chiu Chuang commented on HADOOP-15487: -- The original exception was seen on a CDH5.13.3 (Apache Hadoop 2.6.0 + many, many pathces). Server recovered for this specific exception. > ConcurrentModificationException resulting in Kerberos authentication error. > --- > > Key: HADOOP-15487 > URL: https://issues.apache.org/jira/browse/HADOOP-15487 > Project: Hadoop Common > Issue Type: Bug > Environment: CDH 5.13.3. Kerberized, Hadoop-HA, jdk1.8.0_152 >Reporter: Wei-Chiu Chuang >Priority: Major > > We found the following exception message in a NameNode log. It seems the > ConcurrentModificationException caused Kerberos authentication error. > It appears to be a JDK bug, similar to HADOOP-13433 (Race in > UGI.reloginFromKeytab) but the version of Hadoop (CDH5.13.3) already patched > HADOOP-13433. (The stacktrace also differs) This cluster runs on JDK > 1.8.0_152. > {noformat} > 2018-05-19 04:00:00,182 WARN org.apache.hadoop.security.UserGroupInformation: > PriviledgedActionException as:hdfs/no...@example.com (auth:KERBEROS) > cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by > GSSException: No valid credentials provided (Mechanism level: Failed to find > any Kerberos tgt)] > 2018-05-19 04:00:00,183 INFO org.apache.hadoop.ipc.Server: Socket Reader #1 > for port 8020: readAndProcess from client 10.16.20.122 threw exception > [java.util.ConcurrentModificationException] > java.util.ConcurrentModificationException > at > java.util.LinkedList$ListItr.checkForComodification(LinkedList.java:966) > at java.util.LinkedList$ListItr.next(LinkedList.java:888) > at javax.security.auth.Subject$SecureSet$1.next(Subject.java:1070) > at javax.security.auth.Subject$ClassSet$1.run(Subject.java:1401) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject$ClassSet.populateSet(Subject.java:1399) > at javax.security.auth.Subject$ClassSet.(Subject.java:1372) > at javax.security.auth.Subject.getPrivateCredentials(Subject.java:767) > at > sun.security.jgss.krb5.SubjectComber.findAux(SubjectComber.java:127) > at > sun.security.jgss.krb5.SubjectComber.findMany(SubjectComber.java:69) > at > sun.security.jgss.krb5.ServiceCreds.getInstance(ServiceCreds.java:96) > at sun.security.jgss.krb5.Krb5Util.getServiceCreds(Krb5Util.java:203) > at > sun.security.jgss.krb5.Krb5AcceptCredential$1.run(Krb5AcceptCredential.java:74) > at > sun.security.jgss.krb5.Krb5AcceptCredential$1.run(Krb5AcceptCredential.java:72) > at java.security.AccessController.doPrivileged(Native Method) > at > sun.security.jgss.krb5.Krb5AcceptCredential.getInstance(Krb5AcceptCredential.java:71) > at > sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:127) > at > sun.security.jgss.GSSManagerImpl.getCredentialElement(GSSManagerImpl.java:193) > at sun.security.jgss.GSSCredentialImpl.add(GSSCredentialImpl.java:427) > at > sun.security.jgss.GSSCredentialImpl.(GSSCredentialImpl.java:62) > at > sun.security.jgss.GSSManagerImpl.createCredential(GSSManagerImpl.java:154) > at > com.sun.security.sasl.gsskerb.GssKrb5Server.(GssKrb5Server.java:108) > at > com.sun.security.sasl.gsskerb.FactoryImpl.createSaslServer(FactoryImpl.java:85) > at > org.apache.hadoop.security.SaslRpcServer$FastSaslServerFactory.createSaslServer(SaslRpcServer.java:398) > at > org.apache.hadoop.security.SaslRpcServer$1.run(SaslRpcServer.java:164) > at > org.apache.hadoop.security.SaslRpcServer$1.run(SaslRpcServer.java:161) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920) > at > org.apache.hadoop.security.SaslRpcServer.create(SaslRpcServer.java:160) > at > org.apache.hadoop.ipc.Server$Connection.createSaslServer(Server.java:1742) > at > org.apache.hadoop.ipc.Server$Connection.processSaslMessage(Server.java:1522) > at > org.apache.hadoop.ipc.Server$Connection.saslProcess(Server.java:1433) > at > org.apache.hadoop.ipc.Server$Connection.saslReadAndProcess(Server.java:1396) > at > org.apache.hadoop.ipc.Server$Connection.processRpcOutOfBandRequest(Server.java:2080) > at > org.apache.hadoop.ipc.Server$Connection.processOneRpc(Server.java:1920) > at >
[jira] [Commented] (HADOOP-15487) ConcurrentModificationException resulting in Kerberos authentication error.
[ https://issues.apache.org/jira/browse/HADOOP-15487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16500348#comment-16500348 ] Daryn Sharp commented on HADOOP-15487: -- The second exception is an unrelated jdk bug fixed in 8u161. [JDK-8170278: ticket renewal won't happen with debugging turned on|https://bugs.openjdk.java.net/browse/JDK-8170278]. The gssapi is smart recognizes and handles expired tickets from a keytab. The problem is {{KerberosTicket#toString}} throws the ISE if it's expired. Easy workaround is don't enable debug logging. The original issue is distinct. If there truly are no custom plugins, it may be related to curator/zookeeper/AuthenticatedURL. What is the specific apache release? Did the server recover? We may need to consider using a distinct subject/ugi for rpc servers to prevent other code munging our JASS, but there are a few possible grues lurking there. > ConcurrentModificationException resulting in Kerberos authentication error. > --- > > Key: HADOOP-15487 > URL: https://issues.apache.org/jira/browse/HADOOP-15487 > Project: Hadoop Common > Issue Type: Bug > Environment: CDH 5.13.3. Kerberized, Hadoop-HA, jdk1.8.0_152 >Reporter: Wei-Chiu Chuang >Priority: Major > > We found the following exception message in a NameNode log. It seems the > ConcurrentModificationException caused Kerberos authentication error. > It appears to be a JDK bug, similar to HADOOP-13433 (Race in > UGI.reloginFromKeytab) but the version of Hadoop (CDH5.13.3) already patched > HADOOP-13433. (The stacktrace also differs) This cluster runs on JDK > 1.8.0_152. > {noformat} > 2018-05-19 04:00:00,182 WARN org.apache.hadoop.security.UserGroupInformation: > PriviledgedActionException as:hdfs/no...@example.com (auth:KERBEROS) > cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by > GSSException: No valid credentials provided (Mechanism level: Failed to find > any Kerberos tgt)] > 2018-05-19 04:00:00,183 INFO org.apache.hadoop.ipc.Server: Socket Reader #1 > for port 8020: readAndProcess from client 10.16.20.122 threw exception > [java.util.ConcurrentModificationException] > java.util.ConcurrentModificationException > at > java.util.LinkedList$ListItr.checkForComodification(LinkedList.java:966) > at java.util.LinkedList$ListItr.next(LinkedList.java:888) > at javax.security.auth.Subject$SecureSet$1.next(Subject.java:1070) > at javax.security.auth.Subject$ClassSet$1.run(Subject.java:1401) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject$ClassSet.populateSet(Subject.java:1399) > at javax.security.auth.Subject$ClassSet.(Subject.java:1372) > at javax.security.auth.Subject.getPrivateCredentials(Subject.java:767) > at > sun.security.jgss.krb5.SubjectComber.findAux(SubjectComber.java:127) > at > sun.security.jgss.krb5.SubjectComber.findMany(SubjectComber.java:69) > at > sun.security.jgss.krb5.ServiceCreds.getInstance(ServiceCreds.java:96) > at sun.security.jgss.krb5.Krb5Util.getServiceCreds(Krb5Util.java:203) > at > sun.security.jgss.krb5.Krb5AcceptCredential$1.run(Krb5AcceptCredential.java:74) > at > sun.security.jgss.krb5.Krb5AcceptCredential$1.run(Krb5AcceptCredential.java:72) > at java.security.AccessController.doPrivileged(Native Method) > at > sun.security.jgss.krb5.Krb5AcceptCredential.getInstance(Krb5AcceptCredential.java:71) > at > sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:127) > at > sun.security.jgss.GSSManagerImpl.getCredentialElement(GSSManagerImpl.java:193) > at sun.security.jgss.GSSCredentialImpl.add(GSSCredentialImpl.java:427) > at > sun.security.jgss.GSSCredentialImpl.(GSSCredentialImpl.java:62) > at > sun.security.jgss.GSSManagerImpl.createCredential(GSSManagerImpl.java:154) > at > com.sun.security.sasl.gsskerb.GssKrb5Server.(GssKrb5Server.java:108) > at > com.sun.security.sasl.gsskerb.FactoryImpl.createSaslServer(FactoryImpl.java:85) > at > org.apache.hadoop.security.SaslRpcServer$FastSaslServerFactory.createSaslServer(SaslRpcServer.java:398) > at > org.apache.hadoop.security.SaslRpcServer$1.run(SaslRpcServer.java:164) > at > org.apache.hadoop.security.SaslRpcServer$1.run(SaslRpcServer.java:161) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920) > at > org.apache.hadoop.security.SaslRpcServer.create(SaslRpcServer.java:160) > at >
[jira] [Commented] (HADOOP-15487) ConcurrentModificationException resulting in Kerberos authentication error.
[ https://issues.apache.org/jira/browse/HADOOP-15487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16489733#comment-16489733 ] Wei-Chiu Chuang commented on HADOOP-15487: -- FYI here comes another that looks eerily similar: This one is from a NameNode on a different cluster, CDH5.13.2, jdk1.8.0_74. {noformat} 2018-05-20 14:01:35,314 INFO org.apache.hadoop.ipc.Server: Socket Reader #1 for port 8020: readAndProcess from client 192.168.30.37 threw exception [java.lang.IllegalStateException: This ticket is no longer valid] java.lang.IllegalStateException: This ticket is no longer valid at javax.security.auth.kerberos.KerberosTicket.toString(KerberosTicket.java:638) at java.lang.String.valueOf(String.java:2994) at java.lang.StringBuilder.append(StringBuilder.java:131) at sun.security.jgss.krb5.SubjectComber.findAux(SubjectComber.java:171) at sun.security.jgss.krb5.SubjectComber.find(SubjectComber.java:61) at sun.security.jgss.krb5.ServiceCreds.getInstance(ServiceCreds.java:127) at sun.security.jgss.krb5.Krb5Util.getServiceCreds(Krb5Util.java:203) at sun.security.jgss.krb5.Krb5AcceptCredential$1.run(Krb5AcceptCredential.java:74) at sun.security.jgss.krb5.Krb5AcceptCredential$1.run(Krb5AcceptCredential.java:72) at java.security.AccessController.doPrivileged(Native Method) at sun.security.jgss.krb5.Krb5AcceptCredential.getInstance(Krb5AcceptCredential.java:71) at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:127) at sun.security.jgss.GSSManagerImpl.getCredentialElement(GSSManagerImpl.java:193) at sun.security.jgss.GSSCredentialImpl.add(GSSCredentialImpl.java:427) at sun.security.jgss.GSSCredentialImpl.(GSSCredentialImpl.java:62) at sun.security.jgss.GSSManagerImpl.createCredential(GSSManagerImpl.java:154) at com.sun.security.sasl.gsskerb.GssKrb5Server.(GssKrb5Server.java:108) at com.sun.security.sasl.gsskerb.FactoryImpl.createSaslServer(FactoryImpl.java:85) at org.apache.hadoop.security.SaslRpcServer$FastSaslServerFactory.createSaslServer(SaslRpcServer.java:398) at org.apache.hadoop.security.SaslRpcServer$1.run(SaslRpcServer.java:164) at org.apache.hadoop.security.SaslRpcServer$1.run(SaslRpcServer.java:161) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920) at org.apache.hadoop.security.SaslRpcServer.create(SaslRpcServer.java:160) at org.apache.hadoop.ipc.Server$Connection.createSaslServer(Server.java:1742) at org.apache.hadoop.ipc.Server$Connection.processSaslMessage(Server.java:1522) at org.apache.hadoop.ipc.Server$Connection.saslProcess(Server.java:1433) at org.apache.hadoop.ipc.Server$Connection.saslReadAndProcess(Server.java:1396) at org.apache.hadoop.ipc.Server$Connection.processRpcOutOfBandRequest(Server.java:2080) at org.apache.hadoop.ipc.Server$Connection.processOneRpc(Server.java:1920) at org.apache.hadoop.ipc.Server$Connection.readAndProcess(Server.java:1682) at org.apache.hadoop.ipc.Server$Listener.doRead(Server.java:896) at org.apache.hadoop.ipc.Server$Listener$Reader.doRunLoop(Server.java:752) at org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:723) 2018-05-20 14:01:35,385 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for u...@example.com (auth:KERBEROS) 2018-05-20 14:01:35,411 INFO SecurityLogger.org.apache.hadoop.security.authorize.ServiceAuthorizationManager: Authorization successful for u...@example.com (auth:KERBEROS) for protocol=interface org .apache.hadoop.hdfs.protocol.ClientProtocol 2018-05-20 14:01:35,545 WARN org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:hdfs/nn1.example@example.com (auth:KERBEROS) cause:javax.security.sasl.SaslExcept ion: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)] 2018-05-20 14:01:35,545 WARN org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:hdfs/nn1.example@example.com (auth:KERBEROS) cause:javax.security.sasl.SaslExcept ion: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)] 2018-05-20 14:01:35,545 WARN org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:hdfs/nn1.example@example.com (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)] 2018-05-20 14:01:35,561 WARN
[jira] [Commented] (HADOOP-15487) ConcurrentModificationException resulting in Kerberos authentication error.
[ https://issues.apache.org/jira/browse/HADOOP-15487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16484469#comment-16484469 ] Wei-Chiu Chuang commented on HADOOP-15487: -- [~daryn] bq. Not necessarily, at least if you mean core Hadoop. Are plugins involved? An external permission enforcer? An inode attributes provider? Custom audit logger? That's a good point. Looking at the configuration, it doesn't look like it's got these plugins. I've suspected HADOOP-13433 manipulated the private credentials, but didn't spot the warnings "The first kerberos ticket is not TGT(the server" or any other warnings in fixKerberosTicketOrder() method. > ConcurrentModificationException resulting in Kerberos authentication error. > --- > > Key: HADOOP-15487 > URL: https://issues.apache.org/jira/browse/HADOOP-15487 > Project: Hadoop Common > Issue Type: Bug > Environment: CDH 5.13.3. Kerberized, Hadoop-HA, jdk1.8.0_152 >Reporter: Wei-Chiu Chuang >Priority: Major > > We found the following exception message in a NameNode log. It seems the > ConcurrentModificationException caused Kerberos authentication error. > It appears to be a JDK bug, similar to HADOOP-13433 (Race in > UGI.reloginFromKeytab) but the version of Hadoop (CDH5.13.3) already patched > HADOOP-13433. (The stacktrace also differs) This cluster runs on JDK > 1.8.0_152. > {noformat} > 2018-05-19 04:00:00,182 WARN org.apache.hadoop.security.UserGroupInformation: > PriviledgedActionException as:hdfs/no...@example.com (auth:KERBEROS) > cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by > GSSException: No valid credentials provided (Mechanism level: Failed to find > any Kerberos tgt)] > 2018-05-19 04:00:00,183 INFO org.apache.hadoop.ipc.Server: Socket Reader #1 > for port 8020: readAndProcess from client 10.16.20.122 threw exception > [java.util.ConcurrentModificationException] > java.util.ConcurrentModificationException > at > java.util.LinkedList$ListItr.checkForComodification(LinkedList.java:966) > at java.util.LinkedList$ListItr.next(LinkedList.java:888) > at javax.security.auth.Subject$SecureSet$1.next(Subject.java:1070) > at javax.security.auth.Subject$ClassSet$1.run(Subject.java:1401) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject$ClassSet.populateSet(Subject.java:1399) > at javax.security.auth.Subject$ClassSet.(Subject.java:1372) > at javax.security.auth.Subject.getPrivateCredentials(Subject.java:767) > at > sun.security.jgss.krb5.SubjectComber.findAux(SubjectComber.java:127) > at > sun.security.jgss.krb5.SubjectComber.findMany(SubjectComber.java:69) > at > sun.security.jgss.krb5.ServiceCreds.getInstance(ServiceCreds.java:96) > at sun.security.jgss.krb5.Krb5Util.getServiceCreds(Krb5Util.java:203) > at > sun.security.jgss.krb5.Krb5AcceptCredential$1.run(Krb5AcceptCredential.java:74) > at > sun.security.jgss.krb5.Krb5AcceptCredential$1.run(Krb5AcceptCredential.java:72) > at java.security.AccessController.doPrivileged(Native Method) > at > sun.security.jgss.krb5.Krb5AcceptCredential.getInstance(Krb5AcceptCredential.java:71) > at > sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:127) > at > sun.security.jgss.GSSManagerImpl.getCredentialElement(GSSManagerImpl.java:193) > at sun.security.jgss.GSSCredentialImpl.add(GSSCredentialImpl.java:427) > at > sun.security.jgss.GSSCredentialImpl.(GSSCredentialImpl.java:62) > at > sun.security.jgss.GSSManagerImpl.createCredential(GSSManagerImpl.java:154) > at > com.sun.security.sasl.gsskerb.GssKrb5Server.(GssKrb5Server.java:108) > at > com.sun.security.sasl.gsskerb.FactoryImpl.createSaslServer(FactoryImpl.java:85) > at > org.apache.hadoop.security.SaslRpcServer$FastSaslServerFactory.createSaslServer(SaslRpcServer.java:398) > at > org.apache.hadoop.security.SaslRpcServer$1.run(SaslRpcServer.java:164) > at > org.apache.hadoop.security.SaslRpcServer$1.run(SaslRpcServer.java:161) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920) > at > org.apache.hadoop.security.SaslRpcServer.create(SaslRpcServer.java:160) > at > org.apache.hadoop.ipc.Server$Connection.createSaslServer(Server.java:1742) > at > org.apache.hadoop.ipc.Server$Connection.processSaslMessage(Server.java:1522) > at > org.apache.hadoop.ipc.Server$Connection.saslProcess(Server.java:1433) >
[jira] [Commented] (HADOOP-15487) ConcurrentModificationException resulting in Kerberos authentication error.
[ https://issues.apache.org/jira/browse/HADOOP-15487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16483980#comment-16483980 ] Daryn Sharp commented on HADOOP-15487: -- {quote}This exception is inside NameNode. So clearly must be in Hadoop code. {quote} Not necessarily, at least if you mean core Hadoop. Are plugins involved? An external permission enforcer? An inode attributes provider? Custom audit logger? > ConcurrentModificationException resulting in Kerberos authentication error. > --- > > Key: HADOOP-15487 > URL: https://issues.apache.org/jira/browse/HADOOP-15487 > Project: Hadoop Common > Issue Type: Bug > Environment: CDH 5.13.3. Kerberized, Hadoop-HA, jdk1.8.0_152 >Reporter: Wei-Chiu Chuang >Priority: Major > > We found the following exception message in a NameNode log. It seems the > ConcurrentModificationException caused Kerberos authentication error. > It appears to be a JDK bug, similar to HADOOP-13433 (Race in > UGI.reloginFromKeytab) but the version of Hadoop (CDH5.13.3) already patched > HADOOP-13433. (The stacktrace also differs) This cluster runs on JDK > 1.8.0_152. > {noformat} > 2018-05-19 04:00:00,182 WARN org.apache.hadoop.security.UserGroupInformation: > PriviledgedActionException as:hdfs/no...@example.com (auth:KERBEROS) > cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by > GSSException: No valid credentials provided (Mechanism level: Failed to find > any Kerberos tgt)] > 2018-05-19 04:00:00,183 INFO org.apache.hadoop.ipc.Server: Socket Reader #1 > for port 8020: readAndProcess from client 10.16.20.122 threw exception > [java.util.ConcurrentModificationException] > java.util.ConcurrentModificationException > at > java.util.LinkedList$ListItr.checkForComodification(LinkedList.java:966) > at java.util.LinkedList$ListItr.next(LinkedList.java:888) > at javax.security.auth.Subject$SecureSet$1.next(Subject.java:1070) > at javax.security.auth.Subject$ClassSet$1.run(Subject.java:1401) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject$ClassSet.populateSet(Subject.java:1399) > at javax.security.auth.Subject$ClassSet.(Subject.java:1372) > at javax.security.auth.Subject.getPrivateCredentials(Subject.java:767) > at > sun.security.jgss.krb5.SubjectComber.findAux(SubjectComber.java:127) > at > sun.security.jgss.krb5.SubjectComber.findMany(SubjectComber.java:69) > at > sun.security.jgss.krb5.ServiceCreds.getInstance(ServiceCreds.java:96) > at sun.security.jgss.krb5.Krb5Util.getServiceCreds(Krb5Util.java:203) > at > sun.security.jgss.krb5.Krb5AcceptCredential$1.run(Krb5AcceptCredential.java:74) > at > sun.security.jgss.krb5.Krb5AcceptCredential$1.run(Krb5AcceptCredential.java:72) > at java.security.AccessController.doPrivileged(Native Method) > at > sun.security.jgss.krb5.Krb5AcceptCredential.getInstance(Krb5AcceptCredential.java:71) > at > sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:127) > at > sun.security.jgss.GSSManagerImpl.getCredentialElement(GSSManagerImpl.java:193) > at sun.security.jgss.GSSCredentialImpl.add(GSSCredentialImpl.java:427) > at > sun.security.jgss.GSSCredentialImpl.(GSSCredentialImpl.java:62) > at > sun.security.jgss.GSSManagerImpl.createCredential(GSSManagerImpl.java:154) > at > com.sun.security.sasl.gsskerb.GssKrb5Server.(GssKrb5Server.java:108) > at > com.sun.security.sasl.gsskerb.FactoryImpl.createSaslServer(FactoryImpl.java:85) > at > org.apache.hadoop.security.SaslRpcServer$FastSaslServerFactory.createSaslServer(SaslRpcServer.java:398) > at > org.apache.hadoop.security.SaslRpcServer$1.run(SaslRpcServer.java:164) > at > org.apache.hadoop.security.SaslRpcServer$1.run(SaslRpcServer.java:161) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920) > at > org.apache.hadoop.security.SaslRpcServer.create(SaslRpcServer.java:160) > at > org.apache.hadoop.ipc.Server$Connection.createSaslServer(Server.java:1742) > at > org.apache.hadoop.ipc.Server$Connection.processSaslMessage(Server.java:1522) > at > org.apache.hadoop.ipc.Server$Connection.saslProcess(Server.java:1433) > at > org.apache.hadoop.ipc.Server$Connection.saslReadAndProcess(Server.java:1396) > at > org.apache.hadoop.ipc.Server$Connection.processRpcOutOfBandRequest(Server.java:2080) > at >
[jira] [Commented] (HADOOP-15487) ConcurrentModificationException resulting in Kerberos authentication error.
[ https://issues.apache.org/jira/browse/HADOOP-15487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16483427#comment-16483427 ] Xiao Chen commented on HADOOP-15487: Thanks [~jojochuang] for creating the jira and tagging me. This appears to be the exact symptom I saw in HADOOP-15401. While this confirms that this is an issue in another version, unfortunately it's the symptom of someone else removing via the iterator without synchronizing, and we don't know _where_ this faulty removal happens. In other words, this is the same symptom of HADOOP-15401 where we're being the victim, but the (likely same) culprit remains unclear... > ConcurrentModificationException resulting in Kerberos authentication error. > --- > > Key: HADOOP-15487 > URL: https://issues.apache.org/jira/browse/HADOOP-15487 > Project: Hadoop Common > Issue Type: Bug > Environment: CDH 5.13.3. Kerberized, Hadoop-HA, jdk1.8.0_152 >Reporter: Wei-Chiu Chuang >Priority: Major > > We found the following exception message in a NameNode log. It seems the > ConcurrentModificationException caused Kerberos authentication error. > It appears to be a JDK bug, similar to HADOOP-13433 (Race in > UGI.reloginFromKeytab) but the version of Hadoop (CDH5.13.3) already patched > HADOOP-13433. (The stacktrace also differs) This cluster runs on JDK > 1.8.0_152. > {noformat} > 2018-05-19 04:00:00,182 WARN org.apache.hadoop.security.UserGroupInformation: > PriviledgedActionException as:hdfs/no...@example.com (auth:KERBEROS) > cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by > GSSException: No valid credentials provided (Mechanism level: Failed to find > any Kerberos tgt)] > 2018-05-19 04:00:00,183 INFO org.apache.hadoop.ipc.Server: Socket Reader #1 > for port 8020: readAndProcess from client 10.16.20.122 threw exception > [java.util.ConcurrentModificationException] > java.util.ConcurrentModificationException > at > java.util.LinkedList$ListItr.checkForComodification(LinkedList.java:966) > at java.util.LinkedList$ListItr.next(LinkedList.java:888) > at javax.security.auth.Subject$SecureSet$1.next(Subject.java:1070) > at javax.security.auth.Subject$ClassSet$1.run(Subject.java:1401) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject$ClassSet.populateSet(Subject.java:1399) > at javax.security.auth.Subject$ClassSet.(Subject.java:1372) > at javax.security.auth.Subject.getPrivateCredentials(Subject.java:767) > at > sun.security.jgss.krb5.SubjectComber.findAux(SubjectComber.java:127) > at > sun.security.jgss.krb5.SubjectComber.findMany(SubjectComber.java:69) > at > sun.security.jgss.krb5.ServiceCreds.getInstance(ServiceCreds.java:96) > at sun.security.jgss.krb5.Krb5Util.getServiceCreds(Krb5Util.java:203) > at > sun.security.jgss.krb5.Krb5AcceptCredential$1.run(Krb5AcceptCredential.java:74) > at > sun.security.jgss.krb5.Krb5AcceptCredential$1.run(Krb5AcceptCredential.java:72) > at java.security.AccessController.doPrivileged(Native Method) > at > sun.security.jgss.krb5.Krb5AcceptCredential.getInstance(Krb5AcceptCredential.java:71) > at > sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:127) > at > sun.security.jgss.GSSManagerImpl.getCredentialElement(GSSManagerImpl.java:193) > at sun.security.jgss.GSSCredentialImpl.add(GSSCredentialImpl.java:427) > at > sun.security.jgss.GSSCredentialImpl.(GSSCredentialImpl.java:62) > at > sun.security.jgss.GSSManagerImpl.createCredential(GSSManagerImpl.java:154) > at > com.sun.security.sasl.gsskerb.GssKrb5Server.(GssKrb5Server.java:108) > at > com.sun.security.sasl.gsskerb.FactoryImpl.createSaslServer(FactoryImpl.java:85) > at > org.apache.hadoop.security.SaslRpcServer$FastSaslServerFactory.createSaslServer(SaslRpcServer.java:398) > at > org.apache.hadoop.security.SaslRpcServer$1.run(SaslRpcServer.java:164) > at > org.apache.hadoop.security.SaslRpcServer$1.run(SaslRpcServer.java:161) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920) > at > org.apache.hadoop.security.SaslRpcServer.create(SaslRpcServer.java:160) > at > org.apache.hadoop.ipc.Server$Connection.createSaslServer(Server.java:1742) > at > org.apache.hadoop.ipc.Server$Connection.processSaslMessage(Server.java:1522) > at > org.apache.hadoop.ipc.Server$Connection.saslProcess(Server.java:1433) > at >
[jira] [Commented] (HADOOP-15487) ConcurrentModificationException resulting in Kerberos authentication error.
[ https://issues.apache.org/jira/browse/HADOOP-15487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16483240#comment-16483240 ] Wei-Chiu Chuang commented on HADOOP-15487: -- In fact, it seems HADOOP-13433 is actually the only Hadoop code that modifies Subject's PrivateCredentials... > ConcurrentModificationException resulting in Kerberos authentication error. > --- > > Key: HADOOP-15487 > URL: https://issues.apache.org/jira/browse/HADOOP-15487 > Project: Hadoop Common > Issue Type: Bug > Environment: CDH 5.13.3. Kerberized, Hadoop-HA, jdk1.8.0_152 >Reporter: Wei-Chiu Chuang >Priority: Major > > We found the following exception message in a NameNode log. It seems the > ConcurrentModificationException caused Kerberos authentication error. > It appears to be a JDK bug, similar to HADOOP-13433 (Race in > UGI.reloginFromKeytab) but the version of Hadoop (CDH5.13.3) already patched > HADOOP-13433. (The stacktrace also differs) This cluster runs on JDK > 1.8.0_152. > {noformat} > 2018-05-19 04:00:00,182 WARN org.apache.hadoop.security.UserGroupInformation: > PriviledgedActionException as:hdfs/no...@example.com (auth:KERBEROS) > cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by > GSSException: No valid credentials provided (Mechanism level: Failed to find > any Kerberos tgt)] > 2018-05-19 04:00:00,183 INFO org.apache.hadoop.ipc.Server: Socket Reader #1 > for port 8020: readAndProcess from client 10.16.20.122 threw exception > [java.util.ConcurrentModificationException] > java.util.ConcurrentModificationException > at > java.util.LinkedList$ListItr.checkForComodification(LinkedList.java:966) > at java.util.LinkedList$ListItr.next(LinkedList.java:888) > at javax.security.auth.Subject$SecureSet$1.next(Subject.java:1070) > at javax.security.auth.Subject$ClassSet$1.run(Subject.java:1401) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject$ClassSet.populateSet(Subject.java:1399) > at javax.security.auth.Subject$ClassSet.(Subject.java:1372) > at javax.security.auth.Subject.getPrivateCredentials(Subject.java:767) > at > sun.security.jgss.krb5.SubjectComber.findAux(SubjectComber.java:127) > at > sun.security.jgss.krb5.SubjectComber.findMany(SubjectComber.java:69) > at > sun.security.jgss.krb5.ServiceCreds.getInstance(ServiceCreds.java:96) > at sun.security.jgss.krb5.Krb5Util.getServiceCreds(Krb5Util.java:203) > at > sun.security.jgss.krb5.Krb5AcceptCredential$1.run(Krb5AcceptCredential.java:74) > at > sun.security.jgss.krb5.Krb5AcceptCredential$1.run(Krb5AcceptCredential.java:72) > at java.security.AccessController.doPrivileged(Native Method) > at > sun.security.jgss.krb5.Krb5AcceptCredential.getInstance(Krb5AcceptCredential.java:71) > at > sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:127) > at > sun.security.jgss.GSSManagerImpl.getCredentialElement(GSSManagerImpl.java:193) > at sun.security.jgss.GSSCredentialImpl.add(GSSCredentialImpl.java:427) > at > sun.security.jgss.GSSCredentialImpl.(GSSCredentialImpl.java:62) > at > sun.security.jgss.GSSManagerImpl.createCredential(GSSManagerImpl.java:154) > at > com.sun.security.sasl.gsskerb.GssKrb5Server.(GssKrb5Server.java:108) > at > com.sun.security.sasl.gsskerb.FactoryImpl.createSaslServer(FactoryImpl.java:85) > at > org.apache.hadoop.security.SaslRpcServer$FastSaslServerFactory.createSaslServer(SaslRpcServer.java:398) > at > org.apache.hadoop.security.SaslRpcServer$1.run(SaslRpcServer.java:164) > at > org.apache.hadoop.security.SaslRpcServer$1.run(SaslRpcServer.java:161) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920) > at > org.apache.hadoop.security.SaslRpcServer.create(SaslRpcServer.java:160) > at > org.apache.hadoop.ipc.Server$Connection.createSaslServer(Server.java:1742) > at > org.apache.hadoop.ipc.Server$Connection.processSaslMessage(Server.java:1522) > at > org.apache.hadoop.ipc.Server$Connection.saslProcess(Server.java:1433) > at > org.apache.hadoop.ipc.Server$Connection.saslReadAndProcess(Server.java:1396) > at > org.apache.hadoop.ipc.Server$Connection.processRpcOutOfBandRequest(Server.java:2080) > at > org.apache.hadoop.ipc.Server$Connection.processOneRpc(Server.java:1920) > at > org.apache.hadoop.ipc.Server$Connection.readAndProcess(Server.java:1682)
[jira] [Commented] (HADOOP-15487) ConcurrentModificationException resulting in Kerberos authentication error.
[ https://issues.apache.org/jira/browse/HADOOP-15487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16483232#comment-16483232 ] Wei-Chiu Chuang commented on HADOOP-15487: -- BTW I think this is the same as HADOOP-15401. [~xiaochen] [~daryn] bq. Searched impala code but don't find any faulty usage. Now here's an example. > ConcurrentModificationException resulting in Kerberos authentication error. > --- > > Key: HADOOP-15487 > URL: https://issues.apache.org/jira/browse/HADOOP-15487 > Project: Hadoop Common > Issue Type: Bug > Environment: CDH 5.13.3. Kerberized, Hadoop-HA, jdk1.8.0_152 >Reporter: Wei-Chiu Chuang >Priority: Major > > We found the following exception message in a NameNode log. It seems the > ConcurrentModificationException caused Kerberos authentication error. > It appears to be a JDK bug, similar to HADOOP-13433 (Race in > UGI.reloginFromKeytab) but the version of Hadoop (CDH5.13.3) already patched > HADOOP-13433. (The stacktrace also differs) This cluster runs on JDK > 1.8.0_152. > {noformat} > 2018-05-19 04:00:00,182 WARN org.apache.hadoop.security.UserGroupInformation: > PriviledgedActionException as:hdfs/no...@example.com (auth:KERBEROS) > cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by > GSSException: No valid credentials provided (Mechanism level: Failed to find > any Kerberos tgt)] > 2018-05-19 04:00:00,183 INFO org.apache.hadoop.ipc.Server: Socket Reader #1 > for port 8020: readAndProcess from client 10.16.20.122 threw exception > [java.util.ConcurrentModificationException] > java.util.ConcurrentModificationException > at > java.util.LinkedList$ListItr.checkForComodification(LinkedList.java:966) > at java.util.LinkedList$ListItr.next(LinkedList.java:888) > at javax.security.auth.Subject$SecureSet$1.next(Subject.java:1070) > at javax.security.auth.Subject$ClassSet$1.run(Subject.java:1401) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject$ClassSet.populateSet(Subject.java:1399) > at javax.security.auth.Subject$ClassSet.(Subject.java:1372) > at javax.security.auth.Subject.getPrivateCredentials(Subject.java:767) > at > sun.security.jgss.krb5.SubjectComber.findAux(SubjectComber.java:127) > at > sun.security.jgss.krb5.SubjectComber.findMany(SubjectComber.java:69) > at > sun.security.jgss.krb5.ServiceCreds.getInstance(ServiceCreds.java:96) > at sun.security.jgss.krb5.Krb5Util.getServiceCreds(Krb5Util.java:203) > at > sun.security.jgss.krb5.Krb5AcceptCredential$1.run(Krb5AcceptCredential.java:74) > at > sun.security.jgss.krb5.Krb5AcceptCredential$1.run(Krb5AcceptCredential.java:72) > at java.security.AccessController.doPrivileged(Native Method) > at > sun.security.jgss.krb5.Krb5AcceptCredential.getInstance(Krb5AcceptCredential.java:71) > at > sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:127) > at > sun.security.jgss.GSSManagerImpl.getCredentialElement(GSSManagerImpl.java:193) > at sun.security.jgss.GSSCredentialImpl.add(GSSCredentialImpl.java:427) > at > sun.security.jgss.GSSCredentialImpl.(GSSCredentialImpl.java:62) > at > sun.security.jgss.GSSManagerImpl.createCredential(GSSManagerImpl.java:154) > at > com.sun.security.sasl.gsskerb.GssKrb5Server.(GssKrb5Server.java:108) > at > com.sun.security.sasl.gsskerb.FactoryImpl.createSaslServer(FactoryImpl.java:85) > at > org.apache.hadoop.security.SaslRpcServer$FastSaslServerFactory.createSaslServer(SaslRpcServer.java:398) > at > org.apache.hadoop.security.SaslRpcServer$1.run(SaslRpcServer.java:164) > at > org.apache.hadoop.security.SaslRpcServer$1.run(SaslRpcServer.java:161) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920) > at > org.apache.hadoop.security.SaslRpcServer.create(SaslRpcServer.java:160) > at > org.apache.hadoop.ipc.Server$Connection.createSaslServer(Server.java:1742) > at > org.apache.hadoop.ipc.Server$Connection.processSaslMessage(Server.java:1522) > at > org.apache.hadoop.ipc.Server$Connection.saslProcess(Server.java:1433) > at > org.apache.hadoop.ipc.Server$Connection.saslReadAndProcess(Server.java:1396) > at > org.apache.hadoop.ipc.Server$Connection.processRpcOutOfBandRequest(Server.java:2080) > at > org.apache.hadoop.ipc.Server$Connection.processOneRpc(Server.java:1920) > at >