[ https://issues.apache.org/jira/browse/YARN-554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13693594#comment-13693594 ]
Omkar Vinit Joshi commented on YARN-554: ---------------------------------------- is this valid on trunk? > YARN with security enabled read local block failed > -------------------------------------------------- > > Key: YARN-554 > URL: https://issues.apache.org/jira/browse/YARN-554 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager > Affects Versions: 2.0.2-alpha, 2.0.1-alpha, 2.0.0-alpha, 2.0.3-alpha > Reporter: Zesheng Wu > Labels: hdfs, yarn > > When security is enabled, and the dfs.client.read.shortcircuit is set, the > YARN app will fail when read local blocks, here is the log: > 2013-04-08 14:00:09,103 WARN [main] org.apache.hadoop.hdfs.DFSClient: Failed > to connect to /10.2.201.25:41402 for block, add to deadNodes and continue. > java.io.IOException: Failed on local exception: java.io.IOException: > javax.security.sasl.SaslException: GSS initiate failed [Caused by > GSSException: No valid credentials provided (Mechanism level: Failed to find > any Kerberos tgt)]; Host Details : local host is: > "lg-hadoop-st09.bj/10.2.201.25"; destination host is: > "lg-hadoop-st09.bj":41400; > java.io.IOException: Failed on local exception: java.io.IOException: > javax.security.sasl.SaslException: GSS initiate failed [Caused by > GSSException: No valid credentials provided (Mechanism level: Failed to find > any Kerberos tgt)]; Host Details : local host is: > "lg-hadoop-st09.bj/10.2.201.25"; destination host is: > "lg-hadoop-st09.bj":41400; > at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:759) > at org.apache.hadoop.ipc.Client.call(Client.java:1164) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:202) > at $Proxy14.getBlockLocalPathInfo(Unknown Source) > at > org.apache.hadoop.hdfs.protocolPB.ClientDatanodeProtocolTranslatorPB.getBlockLocalPathInfo(ClientDatanodeProtocolTranslatorPB.java:199) > at > org.apache.hadoop.hdfs.BlockReaderLocal.getBlockPathInfo(BlockReaderLocal.java:254) > at > org.apache.hadoop.hdfs.BlockReaderLocal.newBlockReader(BlockReaderLocal.java:167) > at > org.apache.hadoop.hdfs.DFSClient.getLocalBlockReader(DFSClient.java:767) > at > org.apache.hadoop.hdfs.DFSInputStream.getBlockReader(DFSInputStream.java:888) > at > org.apache.hadoop.hdfs.DFSInputStream.blockSeekTo(DFSInputStream.java:455) > at > org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:645) > at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:689) > at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:518) > at java.io.DataInputStream.readByte(DataInputStream.java:248) > at org.apache.hadoop.io.WritableUtils.readVLong(WritableUtils.java:308) > at org.apache.hadoop.io.WritableUtils.readVInt(WritableUtils.java:329) > at org.apache.hadoop.io.Text.readString(Text.java:446) > at org.apache.hadoop.mapred.MapTask.getSplitDetails(MapTask.java:345) > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:694) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332) > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:152) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:396) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332) > at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:147) > Caused by: java.io.IOException: javax.security.sasl.SaslException: GSS > initiate failed [Caused by GSSException: No valid credentials provided > (Mechanism level: Failed to find any Kerberos tgt)] > at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:548) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:396) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332) > at > org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:512) > at > org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:596) > at org.apache.hadoop.ipc.Client$Connection.access$1700(Client.java:220) > at org.apache.hadoop.ipc.Client.getConnection(Client.java:1213) > at org.apache.hadoop.ipc.Client.call(Client.java:1140) > ... 23 more > Caused by: javax.security.sasl.SaslException: GSS initiate failed [Caused by > GSSException: No valid credentials provided (Mechanism level: Failed to find > any Kerberos tgt)] > at > com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:194) > at > org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:137) > at > org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:423) > at org.apache.hadoop.ipc.Client$Connection.access$1300(Client.java:220) > at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:589) > at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:586) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:396) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332) > at > org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:585) > ... 26 more > Caused by: GSSException: No valid credentials provided (Mechanism level: > Failed to find any Kerberos tgt) > at > sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:130) > at > sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:106) > at > sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:172) > at > sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:209) > at > sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:195) > at > sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:162) > at > com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:175) > ... 35 more > 2013-04-08 14:00:09,278 INFO [main] org.apache.hadoop.hdfs.DFSClient: > Successfully connected to /10.2.201.24:41402 for block -6472166635466414770 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira