[
https://issues.apache.org/jira/browse/MAPREDUCE-3669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13185871#comment-13185871
]
Mahadev konar commented on MAPREDUCE-3669:
------------------------------------------
Ok,
I think I figured out what the issue is, given that I cannot reproduce this.
This is the service classloading that we do that is causing the issue. For
MRClientProtocol, we have two Security info's, HSSecurityInfo and
MRClientSecurityInfo. Depending on which class is loaded first, something will
break, either talking to the HS or AM. This was working until now because
HSSecurityInfo worked only for kerberos and MRClientSecurityInfo only for
tokens. After I added tokens to HSSecurityInfo, this is an issue.
> Getting a lot of PriviledgedActionException / SaslException when running a job
> ------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3669
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3669
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Thomas Graves
> Assignee: Mahadev konar
> Priority: Blocker
>
> On a secure cluster, when running a job we are seeing a lot of
> PriviledgedActionException / SaslExceptions. The job runs fine, its just the
> jobclient can't connect to the AM to get the progress information.
> Its in a very tight loop retrying while getting the exceptions.
> snip of the client log is:
> 12/01/13 15:33:45 INFO security.SecurityUtil: Acquired token Ident: 00 1c 68
> 61 64 6f 6f 70 71 61 40 44 45 56 2e 59 47
> 52 49 44 2e 59 41 48 4f 4f 2e 43 4f 4d 08 6d 61 70 72 65 64 71 61 00 8a 01 34
> d7 b3 ff f5 8a 01 34 fb c0 83 f5 08 02,
> Kind: HDFS_DELEGATION_TOKEN, Service: 10.10.10.10:8020
> 12/01/13 15:33:45 INFO hdfs.DFSClient: Created HDFS_DELEGATION_TOKEN token 8
> for user1 on 10.10.10.10:8020
> 12/01/13 15:33:45 INFO security.TokenCache: Got dt for
> hdfs://host1.domain.com:8020;uri=10.10.10.10:8020;t.service=10.10.10.10:8020
> 12/01/13 15:33:45 WARN conf.Configuration: mapred.used.genericoptionsparser
> is deprecated. Instead, use
> mapreduce.client.genericoptionsparser.used
> 12/01/13 15:33:45 INFO mapreduce.JobSubmitter: number of splits:2
> 12/01/13 15:33:45 INFO mapred.ResourceMgrDelegate: Submitted application
> application_1326410042859_0008 to
> ResourceManager at rmhost.domain/10.10.10.11:8040
> 12/01/13 15:33:45 INFO mapreduce.Job: Running job: job_1326410042859_0008
> 12/01/13 15:33:52 INFO mapred.ClientServiceDelegate: The url to track the job:
> rmhost.domain:8088/proxy/application_1326410042859_0008/
> 12/01/13 15:33:52 ERROR security.UserGroupInformation:
> PriviledgedActionException as:[email protected]
> (auth:SIMPLE) cause:javax.security.sasl.SaslException: GSS initiate failed
> [Caused by GSSException: No valid credentials provided (Mechanism level: Fail
> ed to find any
> Kerberos tgt)]
> 12/01/13 15:33:52 WARN ipc.Client: Exception encountered while connecting to
> the server :
> javax.security.sasl.SaslException: GSS initiate failed [Caused by
> GSSException: No valid credentials provided
> (Mechanism level: Failed to find any Kerberos tgt)]
> 12/01/13 15:33:52 ERROR security.UserGroupInformation:
> PriviledgedActionException as:[email protected]
> (auth:SIMPLE) cause:java.io.IOException: javax.security.sasl.SaslException:
> GSS initiate failed [Caused by GSSException: No valid credentials provided (
> Mechanism level:
> Failed to find any Kerberos tgt)]
> 12/01/13 15:33:52 INFO mapred.ClientServiceDelegate: The url to track the job:
> rmhost.domain:8088/proxy/application_1326410042859_0008/
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
