[
https://issues.apache.org/jira/browse/YARN-2563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14136486#comment-14136486
]
Arpit Gupta commented on YARN-2563:
-----------------------------------
Here is the stack trace
{code}
ozie-oozi-W@wc] Launcher ERROR, reason: Main class
[org.apache.oozie.action.hadoop.MapReduceMain], main() threw exception,
org.apache.hadoop.security.authentication.client.AuthenticationException:
GSSException: No valid credentials provided (Mechanism level: Failed to find
any Kerberos tgt)
2014-09-13 00:00:18,595 WARN MapReduceActionExecutor:546 - USER[hrt_qa]
GROUP[-] TOKEN[] APP[wordcount-wf] JOB[0000001-140912235159138-oozie-oozi-W]
ACTION[0000001-140912235159138-oozie-oozi-W@wc] Launcher exception:
org.apache.hadoop.security.authentication.client.AuthenticationException:
GSSException: No valid credentials provided (Mechanism level: Failed to find
any Kerberos tgt)
java.io.IOException:
org.apache.hadoop.security.authentication.client.AuthenticationException:
GSSException: No valid credentials provided (Mechanism level: Failed to find
any Kerberos tgt)
at
org.apache.hadoop.yarn.client.api.impl.TimelineAuthenticator.getDelegationToken(TimelineAuthenticator.java:135)
at
org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl.getDelegationToken(TimelineClientImpl.java:181)
at
org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.addTimelineDelegationToken(YarnClientImpl.java:275)
at
org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.submitApplication(YarnClientImpl.java:221)
at
org.apache.hadoop.mapred.ResourceMgrDelegate.submitApplication(ResourceMgrDelegate.java:282)
at org.apache.hadoop.mapred.YARNRunner.submitJob(YARNRunner.java:289)
at
org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:437)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1294)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1291)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:1291)
at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:562)
at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:557)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
at
org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:557)
at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:548)
at
org.apache.oozie.action.hadoop.MapReduceMain.submitJob(MapReduceMain.java:108)
at
org.apache.oozie.action.hadoop.MapReduceMain.run(MapReduceMain.java:66)
at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:39)
at
org.apache.oozie.action.hadoop.MapReduceMain.main(MapReduceMain.java:37)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:226)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:450)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)
Caused by:
org.apache.hadoop.security.authentication.client.AuthenticationException:
GSSException: No valid credentials provided (Mechanism level: Failed to find
any Kerberos tgt)
at
org.apache.hadoop.security.authentication.client.KerberosAuthenticator.doSpnegoSequence(KerberosAuthenticator.java:306)
at
org.apache.hadoop.security.authentication.client.KerberosAuthenticator.authenticate(KerberosAuthenticator.java:196)
at
org.apache.hadoop.yarn.client.api.impl.TimelineAuthenticator.authenticate(TimelineAuthenticator.java:101)
at
org.apache.hadoop.security.authentication.client.AuthenticatedURL.openConnection(AuthenticatedURL.java:216)
at
org.apache.hadoop.yarn.client.api.impl.TimelineAuthenticator.getDelegationToken(TimelineAuthenticator.java:121)
... 36 more
Caused by: GSSException: No valid credentials provided (Mechanism level: Failed
to find any Kerberos tgt)
at
sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147)
at
sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:121)
at
sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187)
at
sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:223)
at
sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212)
at
sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
at
org.apache.hadoop.security.authentication.client.KerberosAuthenticator$1.run(KerberosAuthenticator.java:285)
at
org.apache.hadoop.security.authentication.client.KerberosAuthenticator$1.run(KerberosAuthenticator.java:261)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.authentication.client.KerberosAuthenticator.doSpnegoSequence(KerberosAuthenticator.java:261)
... 40 more
2014-09-13 00:00:18,617 DEBUG ActionEndXCommand:549 - USER[hrt_qa] GROUP[-]
TOKEN[] APP[wordcount-wf] JOB[0000001-140912235159138-oozie-oozi-W]
ACTION[0000001-140912235159138-oozie-oozi-W@wc] Execute command [action.end]
key [0000001-140912235159138-oozie-oozi-W]
2014-09-13 00:00:18,617 DEBUG ActionEndXCommand:549 - USER[hrt_qa] GROUP[-]
TOKEN[] APP[wordcount-wf] JOB[0000001-140912235159138-oozie-oozi-W]
ACTION[0000001-140912235159138-oozie-oozi-W@wc] STARTED ActionEndXCommand for
action 0000001-140912235159138-oozie-oozi-W@wc
2014-09-13 00:00:18,623 DEBUG ActionEndXCommand:549 - USER[hrt_qa] GROUP[-]
TOKEN[] APP[wordcount-wf] JOB[0000001-140912235159138-oozie-oozi-W]
ACTION[0000001-140912235159138-oozie-oozi-W@wc] End, name [wc] type
[map-reduce] status[DONE] external status [FAILED/KILLED] signal value [null]
2014-09-13 00:00:18,654 DEBUG HadoopAccessorService:549 - USER[hrt_qa] GROUP[-]
TOKEN[] APP[wordcount-wf] JOB[0000001-140912235159138-oozie-oozi-W]
ACTION[0000001-140912235159138-oozie-oozi-W@wc] Checking if filesystem hdfs is
supported
2014-09-13 00:00:18,685 DEBUG HadoopAccessorService:549 - USER[hrt_qa] GROUP[-]
TOKEN[] APP[wordcount-wf] JOB[0000001-140912235159138-oozie-oozi-W]
ACTION[0000001-140912235159138-oozie-oozi-W@wc] Checking if filesystem hdfs is
supported
2014-09-13 00:00:18,706 INFO ActionEndXCommand:543 - USER[hrt_qa] GROUP[-]
TOKEN[] APP[wordcount-wf] JOB[0000001-140912235159138-oozie-oozi-W]
ACTION[0000001-140912235159138-oozie-oozi-W@wc] ERROR is considered as FAILED
for SLA
2014-09-13 00:00:18,728 DEBUG SignalXCommand:549 - USER[hrt_qa] GROUP[-]
TOKEN[] APP[wordcount-wf] JOB[0000001-140912235159138-oozie-oozi-W]
ACTION[0000001-140912235159138-oozie-oozi-W@wc] Execute command [signal] key
[0000001-140912235159138-oozie-oozi-W]
2014-09-13 00:00:18,728 DEBUG SignalXCommand:549 - USER[hrt_qa] GROUP[-]
TOKEN[] APP[wordcount-wf] JOB[0000001-140912235159138-oozie-oozi-W]
ACTION[0000001-140912235159138-oozie-oozi-W@wc] STARTED SignalCommand for
jobid=0000001-140912235159138-oozie-oozi-W,
actionId=0000001-140912235159138-oozie-oozi-W@wc
2014-09-13 00:00:18,732 DEBUG LiteWorkflowInstance:549 - USER[hrt_qa] GROUP[-]
TOKEN[] APP[wordcount-wf] JOB[0000001-140912235159138-oozie-oozi-W]
ACTION[0000001-140912235159138-oozie-oozi-W@wc] Signaling job execution path
[/] signal value [ERROR]
2014-09-13 00:00:18,732 DEBUG LiteWorkflowInstance:549 - USER[hrt_qa] GROUP[-]
TOKEN[] APP[wordcount-wf] JOB[0000001-140912235159138-oozie-oozi-W]
ACTION[0000001-140912235159138-oozie-oozi-W@wc] Exiting node [wc] with
transition[/#fail]
2014-09-13 00:00:18,732 DEBUG LiteWorkflowInstance:549 - USER[hrt_qa] GROUP[-]
TOKEN[] APP[wordcount-wf] JOB[0000001-140912235159138-oozie-oozi-W]
ACTION[0000001-140912235159138-oozie-oozi-W@wc] Signaling job execution path
[/] signal value [::synch::]
2014-09-13 00:00:18,732 DEBUG LiteWorkflowStore
{code}
> On secure clusters call to timeline server fails with authentication errors
> when running a job via oozie
> --------------------------------------------------------------------------------------------------------
>
> Key: YARN-2563
> URL: https://issues.apache.org/jira/browse/YARN-2563
> Project: Hadoop YARN
> Issue Type: Bug
> Components: timelineserver
> Affects Versions: 2.6.0
> Reporter: Arpit Gupta
> Priority: Blocker
>
> During our nightlies on a secure cluster we have seen oozie jobs fail with
> authentication error to the time line server.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
