Can you paste your workflow definition and job.properties here?
From: Suhas Satish <[email protected]<mailto:[email protected]>>
Reply-To: "[email protected]<mailto:[email protected]>"
<[email protected]<mailto:[email protected]>>
Date: Thursday, July 24, 2014 at 10:45 AM
To: "[email protected]<mailto:[email protected]>"
<[email protected]<mailto:[email protected]>>
Subject: launching oozie jobs on kerberos-secured cluster
Hi,
I have a secure cluster with kerberized jobtracker.
1) I am able to launch a job thru command line to oozie which gets the correct
kinit'd kerberos credentials from cache to the setup job.
2)The setup job completes fine after authenticating to job tracker.
3) It spawns off a child map-reduce task which does not inherit the kerberos
credentials from its parent task.
4) this fails in the sasl RPC communication with the jobtracker.
Is there any workaround/setup I am missing?
-------------------------
------------------------------------------------------------------------------------------------------------------------------------
Status : KILLED
ID
Status Ext ID Ext Status Err Code
------------------------------------------------------------------------------------------------------------------------------------
0000000-140723145019902-oozie-mapr-W@:start:
OK - OK -
------------------------------------------------------------------------------------------------------------------------------------
0000000-140723145019902-oozie-mapr-W@mr-node
ERROR job_201407231357_0003 FAILED/KILLEDJA018
------------------------------------------------------------------------------------------------------------------------------------
0000000-140723145019902-oozie-mapr-W@fail
OK - OK E0729
---
--------------------------
Failed to establish initial contact with all servers. mapred.job.tracker =
maprfs:///
2014-07-23 14:20:20,949 WARN MapReduceActionExecutor:542 - USER[mapr] GROUP[-]
TOKEN[] APP[map-reduce-wf] JOB[0000000-140723095640612-oozie-mapr-W]
ACTION[0000000-140 723095640612-oozie-mapr-W@mr-node] Launcher exception:
Failed to establish initial contact with all servers. mapred.job.tracker =
maprfs:///
java.io.IOException: Failed to establish initial contact with all servers.
mapred.job.tracker = maprfs:///
at
org.apache.hadoop.ipc.RPC$FailoverInvoker.searchActiverServer(RPC.java:467)
at org.apache.hadoop.ipc.RPC$FailoverInvoker.invoke(RPC.java:556)
at org.apache.hadoop.mapred.$Proxy5.getStagingAreaDir(Unknown Source)
at
org.apache.hadoop.mapred.JobClient.getStagingAreaDir(JobClient.java:1337)
at
org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(JobSubmissionFiles.java:102)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:902)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:896)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1122)
at
org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:896)
at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:870)
at
org.apache.oozie.action.hadoop.MapReduceMain.submitJob(MapReduceMain.java:99)
at
org.apache.oozie.action.hadoop.MapReduceMain.run(MapReduceMain.java:59)
at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:37)
at
org.apache.oozie.action.hadoop.MapReduceMain.main(MapReduceMain.java:42)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:232)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:443)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:353)
at org.apache.hadoop.mapred.Child$4.run(Child.java:282)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1122)
at org.apache.hadoop.mapred.Child.main(Child.java:271)
--------------------------
2014-07-23 19:55:56,646 DEBUG ipc.Client [main]: Exception encountered during
SASL client setup.
javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException:
No valid credentials provided (Mechanism level: Failed to find any Kerberos
tgt)]
at
com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:194)
at
org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:103)
at
org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:442)
at org.apache.hadoop.ipc.Client$Connection.access$1200(Client.java:229)
at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:572)
at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:569)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1122)
at
org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:568)
at org.apache.hadoop.ipc.Client$Connection.access$2400(Client.java:229)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1295)
at org.apache.hadoop.ipc.Client.call(Client.java:1120)
at org.apache.hadoop.ipc.Client.call(Client.java:1095)
at org.apache.hadoop.ipc.RPC$FailoverInvoker.invoke(RPC.java:540)
at org.apache.hadoop.mapred.$Proxy5.getStagingAreaDir(Unknown Source)
at
org.apache.hadoop.mapred.JobClient.getStagingAreaDir(JobClient.java:1337)
at
org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(JobSubmissionFiles.java:102)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:902)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:896)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1122)
at
org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:896)
at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:870)
at
org.apache.oozie.action.hadoop.MapReduceMain.submitJob(MapReduceMain.java:99)
at
org.apache.oozie.action.hadoop.MapReduceMain.run(MapReduceMain.java:59)
at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:37)
at
org.apache.oozie.action.hadoop.MapReduceMain.main(MapReduceMain.java:42)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:232)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:443)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:353)
at org.apache.hadoop.mapred.Child$4.run(Child.java:282)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1122)
at org.apache.hadoop.mapred.Child.main(Child.java:271)
Caused by: GSSException: No valid credentials provided (Mechanism level: Failed
to find any Kerberos tgt)
at
sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:130)
at
sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:106)
at
sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:172)
at
sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:209)
at
sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:195)
at
sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:162)
at
com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:175)
... 41 more
-------------------------------------
Attaching my oozie-site.xml -
-------------------------------------
Thanks,
Suhas.
