Hi,
I have a secure cluster with kerberized jobtracker.
1) I am able to launch a job thru command line to oozie which gets the
correct kinit'd kerberos credentials from cache to the setup job.
2)The setup job completes fine after authenticating to job tracker.
3) It spawns off a child map-reduce task which does not inherit the
kerberos credentials from its parent task.
4) this fails in the sasl RPC communication with the jobtracker.
Is there any workaround/setup I am missing?
-------------------------
------------------------------------------------------------------------------------------------------------------------------------
Status : KILLED
ID
Status Ext ID Ext Status Err Code
------------------------------------------------------------------------------------------------------------------------------------
0000000-140723145019902-oozie-mapr-W@:start:
OK - OK -
------------------------------------------------------------------------------------------------------------------------------------
0000000-140723145019902-oozie-mapr-W@mr-node
ERROR job_201407231357_0003 FAILED/KILLEDJA018
------------------------------------------------------------------------------------------------------------------------------------
0000000-140723145019902-oozie-mapr-W@fail
OK - OK E0729
---
--------------------------
Failed to establish initial contact with all servers. mapred.job.tracker =
maprfs:///
2014-07-23 14:20:20,949 WARN MapReduceActionExecutor:542 - USER[mapr]
GROUP[-] TOKEN[] APP[map-reduce-wf]
JOB[0000000-140723095640612-oozie-mapr-W] ACTION[0000000-140
723095640612-oozie-mapr-W@mr-node] Launcher exception: Failed to establish
initial contact with all servers. mapred.job.tracker = maprfs:///
java.io.IOException: Failed to establish initial contact with all servers.
mapred.job.tracker = maprfs:///
at
org.apache.hadoop.ipc.RPC$FailoverInvoker.searchActiverServer(RPC.java:467)
at org.apache.hadoop.ipc.RPC$FailoverInvoker.invoke(RPC.java:556)
at org.apache.hadoop.mapred.$Proxy5.getStagingAreaDir(Unknown
Source)
at
org.apache.hadoop.mapred.JobClient.getStagingAreaDir(JobClient.java:1337)
at
org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(JobSubmissionFiles.java:102)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:902)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:896)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1122)
at
org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:896)
at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:870)
at
org.apache.oozie.action.hadoop.MapReduceMain.submitJob(MapReduceMain.java:99)
at
org.apache.oozie.action.hadoop.MapReduceMain.run(MapReduceMain.java:59)
at
org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:37)
at
org.apache.oozie.action.hadoop.MapReduceMain.main(MapReduceMain.java:42)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:232)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:443)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:353)
at org.apache.hadoop.mapred.Child$4.run(Child.java:282)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1122)
at org.apache.hadoop.mapred.Child.main(Child.java:271)
--------------------------
2014-07-23 19:55:56,646 DEBUG ipc.Client [main]: Exception encountered
during SASL client setup.
javax.security.sasl.SaslException: GSS initiate failed [Caused by
GSSException: No valid credentials provided (Mechanism level: Failed
to find any Kerberos tgt)]
at
com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:194)
at
org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:103)
at
org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:442)
at org.apache.hadoop.ipc.Client$Connection.access$1200(Client.java:229)
at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:572)
at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:569)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1122)
at
org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:568)
at org.apache.hadoop.ipc.Client$Connection.access$2400(Client.java:229)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1295)
at org.apache.hadoop.ipc.Client.call(Client.java:1120)
at org.apache.hadoop.ipc.Client.call(Client.java:1095)
at org.apache.hadoop.ipc.RPC$FailoverInvoker.invoke(RPC.java:540)
at org.apache.hadoop.mapred.$Proxy5.getStagingAreaDir(Unknown Source)
at
org.apache.hadoop.mapred.JobClient.getStagingAreaDir(JobClient.java:1337)
at
org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(JobSubmissionFiles.java:102)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:902)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:896)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1122)
at
org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:896)
at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:870)
at
org.apache.oozie.action.hadoop.MapReduceMain.submitJob(MapReduceMain.java:99)
at
org.apache.oozie.action.hadoop.MapReduceMain.run(MapReduceMain.java:59)
at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:37)
at
org.apache.oozie.action.hadoop.MapReduceMain.main(MapReduceMain.java:42)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:232)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:443)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:353)
at org.apache.hadoop.mapred.Child$4.run(Child.java:282)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1122)
at org.apache.hadoop.mapred.Child.main(Child.java:271)
Caused by: GSSException: No valid credentials provided (Mechanism
level: Failed to find any Kerberos tgt)
at
sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:130)
at
sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:106)
at
sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:172)
at
sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:209)
at
sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:195)
at
sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:162)
at
com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:175)
... 41 more
-------------------------------------
Attaching my oozie-site.xml -
-------------------------------------
Thanks,
Suhas.
