Hey Daryn and Oozie folks as well, I've reported this issue as https://issues.apache.org/jira/browse/HADOOP-9461, posting as I figured you may be interested in it (Oozie gets affected on the outcome).
On Thu, Jan 31, 2013 at 2:36 AM, Daryn Sharp <[email protected]> wrote: > The token renewer needs to be the job tracker principal. I think oozie had > "mr token" hardcoded at one point, but later changed it to use a conf > setting. > > The rest of the log looks very odd - ie. it looks like security is off, but > it can't be. It's trying to renew hdfs tokens issued for the "hdfs" > superuser. Are you running your job as hdfs? With security enabled, the > container executor won't run jobs for hdfs… It's also trying to renew as > "mapred (auth:SIMPLE)". SIMPLE means that security is disabled, which > doesn't make sense since tokens require kerberos. > > The cannot find renewer class for MAPREDUCE_DELEGATION_TOKEN is also odd > because it did subsequently try to renew the token. > > Daryn > > > > On Jan 30, 2013, at 2:14 PM, Alejandro Abdelnur wrote: > > Cobert, > > [Moving thread to [email protected], BCCing [email protected]] > > * What version of Oozie are you using? > * Is the cluster a secure setup (Kerberos enabled)? > * Would you mind posting the complete launcher logs? > > Thx > > > > On Wed, Jan 30, 2013 at 6:14 AM, Corbett Martin <[email protected]> wrote: >> >> Thanks for the tip. >> >> The sqoop command listed in the stdout log file is: >> sqoop >> import >> --driver >> org.apache.derby.jdbc.ClientDriver >> --connect >> jdbc:derby://test-server:1527/mondb >> --username >> monuser >> --password >> xxxxxxxxx >> --table >> MONITOR >> --split-by >> request_id >> --target-dir >> /mon/import >> --append >> --incremental >> append >> --check-column >> request_id >> --last-value >> 2000000 >> >> The following information is from the sterr and stdout log files. >> >> >> /var/log/hadoop-0.20-mapreduce/userlogs/job_201301231648_0029/attempt_201301231648_0029_m_000000_0 >> # more stderr >> No such sqoop tool: sqoop. See 'sqoop help'. >> Intercepting System.exit(1) >> Failing Oozie Launcher, Main class >> [org.apache.oozie.action.hadoop.SqoopMain], exit code [1] >> >> >> >> /var/log/hadoop-0.20-mapreduce/userlogs/job_201301231648_0029/attempt_201301231648_0029_m_000000_0 >> # more stdout >> ... >> ... >> ... >> >>> Invoking Sqoop command line now >>> >> >> 1598 [main] WARN org.apache.sqoop.tool.SqoopTool - $SQOOP_CONF_DIR has >> not been set in the environment. Cannot check for additional configuration. >> Intercepting System.exit(1) >> >> <<< Invocation of Main class completed <<< >> >> Failing Oozie Launcher, Main class >> [org.apache.oozie.action.hadoop.SqoopMain], exit code [1] >> >> Oozie Launcher failed, finishing Hadoop job gracefully >> >> Oozie Launcher ends >> >> >> ~Corbett Martin >> Software Architect >> AbsoluteAR Accounts Receivable Services - An NHIN Solution >> >> -----Original Message----- >> From: Harsh J [mailto:[email protected]] >> Sent: Tuesday, January 29, 2013 10:11 PM >> To: <[email protected]> >> Subject: Re: Oozie workflow error - renewing token issue >> >> The job that Oozie launches for your action, which you are observing is >> failing, does its own logs (task logs) show any errors? >> >> On Wed, Jan 30, 2013 at 4:59 AM, Corbett Martin <[email protected]> wrote: >> > Oozie question >> > >> > >> > >> > I'm trying to run an Oozie workflow (sqoop action) from the Hue >> > console and it fails every time. No exception in the oozie log but I >> > see this in the Job Tracker log file. >> > >> > >> > >> > Two primary issues seem to be >> > >> > 1. Client mapred tries to renew a token with renewer specified as mr >> > token >> > >> > >> > >> > And >> > >> > >> > >> > 2. Cannot find class for token kind MAPREDUCE_DELEGATION_TOKEN >> > >> > >> > >> > Any ideas how to get past this? >> > >> > >> > >> > Full Stacktrace: >> > >> > >> > >> > 2013-01-29 17:11:28,860 INFO >> > >> > org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager: >> > Creating password for identifier: owner=hdfs, renewer=mr token, >> > realUser=oozie, issueDate=1359501088860, maxDate=1360105888860, >> > sequenceNumber=75, masterKeyId=8 >> > >> > 2013-01-29 17:11:28,871 INFO >> > >> > org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager: >> > Creating password for identifier: owner=hdfs, renewer=mr token, >> > realUser=oozie, issueDate=1359501088871, maxDate=1360105888871, >> > sequenceNumber=76, masterKeyId=8 >> > >> > 2013-01-29 17:11:29,202 INFO >> > org.apache.hadoop.mapreduce.security.token.DelegationTokenRenewal: >> > registering token for renewal for service =10.204.12.62:8021 and jobID >> > = >> > job_201301231648_0029 >> > >> > 2013-01-29 17:11:29,211 INFO >> > >> > org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager: >> > Token renewal requested for identifier: owner=hdfs, renewer=mr token, >> > realUser=oozie, issueDate=1359501088871, maxDate=1360105888871, >> > sequenceNumber=76, masterKeyId=8 >> > >> > 2013-01-29 17:11:29,211 ERROR >> > org.apache.hadoop.security.UserGroupInformation: >> > PriviledgedActionException as:mapred (auth:SIMPLE) >> > cause:org.apache.hadoop.security.AccessControlException: Client mapred >> > tries to renew a token with renewer specified as mr token >> > >> > 2013-01-29 17:11:29,211 WARN org.apache.hadoop.security.token.Token: >> > Cannot find class for token kind MAPREDUCE_DELEGATION_TOKEN >> > >> > 2013-01-29 17:11:29,211 INFO org.apache.hadoop.ipc.Server: IPC Server >> > handler 9 on 8021, call renewDelegationToken(Kind: >> > MAPREDUCE_DELEGATION_TOKEN, Service: 10.204.12.62:8021, Ident: 00 04 >> > 68 64 >> > 66 73 08 6d 72 20 74 6f 6b 65 6e 05 6f 6f 7a 69 65 8a 01 3c 88 94 58 >> > 67 8a >> > 01 3c ac a0 dc 67 4c 08), rpc version=2, client version=28, >> > methodsFingerPrint=1830206421 from 10.204.12.62:9706: error: >> > org.apache.hadoop.security.AccessControlException: Client mapred tries >> > to renew a token with renewer specified as mr token >> > >> > org.apache.hadoop.security.AccessControlException: Client mapred tries >> > to renew a token with renewer specified as mr token >> > >> > at >> > org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSec >> > retManager.renewToken(AbstractDelegationTokenSecretManager.java:274) >> > >> > at >> > org.apache.hadoop.mapred.JobTracker.renewDelegationToken(JobTracker.ja >> > va:3738) >> > >> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native >> > Method) >> > >> > at >> > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.j >> > ava:39) >> > >> > at >> > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccess >> > orImpl.java:25) >> > >> > at java.lang.reflect.Method.invoke(Method.java:597) >> > >> > at >> > org.apache.hadoop.ipc.WritableRpcEngine$Server$WritableRpcInvoker.call >> > (WritableRpcEngine.java:474) >> > >> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:898) >> > >> > at >> > org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1693) >> > >> > at >> > org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1689) >> > >> > at java.security.AccessController.doPrivileged(Native >> > Method) >> > >> > at javax.security.auth.Subject.doAs(Subject.java:396) >> > >> > at >> > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformat >> > ion.java:1332) >> > >> > at >> > org.apache.hadoop.ipc.Server$Handler.run(Server.java:1687) >> > >> > 2013-01-29 17:11:29,212 ERROR >> > org.apache.hadoop.security.UserGroupInformation: >> > PriviledgedActionException as:mapred (auth:SIMPLE) >> > >> > cause:org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.AccessControlException): >> > Client mapred tries to renew a token with renewer specified as mr >> > token >> > >> > at >> > org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSec >> > retManager.renewToken(AbstractDelegationTokenSecretManager.java:274) >> > >> > at >> > org.apache.hadoop.mapred.JobTracker.renewDelegationToken(JobTracker.ja >> > va:3738) >> > >> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native >> > Method) >> > >> > at >> > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.j >> > ava:39) >> > >> > at >> > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccess >> > orImpl.java:25) >> > >> > at java.lang.reflect.Method.invoke(Method.java:597) >> > >> > at >> > org.apache.hadoop.ipc.WritableRpcEngine$Server$WritableRpcInvoker.call >> > (WritableRpcEngine.java:474) >> > >> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:898) >> > >> > at >> > org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1693) >> > >> > at >> > org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1689) >> > >> > at java.security.AccessController.doPrivileged(Native >> > Method) >> > >> > at javax.security.auth.Subject.doAs(Subject.java:396) >> > >> > at >> > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformat >> > ion.java:1332) >> > >> > at >> > org.apache.hadoop.ipc.Server$Handler.run(Server.java:1687) >> > >> > >> > >> > 2013-01-29 17:11:29,212 WARN org.apache.hadoop.security.token.Token: >> > Cannot find class for token kind MAPREDUCE_DELEGATION_TOKEN >> > >> > 2013-01-29 17:11:29,212 ERROR >> > org.apache.hadoop.mapreduce.security.token.DelegationTokenRenewal: >> > Exception renewing tokenKind: MAPREDUCE_DELEGATION_TOKEN, Service: >> > 10.204.12.62:8021, >> > Ident: 00 04 68 64 66 73 08 6d 72 20 74 6f 6b 65 6e 05 6f 6f 7a 69 65 >> > 8a 01 3c 88 94 58 67 8a 01 3c ac a0 dc 67 4c 08. Not rescheduled >> > >> > >> > org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.AccessControlException): >> > Client mapred tries to renew a token with renewer specified as mr >> > token >> > >> > at >> > org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSec >> > retManager.renewToken(AbstractDelegationTokenSecretManager.java:274) >> > >> > at >> > org.apache.hadoop.mapred.JobTracker.renewDelegationToken(JobTracker.ja >> > va:3738) >> > >> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native >> > Method) >> > >> > at >> > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.j >> > ava:39) >> > >> > at >> > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccess >> > orImpl.java:25) >> > >> > at java.lang.reflect.Method.invoke(Method.java:597) >> > >> > at >> > org.apache.hadoop.ipc.WritableRpcEngine$Server$WritableRpcInvoker.call >> > (WritableRpcEngine.java:474) >> > >> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:898) >> > >> > at >> > org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1693) >> > >> > at >> > org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1689) >> > >> > at java.security.AccessController.doPrivileged(Native >> > Method) >> > >> > at javax.security.auth.Subject.doAs(Subject.java:396) >> > >> > at >> > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformat >> > ion.java:1332) >> > >> > at >> > org.apache.hadoop.ipc.Server$Handler.run(Server.java:1687) >> > >> > >> > >> > at org.apache.hadoop.ipc.Client.call(Client.java:1160) >> > >> > at >> > org.apache.hadoop.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngi >> > ne.java:225) >> > >> > at >> > org.apache.hadoop.mapred.$Proxy12.renewDelegationToken(Unknown >> > Source) >> > >> > at >> > org.apache.hadoop.mapred.JobClient$Renewer.renew(JobClient.java:541) >> > >> > at >> > org.apache.hadoop.security.token.Token.renew(Token.java:372) >> > >> > at >> > org.apache.hadoop.mapreduce.security.token.DelegationTokenRenewal$Rene >> > walTimerTask$1.run(DelegationTokenRenewal.java:218) >> > >> > at >> > org.apache.hadoop.mapreduce.security.token.DelegationTokenRenewal$Rene >> > walTimerTask$1.run(DelegationTokenRenewal.java:214) >> > >> > at java.security.AccessController.doPrivileged(Native >> > Method) >> > >> > at javax.security.auth.Subject.doAs(Subject.java:396) >> > >> > at >> > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformat >> > ion.java:1332) >> > >> > at >> > org.apache.hadoop.mapreduce.security.token.DelegationTokenRenewal$Rene >> > walTimerTask.run(DelegationTokenRenewal.java:213) >> > >> > at java.util.TimerThread.mainLoop(Timer.java:512) >> > >> > at java.util.TimerThread.run(Timer.java:462) >> > >> > >> > >> > ~Corbett Martin >> > >> > Software Architect >> > >> > AbsoluteAR Accounts Receivable Services - An NHIN Solution >> > >> > >> > >> > >> > ________________________________ >> > This message and its contents (to include attachments) are the >> > property of National Health Systems, Inc. and may contain confidential >> > and proprietary information. This email and any files transmitted with >> > it are intended solely for the use of the individual or entity to whom >> > they are addressed. >> > You are hereby notified that any unauthorized disclosure, copying, or >> > distribution of this message, or the taking of any unauthorized action >> > based on information contained herein is strictly prohibited. >> > Unauthorized use of information contained herein may subject you to >> > civil and criminal prosecution and penalties. If you are not the >> > intended recipient, you should delete this message immediately and >> > notify the sender immediately by telephone or by replying to this >> > transmission. >> >> >> >> -- >> Harsh J >> >> This message and its contents (to include attachments) are the property of >> National Health Systems, Inc. and may contain confidential and proprietary >> information. This email and any files transmitted with it are intended >> solely for the use of the individual or entity to whom they are addressed. >> You are hereby notified that any unauthorized disclosure, copying, or >> distribution of this message, or the taking of any unauthorized action based >> on information contained herein is strictly prohibited. Unauthorized use of >> information contained herein may subject you to civil and criminal >> prosecution and penalties. If you are not the intended recipient, you should >> delete this message immediately and notify the sender immediately by >> telephone or by replying to this transmission. > > > > > -- > Alejandro > > -- Harsh J
