I have also tried running the job again using the Shell action to run the hive script, I've tried doing this as another user (using sudo; and making sure the share lib folder is copied to that user's HDFS folder). The -doas flag doesn't work for some reason - I get a 500 internal server error.
In either case I still get the same errors - JA018, although for the Shell action the log looks like this: 140827143536335-oozie-oozi-W@ScoreData] Start action [0000003-140827143536335-oozie-oozi-W@ScoreData] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10] 2014-08-27 15:16:30,354 WARN ShellActionExecutor:542 - USER[cxr] GROUP[-] TOKEN[] APP[HustingsJe-v1] JOB[0000003-140827143536335-oozie-oozi-W] ACTION[0000003-140827143536335-oozie-oozi-W@ScoreData] credentials is null for the action 2014-08-27 15:16:31,110 INFO ShellActionExecutor:539 - USER[cxr] GROUP[-] TOKEN[] APP[HustingsJe-v1] JOB[0000003-140827143536335-oozie-oozi-W] ACTION[0000003-140827143536335-oozie-oozi-W@ScoreData] Trying to get job [job_1409150341917_0007], attempt [1] 2014-08-27 15:16:31,127 INFO ShellActionExecutor:539 - USER[cxr] GROUP[-] TOKEN[] APP[HustingsJe-v1] JOB[0000003-140827143536335-oozie-oozi-W] ACTION[0000003-140827143536335-oozie-oozi-W@ScoreData] checking action, external ID [job_1409150341917_0007] status [RUNNING] 2014-08-27 15:16:31,129 WARN ActionStartXCommand:542 - USER[cxr] GROUP[-] TOKEN[] APP[HustingsJe-v1] JOB[0000003-140827143536335-oozie-oozi-W] ACTION[0000003-140827143536335-oozie-oozi-W@ScoreData] [***0000003-140827143536335-oozie-oozi-W@ScoreData***]Action status=RUNNING 2014-08-27 15:16:31,129 WARN ActionStartXCommand:542 - USER[cxr] GROUP[-] TOKEN[] APP[HustingsJe-v1] JOB[0000003-140827143536335-oozie-oozi-W] ACTION[0000003-140827143536335-oozie-oozi-W@ScoreData] [***0000003-140827143536335-oozie-oozi-W@ScoreData***]Action updated in DB! 2014-08-27 15:17:09,865 INFO CallbackServlet:539 - USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000003-140827143536335-oozie-oozi-W] ACTION[0000003-140827143536335-oozie-oozi-W@ScoreData] callback for action [0000003-140827143536335-oozie-oozi-W@ScoreData] 2014-08-27 15:17:10,009 INFO ShellActionExecutor:539 - USER[cxr] GROUP[-] TOKEN[] APP[HustingsJe-v1] JOB[0000003-140827143536335-oozie-oozi-W] ACTION[0000003-140827143536335-oozie-oozi-W@ScoreData] Trying to get job [job_1409150341917_0007], attempt [1] 2014-08-27 15:17:10,114 INFO ShellActionExecutor:539 - USER[cxr] GROUP[-] TOKEN[] APP[HustingsJe-v1] JOB[0000003-140827143536335-oozie-oozi-W] ACTION[0000003-140827143536335-oozie-oozi-W@ScoreData] action completed, external ID [null] 2014-08-27 15:17:10,121 WARN ShellActionExecutor:542 - USER[cxr] GROUP[-] TOKEN[] APP[HustingsJe-v1] JOB[0000003-140827143536335-oozie-oozi-W] ACTION[0000003-140827143536335-oozie-oozi-W@ScoreData] Launcher ERROR, reason: Main class [org.apache.oozie.action.hadoop.ShellMain], exit code [1] 2014-08-27 15:17:10,150 INFO ActionEndXCommand:539 - USER[cxr] GROUP[-] TOKEN[] APP[HustingsJe-v1] JOB[0000003-140827143536335-oozie-oozi-W] ACTION[0000003-140827143536335-oozie-oozi-W@ScoreData] end executor for wf action 0000003-140827143536335-oozie-oozi-W with wf job 0000003-140827143536335-oozie-oozi-W 2014-08-27 15:17:10,189 INFO ActionEndXCommand:539 - USER[cxr] GROUP[-] TOKEN[] APP[HustingsJe-v1] JOB[0000003-140827143536335-oozie-oozi-W] ACTION[0000003-140827143536335-oozie-oozi-W@ScoreData] ERROR is considered as FAILED for SLA On 27 August 2014 12:07, Charles Robertson <[email protected]> wrote: > Hi Mohammad, > > I have copied the whole /user/oozie/share folder to /user/root/share. HDFS > permissions are turned off (dfs.permissions.enabled = false). I have > rerun the job (using the Hive action, not the Shell action) and this is > from the job log: > > 2014-08-27 11:00:33,349 INFO HiveActionExecutor:539 - USER[root] GROUP[-] > TOKEN[] APP[HustingsJe-v1] JOB[0000002-140827104656981-oozie-oozi-W] > ACTION[0000002-140827104656981-oozie-oozi-W@ScoreData] Trying to get job > [job_1409136406325_0007], attempt [1] > 2014-08-27 11:00:33,422 INFO HiveActionExecutor:539 - USER[root] GROUP[-] > TOKEN[] APP[HustingsJe-v1] JOB[0000002-140827104656981-oozie-oozi-W] > ACTION[0000002-140827104656981-oozie-oozi-W@ScoreData] action completed, > external ID [null] > 2014-08-27 11:00:33,431 WARN HiveActionExecutor:542 - USER[root] GROUP[-] > TOKEN[] APP[HustingsJe-v1] JOB[0000002-140827104656981-oozie-oozi-W] > ACTION[0000002-140827104656981-oozie-oozi-W@ScoreData] Launcher ERROR, > reason: Main class [org.apache.oozie.action.hadoop.HiveMain], main() threw > exception, org/apache/hadoop/hive/conf/HiveConf > 2014-08-27 11:00:33,431 WARN HiveActionExecutor:542 - USER[root] GROUP[-] > TOKEN[] APP[HustingsJe-v1] JOB[0000002-140827104656981-oozie-oozi-W] > ACTION[0000002-140827104656981-oozie-oozi-W@ScoreData] Launcher > exception: org/apache/hadoop/hive/conf/HiveConf > java.lang.NoClassDefFoundError: org/apache/hadoop/hive/conf/HiveConf > at org.apache.oozie.action.hadoop.HiveMain.setUpHiveSite(HiveMain.java:182) > at org.apache.oozie.action.hadoop.HiveMain.run(HiveMain.java:196) > at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:38) > at org.apache.oozie.action.hadoop.HiveMain.main(HiveMain.java:66) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at > org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:225) > at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54) > at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:430) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342) > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1594) > at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163) > Caused by: java.lang.ClassNotFoundException: > org.apache.hadoop.hive.conf.HiveConf > at java.net.URLClassLoader$1.run(URLClassLoader.java:366) > at java.net.URLClassLoader$1.run(URLClassLoader.java:355) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:354) > at java.lang.ClassLoader.loadClass(ClassLoader.java:425) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) > at java.lang.ClassLoader.loadClass(ClassLoader.java:358) > ... 17 more > > 2014-08-27 11:00:33,450 INFO ActionEndXCommand:539 - USER[root] GROUP[-] > TOKEN[] APP[HustingsJe-v1] JOB[0000002-140827104656981-oozie-oozi-W] > ACTION[0000002-140827104656981-oozie-oozi-W@ScoreData] end executor for > wf action 0000002-140827104656981-oozie-oozi-W with wf job > 0000002-140827104656981-oozie-oozi-W > 2014-08-27 11:00:33,483 INFO ActionEndXCommand:539 - USER[root] GROUP[-] > TOKEN[] APP[HustingsJe-v1] JOB[0000002-140827104656981-oozie-oozi-W] > ACTION[0000002-140827104656981-oozie-oozi-W@ScoreData] ERROR is > considered as FAILED for SLA > > (I know running jobs as root and having permissions turned off is not good > practice, but this is mainly a learning exercise for me, not a production > system.) > > Thanks for your help, > Charles > > > On 27 August 2014 09:14, Mohammad Islam <[email protected]> > wrote: > >> In general, running ooze as root is not a good practice. >> >> If you use 'root', the whole share lib should be in /user/root/share/lib/ >> directory with proper permission for user 'root'. >> >> After that, can you please run shell action and share the error? >> >> Regards, >> Mohammad >> >> >> On Wednesday, August 27, 2014 12:55 AM, Charles Robertson < >> [email protected]> wrote: >> >> >> >> I believe so - the install process put a large number of JARs in HDFS >> under >> /user/oozie/share/lib and I copied the Hive subfolder to >> /user/root/share/lib just in case. (The workflow is running as root). >> >> Regards, >> Charles >> >> >> >> On 26 August 2014 21:40, Mohammad Islam <[email protected]> >> wrote: >> >> > Did you upload the 'sharelib' correctly? >> > >> > >> > >> > On Tuesday, August 26, 2014 8:06 AM, Charles Robertson < >> > [email protected]> wrote: >> > >> > >> > >> > Update: I tried switching the Hive action to a Shell action and calling >> the >> > Hive command. The log was less informative, but I still received error >> code >> > JA018 with error message 'Main class >> > [org.apache.oozie.action.hadoop.ShellMain], exit code [1]'. >> > >> > This smells to me like it can't find the executables when running the >> job. >> > The log shows that 'User' is root (I have submitted and run the job >> from an >> > SSH console logged in as root) which should have access to the JARs - >> > although I can run the Hive command from the console when logged in as >> > root. >> > >> > Perhaps oozie is impersonating root, and not getting the same classpath >> or >> > environment variables or somesuch? >> > >> > Regards, >> > Charles >> > >> > >> > >> > >> > >> > On 26 August 2014 13:33, Charles Robertson <[email protected] >> > >> > wrote: >> > >> > > Hi all, >> > > >> > > My Hive action is failing with the following error: >> > > >> > > Launcher exception: org/apache/hadoop/hive/conf/HiveConf >> > > java.lang.NoClassDefFoundError: org/apache/hadoop/hive/conf/HiveConf >> > > at >> > > >> org.apache.oozie.action.hadoop.HiveMain.setUpHiveSite(HiveMain.java:182) >> > > at org.apache.oozie.action.hadoop.HiveMain.run(HiveMain.java:196) >> > > at >> > > org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:38) >> > > at org.apache.oozie.action.hadoop.HiveMain.main(HiveMain.java:66) >> > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >> > > at >> > > >> > >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) >> > > at >> > > >> > >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >> > > at java.lang.reflect.Method.invoke(Method.java:606) >> > > at >> > > >> > >> org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:225) >> > > at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54) >> > > at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:430) >> > > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342) >> > > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168) >> > > at java.security.AccessController.doPrivileged(Native Method) >> > > at javax.security.auth.Subject.doAs(Subject.java:415) >> > > at >> > > >> > >> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1594) >> > > at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163) >> > > Caused by: java.lang.ClassNotFoundException: >> > > org.apache.hadoop.hive.conf.HiveConf >> > > at java.net.URLClassLoader$1.run(URLClassLoader.java:366) >> > > at java.net.URLClassLoader$1.run(URLClassLoader.java:355) >> > > at java.security.AccessController.doPrivileged(Native Method) >> > > at java.net.URLClassLoader.findClass(URLClassLoader.java:354) >> > > at java.lang.ClassLoader.loadClass(ClassLoader.java:425) >> > > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) >> > > at java.lang.ClassLoader.loadClass(ClassLoader.java:358) >> > > ... 17 more >> > > >> > > Error Code: JA018 >> > > Error Message: org/apache/hadoop/hive/conf/HiveConf >> > > >> > > This is my hive action: >> > > <hive xmlns="uri:oozie:hive-action:0.2"> >> > > <job-tracker>[private DNS name]:8050</job-tracker> >> > > <name-node>hdfs://[private DNS name]:8020</name-node> >> > > <script>score_tweets.sql</script> >> > > </hive> >> > > >> > > job.properties: >> > > nameNode=hdfs://[private DNS name]:8020 >> > > jobTracker=[private DNS name]:8050 >> > > queueName=default >> > > oozie.libpath=${nameNode}/user/oozie/share/lib >> > > oozie.wf.application.path=${nameNode}/user/root/oozie >> > > >> > > I am using oozie 4.0.0 on HDP 2.1 >> > > >> > > Googling produced some useful things (see >> > > >> > >> https://support.pivotal.io/hc/en-us/articles/202563453-Oozie-hive-action-fails-with-java-lang-NoClassDefFoundError-org-apache-hadoop-hive-conf-HiveConf- >> > > and >> > > >> > >> http://stackoverflow.com/questions/18369605/error-while-running-hive-action-in-oozie >> > ), >> > > but I've tried these, and I'm still getting the same error. >> > > >> > > I have: >> > > - ensured that hive-exec is in /user/oozie/share/lib (also in >> > > /user/root/share/lib as this job is run as root) >> > > - ensured that hive-exec exists in a path in the common.loader >> property >> > in >> > > the catalina.properties file >> > > - even put the 'oozie.service.WorkflowAppService.system.libpath' >> property >> > > into hive-site.xml like the stackoverflow answer suggested, even >> though >> > > this doesn't make much sense to me (it is in oozie-site.xml and I have >> > > overriden it to point hard-coded to /user/oozie/share/lib) >> > > - not done anything with hive-default.xml since the documentation says >> > > this is now ignored. >> > > >> > > Does anyone have any suggestions? >> > > >> > > Thanks, >> > > Charles >> > > >> > >> > >
