Re: 答复: Launcher exception: java.lang.NoClassDefFoundError: org/apache/hadoop/hive/conf/HiveConf

[Peter Cseh]

Hi,
I've just created a build from Oozie's master and checked and it seems
hive-exec jar is needed and included in hive2 action's sharelib:


> GezapetiMBP:bin gezapeti$ hdfs dfs -ls share/lib/lib_20160911145035/hive2
> Found 48 items
> ...
> -rw-r--r--   3 gezapeti supergroup 115618 2016-09-11 14:50
> share/lib/lib_20160911145035/hive2/hive-beeline-0.13.1.jar
> -rw-r--r--   3 gezapeti supergroup   15141449 2016-09-11 14:50
> share/lib/lib_20160911145035/hive2/hive-exec-0.13.1.jar

...

I don't know what's the situation for other Hive versions.

I've compiled Oozie with:
mvn clean package assembly:single -DskipTests -Puber -Dhadoop.version=2.4.0

Created sharelib using:

./oozie-setup.sh sharelib create -fs hdfs://localhost:9000 -locallib
${OOZIE_SRC_DIR}/sharelib/target/oozie-sharelib-4.3.0-SNAPSHOT/share/

If -locallib is not specified, the oozie-sharelib-version.tar.gz from
Oozie's home dir (defined by oozie.home.dir property) will be used.

Make sure that the sharelib version matches your Oozie and Hive versions.

Hope it helps,

gp


​

答复: Launcher exception: java.lang.NoClassDefFoundError: org/apache/hadoop/hive/conf/HiveConf

[Huang Meilong]

Thank you Peter, it's helpful. I installed share lib via command:

su -l oozie -c "${OOZIE_HOME}/bin/oozie-setup.sh sharelib create -fs 
hdfs://localhost:9000"


and there are some jars but hive-exec.jar is not there (hive-exec.jar is in 
/lib/lib_xxx/hive not int /lib/lib_xxx/hive2). How can I install the latest 
sharelib?


发件人: Peter Cseh <gezap...@cloudera.com>
发送时间: 2016年9月9日 15:45:49
收件人: user@oozie.apache.org
主题: Re: Launcher exception: java.lang.NoClassDefFoundError: 
org/apache/hadoop/hive/conf/HiveConf

Hi,

It looks like the org.apache.hadoop.hive.conf.HiveConf class is missing
from the classpath when the action runs.
Can you check the output of the  oozie admin -shareliblist hive2 and check
that the hive-exec.jar is there?
You can find more information about the sharelib and how to install it here
<http://blog.cloudera.com/blog/2014/05/how-to-use-the-sharelib-in-apache-oozie-cdh-5/>
.

Peter

On Fri, Sep 9, 2016 at 6:35 AM, Huang Meilong <ims...@outlook.com> wrote:

> Hello,
>
>
> I'm using oozie 4.2.0 on a HA cluster, I got a launcher exception when I
> run the oozie example app hive2:
>
>
> 2016-09-09 10:10:44,927  WARN Hive2ActionExecutor:523 -
> SERVER[emr-header-1.cluster-500031470] USER[oozie] GROUP[-] TOKEN[]
> APP[hive2-wf] JOB[004-160908174647725-oozie-oozi-W] ACTION[004-
> 160908174647725-oozie-oozi-W@hive2-node] Launcher ERROR, reason: Main
> class [org.apache.oozie.action.hadoop.Hive2Main], main() threw exception,
> org/apache/hadoop/hive/conf/HiveConf
> 2016-09-09 10:10:44,931  WARN Hive2ActionExecutor:523 -
> SERVER[emr-header-1.cluster-500031470] USER[oozie] GROUP[-] TOKEN[]
> APP[hive2-wf] JOB[004-160908174647725-oozie-oozi-W] ACTION[004-
> 160908174647725-oozie-oozi-W@hive2-node] Launcher exception:
> org/apache/hadoop/hive/conf/HiveConf
> java.lang.NoClassDefFoundError: org/apache/hadoop/hive/conf/HiveConf
> at org.apache.oozie.action.hadoop.Hive2Main.runBeeline(Hive2Main.java:240)
> at org.apache.oozie.action.hadoop.Hive2Main.run(Hive2Main.java:223)
> at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:47)
> at org.apache.oozie.action.hadoop.Hive2Main.main(Hive2Main.java:56)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(
> NativeMethodAccessorImpl.java:62)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(
> DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.oozie.action.hadoop.LauncherMapper.map(
> LauncherMapper.java:236)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
> at org.apache.hadoop.mapred.LocalContainerLauncher$
> EventHandler.runSubtask(LocalContainerLauncher.java:380)
> at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runTask(
> LocalContainerLauncher.java:301)
> at org.apache.hadoop.mapred.LocalContainerLauncher$
> EventHandler.access$200(LocalContainerLauncher.java:187)
> at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler$1.run(
> LocalContainerLauncher.java:230)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hive.conf.
> HiveConf
> at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> ... 21 more
>
> job definition:
>
>
> 
> 
> 
> 
>
> 
> 
> ${jobTracker}
> ${nameNode}
> 
> 
> 
> 
> 
> 
> mapred.job.queue.name
> ${queueName}
> 
> 
> ${jdbcURL}
> script.q
> INPUT=/user/${wf:user()}/${examplesRoot}/input-data/
> table
> OUTPUT=/user/${wf:user()}/${examplesRoot}/output-
> data/hive2
> 
> 
> 
> 
>
> 
> Hive2 (Beeline) action failed, error
> message[${wf:errorMessage(wf:lastErrorNode())}]
> 
> 
> 
>
> job configuration:
>
>
> 
&g

Re: Launcher exception: java.lang.NoClassDefFoundError: org/apache/hadoop/hive/conf/HiveConf

[Peter Cseh]

Hi,

It looks like the org.apache.hadoop.hive.conf.HiveConf class is missing
from the classpath when the action runs.
Can you check the output of the  oozie admin -shareliblist hive2 and check
that the hive-exec.jar is there?
You can find more information about the sharelib and how to install it here

.

Peter

On Fri, Sep 9, 2016 at 6:35 AM, Huang Meilong  wrote:

> Hello,
>
>
> I'm using oozie 4.2.0 on a HA cluster, I got a launcher exception when I
> run the oozie example app hive2:
>
>
> 2016-09-09 10:10:44,927  WARN Hive2ActionExecutor:523 -
> SERVER[emr-header-1.cluster-500031470] USER[oozie] GROUP[-] TOKEN[]
> APP[hive2-wf] JOB[004-160908174647725-oozie-oozi-W] ACTION[004-
> 160908174647725-oozie-oozi-W@hive2-node] Launcher ERROR, reason: Main
> class [org.apache.oozie.action.hadoop.Hive2Main], main() threw exception,
> org/apache/hadoop/hive/conf/HiveConf
> 2016-09-09 10:10:44,931  WARN Hive2ActionExecutor:523 -
> SERVER[emr-header-1.cluster-500031470] USER[oozie] GROUP[-] TOKEN[]
> APP[hive2-wf] JOB[004-160908174647725-oozie-oozi-W] ACTION[004-
> 160908174647725-oozie-oozi-W@hive2-node] Launcher exception:
> org/apache/hadoop/hive/conf/HiveConf
> java.lang.NoClassDefFoundError: org/apache/hadoop/hive/conf/HiveConf
> at org.apache.oozie.action.hadoop.Hive2Main.runBeeline(Hive2Main.java:240)
> at org.apache.oozie.action.hadoop.Hive2Main.run(Hive2Main.java:223)
> at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:47)
> at org.apache.oozie.action.hadoop.Hive2Main.main(Hive2Main.java:56)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(
> NativeMethodAccessorImpl.java:62)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(
> DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.oozie.action.hadoop.LauncherMapper.map(
> LauncherMapper.java:236)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
> at org.apache.hadoop.mapred.LocalContainerLauncher$
> EventHandler.runSubtask(LocalContainerLauncher.java:380)
> at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runTask(
> LocalContainerLauncher.java:301)
> at org.apache.hadoop.mapred.LocalContainerLauncher$
> EventHandler.access$200(LocalContainerLauncher.java:187)
> at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler$1.run(
> LocalContainerLauncher.java:230)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hive.conf.
> HiveConf
> at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> ... 21 more
>
> job definition:
>
>
> 
> 
> 
> 
>
> 
> 
> ${jobTracker}
> ${nameNode}
> 
> 
> 
> 
> 
> 
> mapred.job.queue.name
> ${queueName}
> 
> 
> ${jdbcURL}
> script.q
> INPUT=/user/${wf:user()}/${examplesRoot}/input-data/
> table
> OUTPUT=/user/${wf:user()}/${examplesRoot}/output-
> data/hive2
> 
> 
> 
> 
>
> 
> Hive2 (Beeline) action failed, error
> message[${wf:errorMessage(wf:lastErrorNode())}]
> 
> 
> 
>
> job configuration:
>
>
> 
>   
> examplesRoot
> examples
>   
>   
> oozie.wf.application.path
> hdfs://emr-cluster/user/oozie/examples/apps/hive2
>   
>   
> oozie.use.system.libpath
> true
>   
>   
> queueName
> default
>   
>   
> jdbcURL
> jdbc:hive2://localhost:1/default
>   
>   
> user.name
> oozie
>   
>   
> jobTracker
> rm1,rm2
>   
>   
> mapreduce.job.user.name
> oozie
>   
>   
> nameNode
> hdfs://emr-cluster
>   
> 
>
>
>
> How can I fix this error, can you give me a hand, thanks in advance!
>
>


-- 
Peter Cseh
Software Engineer

Launcher exception: java.lang.NoClassDefFoundError: org/apache/hadoop/hive/conf/HiveConf

[Huang Meilong]

Hello,


I'm using oozie 4.2.0 on a HA cluster, I got a launcher exception when I run 
the oozie example app hive2:


2016-09-09 10:10:44,927  WARN Hive2ActionExecutor:523 - 
SERVER[emr-header-1.cluster-500031470] USER[oozie] GROUP[-] TOKEN[] 
APP[hive2-wf] JOB[004-160908174647725-oozie-oozi-W] 
ACTION[004-160908174647725-oozie-oozi-W@hive2-node] Launcher ERROR, reason: 
Main class [org.apache.oozie.action.hadoop.Hive2Main], main() threw exception, 
org/apache/hadoop/hive/conf/HiveConf
2016-09-09 10:10:44,931  WARN Hive2ActionExecutor:523 - 
SERVER[emr-header-1.cluster-500031470] USER[oozie] GROUP[-] TOKEN[] 
APP[hive2-wf] JOB[004-160908174647725-oozie-oozi-W] 
ACTION[004-160908174647725-oozie-oozi-W@hive2-node] Launcher exception: 
org/apache/hadoop/hive/conf/HiveConf
java.lang.NoClassDefFoundError: org/apache/hadoop/hive/conf/HiveConf
at org.apache.oozie.action.hadoop.Hive2Main.runBeeline(Hive2Main.java:240)
at org.apache.oozie.action.hadoop.Hive2Main.run(Hive2Main.java:223)
at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:47)
at org.apache.oozie.action.hadoop.Hive2Main.main(Hive2Main.java:56)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:236)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
at 
org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runSubtask(LocalContainerLauncher.java:380)
at 
org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runTask(LocalContainerLauncher.java:301)
at 
org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.access$200(LocalContainerLauncher.java:187)
at 
org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler$1.run(LocalContainerLauncher.java:230)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.ClassNotFoundException: 
org.apache.hadoop.hive.conf.HiveConf
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 21 more

job definition:









${jobTracker}
${nameNode}






mapred.job.queue.name
${queueName}


${jdbcURL}
script.q

INPUT=/user/${wf:user()}/${examplesRoot}/input-data/table

OUTPUT=/user/${wf:user()}/${examplesRoot}/output-data/hive2






Hive2 (Beeline) action failed, error 
message[${wf:errorMessage(wf:lastErrorNode())}]




job configuration:



  
examplesRoot
examples
  
  
oozie.wf.application.path
hdfs://emr-cluster/user/oozie/examples/apps/hive2
  
  
oozie.use.system.libpath
true
  
  
queueName
default
  
  
jdbcURL
jdbc:hive2://localhost:1/default
  
  
user.name
oozie
  
  
jobTracker
rm1,rm2
  
  
mapreduce.job.user.name
oozie
  
  
nameNode
hdfs://emr-cluster
  




How can I fix this error, can you give me a hand, thanks in advance!