Rohini,
Thanks for pointing out where the Job's configuration is and I did find it.
Learned something today!
However, the hbase.jar is actually in the config. I am pasting the exact values
here.
mapreduce.job.cache.files:
hdfs://hqitrtd91.pclc0.merkle.local:8020/user/cloudera/oozie-oozi/0000052-130210101321239-oozie-oozi-W/java-aggregator--java/java-launcher.jar,hdfs://hqitrtd91.pclc0.merkle.local:8020/user/cloudera/ltd/cr-wf2/lib/cr-1.0-SNAPSHOT.jar,/user/cloudera/ltd/cr-wf2/lib/cr-run-config.xml#cr-run-config.xml,hdfs://hqitrtd91.pclc0.merkle.local:8020/user/oozie/hbase-shared/hbase.jar,hdfs://hqitrtd91.pclc0.merkle.local:8020/user/oozie/hbase-shared/zookeeper.jar,hdfs://hqitrtd91.pclc0.merkle.local:8020/user/oozie/share/lib/oozie/json-simple-1.1.jar
mapreduce.job.cache.files.filesizes: 19018,147377,1508,3258651,1301515,16046
mapreduce.job.cache.files.timestamps:
1361298149677,1361297463000,1360880488031,1360619188779,1360619228172,1358357320213
mapreduce.job.cache.files.visibilities true,true,true,true,true,true
===============================
And here is the error:
Error: java.lang.RuntimeException: java.lang.reflect.InvocationTargetException
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:128)
at
org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:621)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:396)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:152)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:147)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:126)
... 7 more
Caused by: java.lang.NoClassDefFoundError:
org/apache/hadoop/hbase/client/HTableInterface
at
com.merkleinc.cr.etl.aggregator.OmnitureHitReducerListener.<init>(OmnitureHitReducerListener.java:53)
at
com.merkleinc.cr.etl.hadoop.ReduceDispatcher.<init>(ReduceDispatcher.java:18)
at com.merkleinc.cr.etl.hadoop.CRReducer.<init>(CRReducer.java:42)
... 12 more
Caused by: java.lang.ClassNotFoundException:
org.apache.hadoop.hbase.client.HTableInterface
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
-----Original Message-----
From: Rohini Palaniswamy [mailto:[email protected]]
Sent: Monday, February 18, 2013 8:15 PM
To: [email protected]
Subject: Re: NoClassDefFoundError:
org/apache/hadoop/hbase/client/HTableInterface
Jinye,
The shared jars of oozie action are added to the hadoop's DistributedCache
so that it is available for the map/reduce tasks in their classpath. You can
verify if the jars are in the distributed cache of a job by looking at that
Job's Configuration mapreduce.job.cache.files property.
In the yarn RM page, go to the job link. On the left side you will see a Job
pane with the following.
Job
Overview
Counters
Configuration
Map tasks
Reduce tasks
Click on the Configuration link. You can then search for cache in the key
section.
Regards,
Rohini
On Sat, Feb 16, 2013 at 9:46 PM, Jinye Luo <[email protected]> wrote:
> Rohini,
>
> Thanks for your reply.
>
> I got the java.class.path from the MR driver log. From what I see,
> hbase.jar is visible to the driver class, because
> "HBaseConfiguration.create()" called from the driver was successful.
> But somehow the jar is not visible to Reducer running from a different
> machine.
>
> Can you give me a bit detail how to find "configuration tab of the
> job"? I looked everywhere in "oozie console" and "yarn
> ResourceManager" and not able to find it.
>
> After all, do you know if there is any document on how to use shared
> jars from an oozie action? I got information from a blog which might
> not be completed.
>
> Thx.
>
> Jinye
>
> -----Original Message-----
> From: Rohini Palaniswamy [mailto:[email protected]]
> Sent: Saturday, February 16, 2013 10:13 PM
> To: [email protected]
> Subject: Re: NoClassDefFoundError:
> org/apache/hadoop/hbase/client/HTableInterface
>
> Jinye,
> This is happening in ApplicationMaster code and you will be seeing
> the error in AM logs.
>
>
> http://svn.apache.org/viewvc/hadoop/common/branches/branch-2.0.1-alpha
> /hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-cli
> ent-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/JobI
> mpl.java?revision=1439715&view=markup
>
> It tries to load the reducer to see if it is a ChainReducer. It is
> able to find your reducer class, but while trying to load it, it
> failed with java.lang.NoClassDefFoundError as
> org/apache/hadoop/hbase/client/HTableInterface
> is not in its classpath.
>
> One thing I can say for sure is that hbase.jar is not in the
> classpath. In which log do you see the java.class.path system property
> printed? Can you check if hbase.jar is in the
> mapreduce.job.cache.files setting of the failed job (Search in the
> Configuration tab of the job).
>
> Regards,
> Rohini
>
>
>
> On Mon, Feb 11, 2013 at 10:18 PM, Jinye Luo <[email protected]> wrote:
>
> > I figured someone smart out there must have done it, hence asking.
> >
> > I have a M/R job accessing Hbase and it is launched by oozie. The
> > job is successful if I put all dependencies in a big jar, however,
> > everyone understands it is not a good idea to deploy hadoop/hbase
> > files along with application jars. Hence I am trying to configure it
> > to use the jars in some shared fashion but with no luck.
> >
> > First I put hbase.jar and zookeeper.jar in /user/oozie/share/lib,
> > but from the log, it is clear that hbase.jar is just not visible in
> > the
> classpath.
> >
> > Then I tried a second option to take advantage of libpath by
> > creating a directory "/user/oozie/hbase-shared" in hdfs and putting
> > the two jars there. And I pointed oozie.libpath to the dir as
> > "oozie.libpath=${nameNode}/user/oozie/hbase-shared" in
> > job.properties
> file.
> > It is doing better because the call to "HBaseConfiguration.create()"
> > from my driver was successful and HBaseConfiguration is a class in
> > hbase.jar, so I know hbase.jar is indeed visible to the driver class.
> > However, somewhere down the line I still received exception of
> "NoClassDefFoundError:
> > org/apache/hadoop/hbase/client/HTableInterface" and I only use
> > HTableInterface in the reducer.
> >
> > So any idea please? Also, please let me know if there is a better
> > way to do it.
> >
> > And here is system properties I found in the log, which clearly
> > shows hbase.jar is in the classpath.
> >
> >
> > #Tue Feb 12 00:22:20 EST 2013
> > java.runtime.name=Java(TM) SE Runtime Environment
> >
> > oozie.action.externalChildIDs.properties=/var/lib/hadoop-yarn/cache/
> > ya
> > rn/nm-local-dir/usercache/cloudera/appcache/application_136064183061
> > 8_
> > 0011/container_1360641830618_0011_01_000002/externalChildIds.propert
> > ie
> > s
> > sun.boot.library.path=/usr/java/jdk1.6.0_31/jre/lib/amd64
> > java.vm.version=20.6-b01
> > hadoop.root.logger=INFO,CLA
> > java.vm.vendor=Sun Microsystems Inc.
> > java.vendor.url=http\://java.sun.com/
> > path.separator=\:
> > java.vm.name=Java HotSpot(TM) 64-Bit Server VM
> > file.encoding.pkg=sun.io sun.java.launcher=SUN_STANDARD
> > user.country=US sun.os.patch.level=unknown
> > java.vm.specification.name=Java Virtual Machine Specification
> >
> > user.dir=/var/lib/hadoop-yarn/cache/yarn/nm-local-dir/usercache/clou
> > de
> > ra/appcache/application_1360641830618_0011/container_1360641830618_0
> > 01
> > 1_01_000002
> > java.runtime.version=1.6.0_31-b04
> > java.awt.graphicsenv=sun.awt.X11GraphicsEnvironment
> > java.endorsed.dirs=/usr/java/jdk1.6.0_31/jre/lib/endorsed
> > os.arch=amd64
> > oozie.job.id=0000015-130210101321239-oozie-oozi-W
> > oozie.action.id=0000015-130210101321239-oozie-oozi-W@java-aggregator
> >
> > java.io.tmpdir=/var/lib/hadoop-yarn/cache/yarn/nm-local-dir/usercach
> > e/
> > cloudera/appcache/application_1360641830618_0011/container_136064183
> > 06
> > 18_0011_01_000002/tmp
> > line.separator=\n
> >
> > oozie.action.output.properties=/var/lib/hadoop-yarn/cache/yarn/nm-lo
> > ca
> > l-dir/usercache/cloudera/appcache/application_1360641830618_0011/con
> > ta iner_1360641830618_0011_01_000002/output.properties
> > java.vm.specification.vendor=Sun Microsystems Inc.
> > os.name=Linux
> > log4j.configuration=container-log4j.properties
> > sun.jnu.encoding=ANSI_X3.4-1968
> >
> > java.library.path=/usr/java/jdk1.6.0_31/jre/lib/amd64/server\:/usr/j
> > av
> > a/jdk1.6.0_31/jre/lib/amd64\:/usr/java/jdk1.6.0_31/jre/../lib/amd64\
> > :/
> > var/lib/hadoop-yarn/cache/yarn/nm-local-dir/usercache/cloudera/appca
> > ch
> > e/application_1360641830618_0011/container_1360641830618_0011_01_000
> > 00
> > 2\:/usr/lib/hadoop/lib/native\:/usr/java/packages/lib/amd64\:/usr/li
> > b6
> > 4\:/lib64\:/lib\:/usr/lib
> > yarn.app.mapreduce.container.log.filesize=0
> >
> > oozie.action.conf.xml=/var/lib/hadoop-yarn/cache/yarn/nm-local-dir/u
> > se
> > rcache/cloudera/appcache/application_1360641830618_0011/container_13
> > 60
> > 641830618_0011_01_000002/action.xml
> > hadoop.metrics.log.level=WARN
> > java.specification.name=Java Platform API Specification
> > java.class.version=50.0
> > java.net.preferIPv4Stack=true
> > sun.management.compiler=HotSpot 64-Bit Tiered Compilers
> > os.version=2.6.32-220.23.1.el6.x86_64
> >
> > yarn.app.mapreduce.container.log.dir=/var/log/hadoop-yarn/containers
> > /a
> > pplication_1360641830618_0011/container_1360641830618_0011_01_000002
> > user.home=/var/lib/hadoop-yarn
> > user.timezone=America/New_York
> > java.awt.printerjob=sun.print.PSPrinterJob
> > file.encoding=ANSI_X3.4-1968
> > java.specification.version=1.6
> >
> > java.class.path=/var/lib/hadoop-yarn/cache/yarn/nm-local-dir/usercac
> > he
> > /cloudera/appcache/application_1360641830618_0011/container_13606418
> > 30
> > 618_0011_01_000002\:/etc/hadoop/conf\:/usr/lib/hadoop/hadoop-annotat
> > io
> > ns-2.0.0-cdh4.1.1.jar\:/usr/lib/hadoop/hadoop-auth.jar\:/usr/lib/had
> > oo
> > p/hadoop-common-2.0.0-cdh4.1.1-tests.jar\:/usr/lib/hadoop/hadoop-aut
> > h-
> > 2.0.0-cdh4.1.1.jar\:/usr/lib/hadoop/hadoop-common-2.0.0-cdh4.1.1.jar\:
> > /usr/lib/hadoop/hadoop-annotations.jar\:/usr/lib/hadoop/hadoop-common.
> > jar\:/usr/lib/hadoop/lib/log4j-1.2.17.jar\:/usr/lib/hadoop/lib/commo
> > ns
> > -logging-1.1.1.jar\:/usr/lib/hadoop/lib/servlet-api-2.5.jar\:/usr/li
> > b/
> > hadoop/lib/jline-0.9.94.jar\:/usr/lib/hadoop/lib/commons-configurati
> > on
> > -1.6.jar\:/usr/lib/hadoop/lib/jersey-json-1.8.jar\:/usr/lib/hadoop/l
> > ib
> > /commons-httpclient-3.1.jar\:/usr/lib/hadoop/lib/jaxb-api-2.2.2.jar\
> > :/
> > usr/lib/hadoop/lib/jetty-6.1.26.cloudera.2.jar\:/usr/lib/hadoop/lib/
> > zo
> > okeeper-3.4.3-cdh4.1.1.jar\:/usr/lib/hadoop/lib/commons-cli-1.2.jar\
> > :/
> > usr/lib/hadoop/lib/jersey-server-1.8.jar\:/usr/lib/hadoop/lib/hue-pl
> > ug
> > ins-2.1.0-cdh4.1.1.jar\:/usr/lib/hadoop/lib/commons-el-1.0.jar\:/usr
> > /l
> > ib/hadoop/lib/jackson-core-asl-1.8.8.jar\:/usr/lib/hadoop/lib/jsp-ap
> > i-
> > 2.1.jar\:/usr/lib/hadoop/lib/commons-net-3.1.jar\:/usr/lib/hadoop/li
> > b/
> > jasper-compiler-5.5.23.jar\:/usr/lib/hadoop/lib/jettison-1.1.jar\:/u
> > sr
> > /lib/hadoop/lib/jsr305-1.3.9.jar\:/usr/lib/hadoop/lib/paranamer-2.3.
> > ja
> > r\:/usr/lib/hadoop/lib/mockito-all-1.8.5.jar\:/usr/lib/hadoop/lib/co
> > mm
> > ons-beanutils-1.7.0.jar\:/usr/lib/hadoop/lib/stax-api-1.0.1.jar\:/us
> > r/
> > lib/hadoop/lib/jets3t-0.6.1.jar\:/usr/lib/hadoop/lib/jasper-runtime-5.
> > 5.23.jar\:/usr/lib/hadoop/lib/jaxb-impl-2.2.3-1.jar\:/usr/lib/hadoop
> > /l
> > ib/protobuf-java-2.4.0a.jar\:/usr/lib/hadoop/lib/jackson-xc-1.8.8.ja
> > r\
> > :/usr/lib/hadoop/lib/commons-digester-1.8.jar\:/usr/lib/hadoop/lib/j
> > et
> > ty-util-6.1.26.cloudera.2.jar\:/usr/lib/hadoop/lib/activation-1.1.ja
> > r\
> > :/usr/lib/hadoop/lib/snappy-java-1.0.4.1.jar\:/usr/lib/hadoop/lib/sl
> > f4
> > j-api-1.6.1.jar\:/usr/lib/hadoop/lib/avro-1.7.1.cloudera.2.jar\:/usr
> > /l
> > ib/hadoop/lib/commons-beanutils-core-1.8.0.jar\:/usr/lib/hadoop/lib/
> > ja
> > ckson-mapper-asl-1.8.8.jar\:/usr/lib/hadoop/lib/jsch-0.1.42.jar\:/us
> > r/
> > lib/hadoop/lib/kfs-0.3.jar\:/usr/lib/hadoop/lib/commons-math-2.1.jar\:
> > /usr/lib/hadoop/lib/xmlenc-0.52.jar\:/usr/lib/hadoop/lib/commons-col
> > le
> > ctions-3.2.1.jar\:/usr/lib/hadoop/lib/guava-11.0.2.jar\:/usr/lib/had
> > oo
> > p/lib/commons-io-2.1.jar\:/usr/lib/hadoop/lib/junit-4.8.2.jar\:/usr/
> > li
> > b/hadoop/lib/jackson-jaxrs-1.8.8.jar\:/usr/lib/hadoop/lib/asm-3.2.ja
> > r\
> > :/usr/lib/hadoop/lib/commons-codec-1.4.jar\:/usr/lib/hadoop/lib/comm
> > on
> > s-lang-2.5.jar\:/usr/lib/hadoop/lib/slf4j-log4j12-1.6.1.jar\:/usr/li
> > b/
> > hadoop/lib/jersey-core-1.8.jar\:/usr/lib/hadoop-hdfs/hadoop-hdfs.jar\:
> > /usr/lib/hadoop-hdfs/hadoop-hdfs-2.0.0-cdh4.1.1-tests.jar\:/usr/lib/
> > ha
> > doop-hdfs/hadoop-hdfs-2.0.0-cdh4.1.1.jar\:/usr/lib/hadoop-hdfs/lib/l
> > og
> > 4j-1.2.17.jar\:/usr/lib/hadoop-hdfs/lib/commons-logging-1.1.1.jar\:/
> > us
> > r/lib/hadoop-hdfs/lib/servlet-api-2.5.jar\:/usr/lib/hadoop-hdfs/lib/
> > jl
> > ine-0.9.94.jar\:/usr/lib/hadoop-hdfs/lib/jetty-6.1.26.cloudera.2.jar\:
> > /usr/lib/hadoop-hdfs/lib/zookeeper-3.4.3-cdh4.1.1.jar\:/usr/lib/hado
> > op
> > -hdfs/lib/commons-daemon-1.0.3.jar\:/usr/lib/hadoop-hdfs/lib/commons
> > -c
> > li-1.2.jar\:/usr/lib/hadoop-hdfs/lib/jersey-server-1.8.jar\:/usr/lib
> > /h
> > adoop-hdfs/lib/commons-el-1.0.jar\:/usr/lib/hadoop-hdfs/lib/jackson-
> > co
> > re-asl-1.8.8.jar\:/usr/lib/hadoop-hdfs/lib/jsp-api-2.1.jar\:/usr/lib
> > /h
> > adoop-hdfs/lib/jsr305-1.3.9.jar\:/usr/lib/hadoop-hdfs/lib/jasper-run
> > ti
> > me-5.5.23.jar\:/usr/lib/hadoop-hdfs/lib/protobuf-java-2.4.0a.jar\:/u
> > sr
> > /lib/hadoop-hdfs/lib/jetty-util-6.1.26.cloudera.2.jar\:/usr/lib/hado
> > op
> > -hdfs/lib/jackson-mapper-asl-1.8.8.jar\:/usr/lib/hadoop-hdfs/lib/xml
> > en
> > c-0.52.jar\:/usr/lib/hadoop-hdfs/lib/guava-11.0.2.jar\:/usr/lib/hado
> > op
> > -hdfs/lib/commons-io-2.1.jar\:/usr/lib/hadoop-hdfs/lib/asm-3.2.jar\:
> > /u
> > sr/lib/hadoop-hdfs/lib/commons-codec-1.4.jar\:/usr/lib/hadoop-hdfs/l
> > ib
> > /commons-lang-2.5.jar\:/usr/lib/hadoop-hdfs/lib/jersey-core-1.8.jar\
> > :/
> > usr/lib/hadoop-mapreduce/hadoop-extras-2.0.0-cdh4.1.1.jar\:/usr/lib/
> > ha
> > doop-mapreduce/hadoop-distcp-2.0.0-cdh4.1.1.jar\:/usr/lib/hadoop-map
> > re
> > duce/hadoop-mapreduce-client-jobclient-2.0.0-cdh4.1.1-tests.jar\:/us
> > r/
> > lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar\:/usr/lib/hadoop-
> > ma
> > preduce/hadoop-streaming-2.0.0-cdh4.1.1.jar\:/usr/lib/hadoop-mapredu
> > ce
> > /hadoop-mapreduce-client-app.jar\:/usr/lib/hadoop-mapreduce/hadoop-d
> > at
> > ajoin-2.0.0-cdh4.1.1.jar\:/usr/lib/hadoop-mapreduce/hadoop-mapreduce
> > -c
> > lient-shuffle.jar\:/usr/lib/hadoop-mapreduce/hadoop-extras.jar\:/usr
> > /l
> > ib/hadoop-mapreduce/hadoop-mapreduce-client-jobclient-2.0.0-cdh4.1.1
> > .j
> > ar\:/usr/lib/hadoop-mapreduce/hadoop-distcp.jar\:/usr/lib/hadoop-map
> > re
> > duce/hadoop-mapreduce-client-shuffle-2.0.0-cdh4.1.1.jar\:/usr/lib/ha
> > do
> > op-mapreduce/hadoop-rumen.jar\:/usr/lib/hadoop-mapreduce/hadoop-mapr
> > ed
> > uce-examples-2.0.0-cdh4.1.1.jar\:/usr/lib/hadoop-mapreduce/hadoop-ar
> > ch
> > ives.jar\:/usr/lib/hadoop-mapreduce/hadoop-rumen-2.0.0-cdh4.1.1.jar\
> > :/
> > usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-core.jar\:/usr/lib/
> > ha
> > doop-mapreduce/hadoop-gridmix.jar\:/usr/lib/hadoop-mapreduce/hadoop-
> > ma
> > preduce-client-app-2.0.0-cdh4.1.1.jar\:/usr/lib/hadoop-mapreduce/had
> > oo
> > p-mapreduce-client-common.jar\:/usr/lib/hadoop-mapreduce/hadoop-arch
> > iv
> > es-2.0.0-cdh4.1.1.jar\:/usr/lib/hadoop-mapreduce/hadoop-datajoin.jar\:
> > /usr/lib/hadoop-mapreduce/hadoop-streaming.jar\:/usr/lib/hadoop-mapr
> > ed
> > uce/hadoop-mapreduce-client-common-2.0.0-cdh4.1.1.jar\:/usr/lib/hado
> > op
> > -mapreduce/hadoop-gridmix-2.0.0-cdh4.1.1.jar\:/usr/lib/hadoop-mapred
> > uc
> > e/hadoop-mapreduce-client-core-2.0.0-cdh4.1.1.jar\:/usr/lib/hadoop-m
> > ap
> > reduce/hadoop-mapreduce-client-hs-2.0.0-cdh4.1.1.jar\:/usr/lib/hadoo
> > p-
> > mapreduce/hadoop-mapreduce-client-hs.jar\:/usr/lib/hadoop-mapreduce/
> > ha
> > doop-mapreduce-client-jobclient.jar\:/usr/lib/hadoop-mapreduce/lib/l
> > og
> > 4j-1.2.17.jar\:/usr/lib/hadoop-mapreduce/lib/javax.inject-1.jar\:/us
> > r/
> > lib/hadoop-mapreduce/lib/netty-3.2.4.Final.jar\:/usr/lib/hadoop-mapr
> > ed
> > uce/lib/jersey-server-1.8.jar\:/usr/lib/hadoop-mapreduce/lib/jackson
> > -c
> > ore-asl-1.8.8.jar\:/usr/lib/hadoop-mapreduce/lib/paranamer-2.3.jar\:
> > /u
> > sr/lib/hadoop-mapreduce/lib/jersey-guice-1.8.jar\:/usr/lib/hadoop-ma
> > pr
> > educe/lib/protobuf-java-2.4.0a.jar\:/usr/lib/hadoop-mapreduce/lib/gu
> > ic
> > e-servlet-3.0.jar\:/usr/lib/hadoop-mapreduce/lib/aopalliance-1.0.jar\:
> > /usr/lib/hadoop-mapreduce/lib/snappy-java-1.0.4.1.jar\:/usr/lib/hado
> > op
> > -mapreduce/lib/avro-1.7.1.cloudera.2.jar\:/usr/lib/hadoop-mapreduce/
> > li
> > b/guice-3.0.jar\:/usr/lib/hadoop-mapreduce/lib/jackson-mapper-asl-1.8.
> > 8.jar\:/usr/lib/hadoop-mapreduce/lib/commons-io-2.1.jar\:/usr/lib/ha
> > do
> > op-mapreduce/lib/asm-3.2.jar\:/usr/lib/hadoop-mapreduce/lib/jersey-c
> > or
> > e-1.8.jar\:/usr/lib/hadoop-yarn/hadoop-yarn-server-tests-2.0.0-cdh4.1.
> > 1.jar\:/usr/lib/hadoop-yarn/hadoop-yarn-server-nodemanager-2.0.0-cdh4.
> > 1.1.jar\:/usr/lib/hadoop-yarn/hadoop-yarn-server-resourcemanager.jar\:
> > /usr/lib/hadoop-yarn/hadoop-yarn-applications-distributedshell.jar\:
> > /u
> > sr/lib/hadoop-yarn/hadoop-yarn-api.jar\:/usr/lib/hadoop-yarn/hadoop-
> > ya
> > rn-api-2.0.0-cdh4.1.1.jar\:/usr/lib/hadoop-yarn/hadoop-yarn-server-t
> > es
> > ts-2.0.0-cdh4.1.1-tests.jar\:/usr/lib/hadoop-yarn/hadoop-yarn-site-2
> > .0
> > .0-cdh4.1.1.jar\:/usr/lib/hadoop-yarn/hadoop-yarn-server-web-proxy-2
> > .0
> > .0-cdh4.1.1.jar\:/usr/lib/hadoop-yarn/hadoop-yarn-server-nodemanager
> > .j
> > ar\:/usr/lib/hadoop-yarn/hadoop-yarn-server-common-2.0.0-cdh4.1.1.ja
> > r\
> > :/usr/lib/hadoop-yarn/hadoop-yarn-server-web-proxy.jar\:/usr/lib/had
> > oo
> > p-yarn/hadoop-yarn-applications-distributedshell-2.0.0-cdh4.1.1.jar\
> > :/
> > usr/lib/hadoop-yarn/hadoop-yarn-server-resourcemanager-2.0.0-cdh4.1.1.
> > jar\:/usr/lib/hadoop-yarn/hadoop-yarn-common-2.0.0-cdh4.1.1.jar\:/us
> > r/
> > lib/hadoop-yarn/hadoop-yarn-site.jar\:/usr/lib/hadoop-yarn/hadoop-ya
> > rn
> > -server-tests.jar\:/usr/lib/hadoop-yarn/hadoop-yarn-server-common.ja
> > r\
> > :/usr/lib/hadoop-yarn/hadoop-yarn-common.jar\:/usr/lib/hadoop-yarn/l
> > ib
> > /log4j-1.2.17.jar\:/usr/lib/hadoop-yarn/lib/javax.inject-1.jar\:/usr
> > /l
> > ib/hadoop-yarn/lib/netty-3.2.4.Final.jar\:/usr/lib/hadoop-yarn/lib/j
> > er
> > sey-server-1.8.jar\:/usr/lib/hadoop-yarn/lib/jackson-core-asl-1.8.8.
> > ja
> > r\:/usr/lib/hadoop-yarn/lib/paranamer-2.3.jar\:/usr/lib/hadoop-yarn/
> > li
> > b/jersey-guice-1.8.jar\:/usr/lib/hadoop-yarn/lib/protobuf-java-2.4.0a.
> > jar\:/usr/lib/hadoop-yarn/lib/guice-servlet-3.0.jar\:/usr/lib/hadoop
> > -y
> > arn/lib/aopalliance-1.0.jar\:/usr/lib/hadoop-yarn/lib/snappy-java-1.0.
> > 4.1.jar\:/usr/lib/hadoop-yarn/lib/avro-1.7.1.cloudera.2.jar\:/usr/li
> > b/
> > hadoop-yarn/lib/guice-3.0.jar\:/usr/lib/hadoop-yarn/lib/jackson-mapp
> > er
> > -asl-1.8.8.jar\:/usr/lib/hadoop-yarn/lib/commons-io-2.1.jar\:/usr/li
> > b/
> > hadoop-yarn/lib/asm-3.2.jar\:/usr/lib/hadoop-yarn/lib/jersey-core-1.8.
> > jar\:job.jar/\:job.jar/classes/\:job.jar/lib/*\:/var/lib/hadoop-yarn
> > /c
> > ache/yarn/nm-local-dir/usercache/cloudera/appcache/application_13606
> > 41
> > 830618_0011/container_1360641830618_0011_01_000002/zookeeper.jar\:/v
> > ar
> > /lib/hadoop-yarn/cache/yarn/nm-local-dir/usercache/cloudera/appcache
> > /a
> > pplication_1360641830618_0011/container_1360641830618_0011_01_000002
> > /j
> > son-simple-1.1.jar\:/var/lib/hadoop-yarn/cache/yarn/nm-local-dir/use
> > rc
> > ache/cloudera/appcache/application_1360641830618_0011/container_1360
> > 64
> > 1830618_0011_01_000002/java-launcher.jar\:/var/lib/hadoop-yarn/cache
> > /y
> > arn/nm-local-dir/usercache/cloudera/appcache/application_13606418306
> > 18
> > _0011/container_1360641830618_0011_01_000002/cr-1.0-SNAPSHOT.jar\:/v
> > ar
> > /lib/hadoop-yarn/cache/yarn/nm-local-dir/usercache/cloudera/appcache
> > /a
> > pplication_1360641830618_0011/container_1360641830618_0011_01_000002
> > /h
> > base.jar
> > user.name=yarn
> >
> > oozie.action.newId.properties=/var/lib/hadoop-yarn/cache/yarn/nm-loc
> > al
> > -dir/usercache/cloudera/appcache/application_1360641830618_0011/cont
> > ai ner_1360641830618_0011_01_000002/newId.properties
> > java.vm.specification.version=1.0
> > sun.java.command=org.apache.hadoop.mapred.YarnChild 192.168.77.82
> > 37020
> > attempt_1360641830618_0011_m_000000_0 2
> > java.home=/usr/java/jdk1.6.0_31/jre
> > sun.arch.data.model=64
> > user.language=en
> > java.specification.vendor=Sun Microsystems Inc.
> > java.vm.info=mixed mode
> > java.version=1.6.0_31
> > java.ext.dirs=/usr/java/jdk1.6.0_31/jre/lib/ext\:/usr/java/packages/
> > li
> > b/ext
> >
> > sun.boot.class.path=/usr/java/jdk1.6.0_31/jre/lib/resources.jar\:/us
> > r/
> > java/jdk1.6.0_31/jre/lib/rt.jar\:/usr/java/jdk1.6.0_31/jre/lib/sunrs
> > as
> > ign.jar\:/usr/java/jdk1.6.0_31/jre/lib/jsse.jar\:/usr/java/jdk1.6.0_
> > 31
> > /jre/lib/jce.jar\:/usr/java/jdk1.6.0_31/jre/lib/charsets.jar\:/usr/j
> > av
> > a/jdk1.6.0_31/jre/lib/modules/jdk.boot.jar\:/usr/java/jdk1.6.0_31/jr
> > e/
> > classes
> > java.vendor=Sun Microsystems Inc.
> > file.separator=/
> > oozie.launcher.job.id=job_1360641830618_0011
> >
> > oozie.action.stats.properties=/var/lib/hadoop-yarn/cache/yarn/nm-loc
> > al
> > -dir/usercache/cloudera/appcache/application_1360641830618_0011/cont
> > ai ner_1360641830618_0011_01_000002/stats.properties
> > java.vendor.url.bug=http\://java.sun.com/cgi-bin/bugreport.cgi
> > sun.io.unicode.encoding=UnicodeLittle
> > sun.cpu.endian=little
> > sun.cpu.isalist=
> >
> > And here is the stacktrace:
> >
> > 2013-02-12 00:59:15,743 INFO [main]
> > org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled
> > snapshot period at 10 second(s).
> > 2013-02-12 00:59:15,743 INFO [main]
> > org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MRAppMaster
> > metrics system started
> > 2013-02-12 00:59:15,762 INFO [main]
> > org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Adding job
> > token for
> > job_1360641830618_0015 to jobTokenSecretManager
> > 2013-02-12 00:59:15,967 FATAL [main]
> > org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Error starting
> > MRAppMaster
> > java.lang.NoClassDefFoundError:
> > org/apache/hadoop/hbase/client/HTableInterface
> > at java.lang.Class.forName0(Native Method)
> > at java.lang.Class.forName(Class.java:169)
> > at
> >
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.isChainJob(JobImpl
> .java:918)
> > at
> >
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.makeUberDecision(J
> obImpl.java:849)
> > at
> >
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.access$2600(JobImp
> l.java:114)
> > at
> >
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl$InitTransition.tra
> nsition(JobImpl.java:995)
> > at
> >
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl$InitTransition.tra
> nsition(JobImpl.java:938)
> > at
> >
> org.apache.hadoop.yarn.state.StateMachineFactory$MultipleInternalArc.d
> oTransition(StateMachineFactory.java:380)
> > at
> >
> org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMac
> hineFactory.java:298)
> > at
> >
> org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachi
> neFactory.java:43)
> > at
> >
> org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.
> doTransition(StateMachineFactory.java:443)
> > at
> >
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.handle(JobImpl.jav
> a:666)
> > at
> >
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.handle(JobImpl.jav
> a:113)
> > at
> >
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobEventDispatcher.hand
> le(MRAppMaster.java:890)
> > at
> >
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.start(MRAppMaster.java:
> 840)
> > at
> >
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$1.run(MRAppMaster.java:
> 1050)
> > at
> > java.security.AccessController.doPrivileged(Native
> > Method)
> > at javax.security.auth.Subject.doAs(Subject.java:396)
> > at
> >
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformat
> ion.java:1332)
> > at
> >
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(M
> RAppMaster.java:1046)
> > at
> > org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java
> > :1
> > 010) Caused by: java.lang.ClassNotFoundException:
> > org.apache.hadoop.hbase.client.HTableInterface
> > at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> > at
> > java.security.AccessController.doPrivileged(Native
> > Method)
> > at
> > java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> > at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
> > at
> > sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
> > at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
> > ... 21 more
> > 2013-02-12 00:59:15,972 INFO [Thread-1]
> > org.apache.hadoop.mapreduce.v2.app.MRAppMaster: MRAppMaster received
> > a signal. Signaling RMCommunicator and JobHistoryEventHandler.
> >
> >
>
>
