Hi Chris, Are you able to get pass the issue? One thing I noticed recently is that if your cluster is having different hadoop version that the one used to compile the hello world example (default is 2.3.0), it might not work. You can try to compile the example by `mvn clean package -DskipTests -Dhadoop.version=[cluster_hadoop_version]`.
Terence On Tue, Jun 13, 2017 at 8:16 PM, Terence Yim <cht...@gmail.com> wrote: > Hi Chris, > > Are you able to get the stdout from the failure AM container? It shows > exactly what classpath it uses. Also, just to confirm, the jar paths that > you see when you run `hadoop classpath` from the client side are actually > exist, right? > > Terence > > On Tue, Jun 13, 2017 at 9:26 AM, Chris Hebert <chris.hebert-int@ > digitalreasoning.com> wrote: > >> Yes. I verified that that all those environment variables resolve >> correctly >> on all the relevant nodes for all users and also in all the hadoop-env.sh >> files. >> >> Is there anything else I may be missing? >> >> On Mon, Jun 12, 2017 at 4:27 PM, Yuliya Feldman <yul...@dremio.com> >> wrote: >> >> > Do those env vars resolve on the NodeManager nodes correctly? >> > >> > On Mon, Jun 12, 2017 at 2:05 PM, Chris Hebert < >> > chris.hebert-...@digitalreasoning.com> wrote: >> > >> > > Other YARN apps work fine. For example, I just successfully ran the >> stock >> > > MapReduce wordcount example (and of course, MapReduce is a YARN >> > > application). >> > > >> > > I ran HelloWorld in debug mode earlier and found that yarnClasspath >> > > contains the following: >> > > $HADOOP_CONF_DIR >> > > $HADOOP_COMMON_HOME/* >> > > $HADOOP_COMMON_HOME/lib/* >> > > $HADOOP_HDFS_HOME/* >> > > $HADOOP_HDFS_HOME/lib/* >> > > $HADOOP_MAPRED_HOME/* >> > > $HADOOP_MAPRED_HOME/lib/* >> > > $HADOOP_YARN_HOME/* >> > > $HADOOP_YARN_HOME/lib/* >> > > >> > > I don't know whether these environment variables are supposed to be >> > > resolved to path variables already or if that happens later. I also >> don't >> > > know if I'm supposed to explicitly declare these environment variables >> > > somewhere, and if so, I do not know where in the Hadoop configuration >> it >> > is >> > > ideal for me to declare them. I tried declaring them in hadoop_env.sh >> and >> > > core-site.xml, but I'm not sure I did it right, and at least in my >> > initial >> > > efforts declaring these variables did not seem to prevent the error. >> If >> > it >> > > is the correct thing for me to set these variables appropriately >> > somewhere, >> > > then I will continue to try to do so. >> > > >> > > On Mon, Jun 12, 2017 at 3:46 PM, Yuliya Feldman <yul...@dremio.com> >> > wrote: >> > > >> > > > I don't think it is an issue with classpath on the client - since it >> > gets >> > > > to start AppMaster container >> > > > >> > > > Is any other YARN app runs OK? >> > > > >> > > > May be YarnConfiguration.YARN_APPLICATION_CLASSPATH is not >> producing >> > > right >> > > > jars >> > > > >> > > > Look at HelloWorld code for yarnClasspath >> > > > >> > > > >> > > > >> > > > On Mon, Jun 12, 2017 at 1:22 PM, Chris Hebert < >> > > > chris.hebert-...@digitalreasoning.com> wrote: >> > > > >> > > > > I hate to ask this here, but it won't work, so whatever. >> > > > > >> > > > > I followed the HelloWorld section of the Getting Started guide < >> > > > > http://twill.apache.org/GettingStarted.html> on my cluster with >> > Hadoop >> > > > and >> > > > > Zookeeper set up and functioning properly. >> > > > > >> > > > > git clone https://github.com/apache/twill.git >> > > > > cd twill >> > > > > mvn clean install -DskipTests >> > > > > >> > > > > export >> > > > > CP=twill-examples/yarn/target/twill-examples-yarn-0.12.0- >> > > > > SNAPSHOT.jar:`hadoop >> > > > > classpath` >> > > > > java -cp $CP org.apache.twill.example.yarn.HelloWorld >> > > > > my.zookeeper.domain:2181 >> > > > > >> > > > > Yes, `hadoop classpath` echoes all the relevant jar directories. >> > > > > >> > > > > The command runs well for a bit with multiple: >> > > > > [ STARTING] DEBUG o.a.twill.yarn.YarnTwillController - Yarn >> > > application >> > > > > status for HelloWorldRunnable application_1111111111111_0001: >> > ACCEPTED >> > > > > >> > > > > until: >> > > > > [ STARTING] DEBUG o.a.hadoop.service.AbstractService - Service: >> > > > > org.apache.hadoop.yarn.client.api.impl.YarnClientImpl entered >> state >> > > > > STOPPED >> > > > > [ STARTING] DEBUG org.apache.hadoop.ipc.Client - stopping client >> from >> > > > > cache: org.apache.hadoop.ipc.Client@4d465b11 >> > > > > [ STARTING] DEBUG o.a.twill.yarn.YarnTwillController - Yarn >> > > application >> > > > > status for HelloWorldRunnable application_1111111111111_0001: >> FAILED >> > > > > ... >> > > > > java.util.concurrent.ExecutionException: >> java.lang.RuntimeException: >> > > > Yarn >> > > > > application completed with failure HelloWorldRunnable... >> > > > > >> > > > > The ResourceManager reveals: >> > > > > Application application_1111111111111_0001 failed 2 times due to >> AM >> > > > > Container for appattempt_1111111111111_0001_000002 exited with >> > > > exitCode: 1 >> > > > > ... >> > > > > Diagnostics: Exception from container-launch. >> > > > > >> > > > > The corresponding YARN logs for each DataNode reveal: >> > > > > Exception in thread "main" java.lang.NoClassDefFoundError: >> > > > > org/apache/hadoop/conf/Configuration >> > > > > at java.lang.Class.getDeclaredMethods0(Native Method) >> > > > > at java.lang.Class.privateGetDeclaredMethods(Class.java:2701) >> > > > > at java.lang.Class.privateGetMethodRecursive(Class.java:3048) >> > > > > at java.lang.Class.getMethod0(Class.java:3018) >> > > > > at java.lang.Class.getMethod(Class.java:1784) >> > > > > at org.apache.twill.launcher.TwillLauncher.main( >> > TwillLauncher.java:70) >> > > > > Caused by: java.lang.ClassNotFoundException: >> > > > > org.apache.hadoop.conf.Configuration >> > > > > at java.net.URLClassLoader.findClass(URLClassLoader.java:381) >> > > > > at java.lang.ClassLoader.loadClass(ClassLoader.java:424) >> > > > > at java.lang.ClassLoader.loadClass(ClassLoader.java:357) >> > > > > ... 6 more >> > > > > Launch class (org.apache.twill.internal.appmaster. >> > > ApplicationMasterMain) >> > > > > using classloader java.net.URLClassLoader with classpath: [ >> > > > > *A list of several classpaths like >> > > > > "file:/some/path/yarn/local/usercache/my.username/ >> > > appcache/application_ >> > > > > 1111111111111_0001/container_e10_ >> > > > > 1111111111111 >> > > > > _0001_02_000001/application.jar/lib/twill-examples-yarn-0. >> > > > > 12.0-SNAPSHOT.jar" >> > > > > But none of which are paths to any Hadoop jars of the sort that >> are >> > > > > referenced in $CP* >> > > > > ] >> > > > > >> > > > > What am I missing? >> > > > > >> > > > > I've spent an embarrassingly large amount of time on this fiddling >> > with >> > > > > environment variables and Hadoop configuration. (I'm an intern >> > learning >> > > > > this stuff the hard way, so it's not really embarrassing, just >> > > > > substantial.) >> > > > > >> > > > >> > > >> > >> > >
