Sorry for the late reply, I was busy with other things. Actually, I got a plain yarn application working. It will do for now, but I might consider Twill another time.
Thanks for your help! On Mon, Jan 25, 2016 at 10:20 PM, Poorna Chandra <[email protected]> wrote: > Since container 000002 could not be started successfully, you'll not find > the logs in resource manager UI. You'll have to find the logs on the box > where the container was launched. > > If you look at App Master logs, you'll see a line like - > 12:49:33.417 [ApplicationMasterService] INFO > o.a.t.i.a.RunnableProcessLauncher - Launching in container > container_e29_1453498444043_0012_01_000002 at > *hdfs-ix03.se-ix.delta.prod*:45454, [$JAVA_HOME/bin/java > -Djava.io.tmpdir=tmp -Dyarn.container=$YARN_CONTAINER_ID > -Dtwill.runnable=$TWILL_APP_NAME.$TWILL_RUNNABLE_NAME -cp > launcher.jar:$HADOOP_CONF_DIR -Xmx359m > org.apache.twill.launcher.TwillLauncher container.jar > org.apache.twill.internal.container.TwillContainerMain true > 1><LOG_DIR>/stdout 2><LOG_DIR>/stderr] > > The stdout/stderr logs for container 000002 will be on the box where the > container was launched (hdfs-ix03.se-ix.delta.prod in the above case). They > should be in the haddop logs directory, which typically is > /var/log/hadoop-yarn/container/<application-id>/<container-id/ > > Poorna. > > > On Mon, Jan 25, 2016 at 6:15 AM, Kristoffer Sjögren <[email protected]> > wrote: > >> I got a tip on the hadoop mailing list to set >> yarn.nodemanager.delete.debug-delay-sec which prevented yarn from >> deleting the app resources and logs immediately. >> >> However, the 000002 container logs is nowhere to be found even with >> this property set? Are you sure that the container got a chance to >> start? >> >> On Sun, Jan 24, 2016 at 12:55 PM, Kristoffer Sjögren <[email protected]> >> wrote: >> > I'm not sure where I can find those logs? There is no container or >> > application with this id in the yarn UI. And there is no directory >> > with that name on the machine that started the application. >> > >> > On Sat, Jan 23, 2016 at 11:17 PM, Poorna Chandra <[email protected]> wrote: >> >> The logs pasted in your previous post are from the App Master - >> >> container_e29_1453498444043_0012_01_000001. >> >> >> >> The App Master starts up fine now, and launches the application >> container - >> >> container_e29_1453498444043_0012_01_000002. It is the application >> container >> >> that dies on launch. We'll need the logs for the application container >> to >> >> see why is is dying. >> >> >> >> Poorna. >> >> >> >> On Sat, Jan 23, 2016 at 1:52 PM, Kristoffer Sjögren <[email protected]> >> >> wrote: >> >> >> >>> I pasted both stdout and stderr in my previous post. >> >>> Den 23 jan 2016 22:50 skrev "Poorna Chandra" <[email protected]>: >> >>> >> >>> > Hi Kristoffer, >> >>> > >> >>> > Looks like container_e29_1453498444043_0012_01_000002 could not be >> >>> started >> >>> > due to some issue. Can you attach the stdout and stderr logs for >> >>> > container_e29_1453498444043_0012_01_000002? >> >>> > >> >>> > Poorna. >> >>> > >> >>> > >> >>> > On Sat, Jan 23, 2016 at 3:53 AM, Kristoffer Sjögren < >> [email protected]> >> >>> > wrote: >> >>> > >> >>> > > Yes, that almost worked. Now the application starts on Yarn and >> after >> >>> > > a while an exception is thrown and the application exits with code >> 10. >> >>> > > >> >>> > > >> >>> > > Application >> >>> > > >> >>> > > About >> >>> > > Jobs >> >>> > > >> >>> > > Tools >> >>> > > >> >>> > > Log Type: stdout >> >>> > > >> >>> > > Log Upload Time: Sat Jan 23 12:49:41 +0100 2016 >> >>> > > >> >>> > > Log Length: 21097 >> >>> > > >> >>> > > UnJar appMaster.jar to tmp/twill.launcher-1453549768670-0 >> >>> > > Launch class >> >>> (org.apache.twill.internal.appmaster.ApplicationMasterMain) >> >>> > > with classpath: >> >>> > > >> >>> > > >> >>> > >> >>> >> [file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/, >> >>> > > >> >>> > > >> >>> > >> >>> >> file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/classes, >> >>> > > >> >>> > > >> >>> > >> >>> >> file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/resources, >> >>> > > >> >>> > > >> >>> > >> >>> >> file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/commons-cli-1.2.jar, >> >>> > > >> >>> > > >> >>> > >> >>> >> file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/scala-library-2.10.1.jar, >> >>> > > >> >>> > > >> >>> > >> >>> >> file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/logback-classic-1.0.9.jar, >> >>> > > >> >>> > > >> >>> > >> >>> >> file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/commons-math3-3.1.1.jar, >> >>> > > >> >>> > > >> >>> > >> >>> >> file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/logback-core-1.0.9.jar, >> >>> > > >> >>> > > >> >>> > >> >>> >> file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/xmlenc-0.52.jar, >> >>> > > >> >>> > > >> >>> > >> >>> >> file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/jsch-0.1.42.jar, >> >>> > > >> >>> > > >> >>> > >> >>> >> file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/httpclient-4.1.2.jar, >> >>> > > >> >>> > > >> >>> > >> >>> >> file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/commons-configuration-1.6.jar, >> >>> > > >> >>> > > >> >>> > >> >>> >> file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/metrics-core-2.2.0.jar, >> >>> > > >> >>> > > >> >>> > >> >>> >> file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/jetty-6.1.26.jar, >> >>> > > >> >>> > > >> >>> > >> >>> >> file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/hadoop-yarn-api-2.7.1.jar, >> >>> > > >> >>> > > >> >>> > >> >>> >> file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/hadoop-annotations-2.7.1.jar, >> >>> > > >> >>> > > >> >>> > >> >>> >> file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/guice-3.0.jar, >> >>> > > >> >>> > > >> >>> > >> >>> >> file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/commons-net-3.1.jar, >> >>> > > >> >>> > > >> >>> > >> >>> >> file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/jetty-util-6.1.26.jar, >> >>> > > >> >>> > > >> >>> > >> >>> >> file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/kafka_2.10-0.8.0.jar, >> >>> > > >> >>> > > >> >>> > >> >>> >> file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/twill-api-0.6.0-incubating.jar, >> >>> > > >> >>> > > >> >>> > >> >>> >> file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/slf4j-api-1.7.10.jar, >> >>> > > >> >>> > > >> >>> > >> >>> >> file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/paranamer-2.3.jar, >> >>> > > >> >>> > > >> >>> > >> >>> >> file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/protobuf-java-2.5.0.jar, >> >>> > > >> >>> > > >> >>> > >> >>> >> file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/apacheds-kerberos-codec-2.0.0-M15.jar, >> >>> > > >> >>> > > >> >>> > >> >>> >> file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/avro-1.7.4.jar, >> >>> > > >> >>> > > >> >>> > >> >>> >> file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/commons-compress-1.4.1.jar, >> >>> > > >> >>> > > >> >>> > >> >>> >> file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/hadoop-auth-2.7.1.jar, >> >>> > > >> >>> > > >> >>> > >> >>> >> file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/zookeeper-3.4.6.jar, >> >>> > > >> >>> > > >> >>> > >> >>> >> file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/jersey-core-1.9.jar, >> >>> > > >> >>> > > >> >>> > >> >>> >> file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/curator-client-2.7.1.jar, >> >>> > > >> >>> > > >> >>> > >> >>> >> file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/twill-zookeeper-0.6.0-incubating.jar, >> >>> > > >> >>> > > >> >>> > >> >>> >> file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/jersey-client-1.9.jar, >> >>> > > >> >>> > > >> >>> > >> >>> >> file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/gson-2.2.4.jar, >> >>> > > >> >>> > > >> >>> > >> >>> >> file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/hadoop-yarn-common-2.7.1.jar, >> >>> > > >> >>> > > >> >>> > >> >>> >> file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/hadoop-hdfs-2.7.1.jar, >> >>> > > >> >>> > > >> >>> > >> >>> >> file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/api-asn1-api-1.0.0-M20.jar, >> >>> > > >> >>> > > >> >>> > >> >>> >> file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/twill-discovery-core-0.6.0-incubating.jar, >> >>> > > >> >>> > > >> >>> > >> >>> >> file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/commons-collections-3.2.1.jar, >> >>> > > >> >>> > > >> >>> > >> >>> >> file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/netty-3.7.0.Final.jar, >> >>> > > >> >>> > > >> >>> > >> >>> >> file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/slf4j-log4j12-1.7.10.jar, >> >>> > > >> >>> > > >> >>> > >> >>> >> file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/hadoop-common-2.7.1.jar, >> >>> > > >> >>> > > >> >>> > >> >>> >> file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/jackson-mapper-asl-1.9.13.jar, >> >>> > > >> >>> > > >> >>> > >> >>> >> file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/zkclient-0.3.jar, >> >>> > > >> >>> > > >> >>> > >> >>> >> file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/jackson-jaxrs-1.9.13.jar, >> >>> > > >> >>> > > >> >>> > >> >>> >> file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/jackson-xc-1.9.13.jar, >> >>> > > >> >>> > > >> >>> > >> >>> >> file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/jsr305-3.0.0.jar, >> >>> > > >> >>> > > >> >>> > >> >>> >> file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/snappy-java-1.0.4.1.jar, >> >>> > > >> >>> > > >> >>> > >> >>> >> file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/log4j-1.2.17.jar, >> >>> > > >> >>> > > >> >>> > >> >>> >> file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/commons-codec-1.4.jar, >> >>> > > >> >>> > > >> >>> > >> >>> >> file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/asm-all-5.0.2.jar, >> >>> > > >> >>> > > >> >>> > >> >>> >> file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/netty-all-4.0.23.Final.jar, >> >>> > > >> >>> > > >> >>> > >> >>> >> file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/servlet-api-2.5.jar, >> >>> > > >> >>> > > >> >>> > >> >>> >> file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/guava-13.0.jar, >> >>> > > >> >>> > > >> >>> > >> >>> >> file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/jopt-simple-3.2.jar, >> >>> > > >> >>> > > >> >>> > >> >>> >> file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/curator-framework-2.7.1.jar, >> >>> > > >> >>> > > >> >>> > >> >>> >> file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/hadoop-yarn-client-2.7.1.jar, >> >>> > > >> >>> > > >> >>> > >> >>> >> file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/commons-httpclient-3.1.jar, >> >>> > > >> >>> > > >> >>> > >> >>> >> file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/twill-discovery-api-0.6.0-incubating.jar, >> >>> > > >> >>> > > >> >>> > >> >>> >> file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/commons-lang-2.6.jar, >> >>> > > >> >>> > > >> >>> > >> >>> >> file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/httpcore-4.1.2.jar, >> >>> > > >> >>> > > >> >>> > >> >>> >> file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/twill-yarn-0.6.0-incubating.jar, >> >>> > > >> >>> > > >> >>> > >> >>> >> file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/api-util-1.0.0-M20.jar, >> >>> > > >> >>> > > >> >>> > >> >>> >> file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/htrace-core-3.1.0-incubating.jar, >> >>> > > >> >>> > > >> >>> > >> >>> >> file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/twill-common-0.6.0-incubating.jar, >> >>> > > >> >>> > > >> >>> > >> >>> >> file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/commons-io-2.4.jar, >> >>> > > >> >>> > > >> >>> > >> >>> >> file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/jersey-server-1.9.jar, >> >>> > > >> >>> > > >> >>> > >> >>> >> file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/apacheds-i18n-2.0.0-M15.jar, >> >>> > > >> >>> > > >> >>> > >> >>> >> file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/commons-logging-1.1.3.jar, >> >>> > > >> >>> > > >> >>> > >> >>> >> file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/twill-core-0.6.0-incubating.jar, >> >>> > > >> >>> > > >> >>> > >> >>> >> file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/jackson-core-asl-1.9.13.jar, >> >>> > > >> >>> > > >> >>> > >> >>> >> file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/javax.inject-1.jar] >> >>> > > Launching main: public static void >> >>> > > >> >>> > > >> >>> > >> >>> >> org.apache.twill.internal.appmaster.ApplicationMasterMain.main(java.lang.String[]) >> >>> > > throws java.lang.Exception [] >> >>> > > 12:49:29.586 [main] DEBUG o.a.h.s.a.util.KerberosName - Kerberos >> krb5 >> >>> > > configuration not found, setting default realm to empty >> >>> > > 12:49:30.083 [main] DEBUG o.a.h.h.p.d.s.DataTransferSaslUtil - >> >>> > > DataTransferProtocol not using SaslPropertiesResolver, no QOP >> found in >> >>> > > configuration for dfs.data.transfer.protection >> >>> > > 12:49:30.552 [main] INFO o.apache.twill.internal.ServiceMain - >> >>> > > Starting service ApplicationMasterService [NEW]. >> >>> > > 12:49:30.600 [kafka-publisher] WARN >> o.a.t.i.k.c.SimpleKafkaPublisher >> >>> > > - Broker list is empty. No Kafka producer is created. >> >>> > > 12:49:30.704 [TrackerService STARTING] INFO >> >>> > > o.a.t.i.appmaster.TrackerService - Tracker service started at >> >>> > > http://hdfs-ix03.se-ix.delta.prod:51793 >> >>> > > 12:49:30.922 [TwillZKPathService STARTING] INFO >> >>> > > o.a.t.i.ServiceMain$TwillZKPathService - Creating container ZK >> path: >> >>> > > >> >>> >> zookeeper01.se-ix.delta.prod/JarApp/be4bbf01-5e72-4147-b2eb-b84e19214b5b >> >>> > > 12:49:31.102 [kafka-publisher] INFO >> o.a.t.i.k.c.SimpleKafkaPublisher >> >>> > > - Update Kafka producer broker list: >> hdfs-ix03.se-ix.delta.prod:58668 >> >>> > > 12:49:31.288 [ApplicationMasterService] INFO >> >>> > > o.a.t.internal.AbstractTwillService - Create live node >> >>> > > >> >>> > > >> >>> > >> >>> >> zookeeper01.se-ix.delta.prod/JarApp/instances/be4bbf01-5e72-4147-b2eb-b84e19214b5b >> >>> > > 12:49:31.308 [ApplicationMasterService] INFO >> >>> > > o.a.t.i.a.ApplicationMasterService - Start application master with >> >>> > > spec: >> >>> > > >> >>> > >> >>> >> {"name":"JarApp","runnables":{"JarRunnable":{"name":"JarRunnable","runnable":{"classname":"org.apache.twill.ext.BundledJarRunnable","name":"JarRunnable","arguments":{}},"resources":{"cores":1,"memorySize":512,"instances":1,"uplink":-1,"downlink":-1},"files":[{"name":"twill-app-1.0.0-SNAPSHOT.jar","uri":"hdfs://hdpcluster/user/stoffe/JarApp/be4bbf01-5e72-4147-b2eb-b84e19214b5b/twill-app-1.0.0-SNAPSHOT.jar.e09cf92c-56f7-42a8-84ac-53f2665afa1d.jar","lastModified":1453549766870,"size":7090,"archive":false,"pattern":null}]}},"orders":[{"names":["JarRunnable"],"type":"STARTED"}],"placementPolicies":[],"handler":{"classname":"org.apache.twill.internal.LogOnlyEventHandler","configs":{}}} >> >>> > > 12:49:31.318 [main] INFO o.apache.twill.internal.ServiceMain - >> >>> > > Service ApplicationMasterService [RUNNING] started. >> >>> > > 12:49:31.344 [ApplicationMasterService] INFO >> >>> > > o.a.t.i.a.ApplicationMasterService - Request 1 container with >> >>> > > capability <memory:512, vCores:1> for runnable JarRunnable >> >>> > > 12:49:33.368 [ApplicationMasterService] INFO >> >>> > > o.a.t.i.a.ApplicationMasterService - Got container >> >>> > > container_e29_1453498444043_0012_01_000002 >> >>> > > 12:49:33.369 [ApplicationMasterService] INFO >> >>> > > o.a.t.i.a.ApplicationMasterService - Starting runnable JarRunnable >> >>> > > with >> >>> > > >> >>> > >> >>> >> RunnableProcessLauncher{container=org.apache.twill.internal.yarn.Hadoop21YarnContainerInfo@5e82cebd >> >>> > > } >> >>> > > 12:49:33.417 [ApplicationMasterService] INFO >> >>> > > o.a.t.i.a.RunnableProcessLauncher - Launching in container >> >>> > > container_e29_1453498444043_0012_01_000002 at >> >>> > > hdfs-ix03.se-ix.delta.prod:45454, [$JAVA_HOME/bin/java >> >>> > > -Djava.io.tmpdir=tmp -Dyarn.container=$YARN_CONTAINER_ID >> >>> > > -Dtwill.runnable=$TWILL_APP_NAME.$TWILL_RUNNABLE_NAME -cp >> >>> > > launcher.jar:$HADOOP_CONF_DIR -Xmx359m >> >>> > > org.apache.twill.launcher.TwillLauncher container.jar >> >>> > > org.apache.twill.internal.container.TwillContainerMain true >> >>> > > 1><LOG_DIR>/stdout 2><LOG_DIR>/stderr] >> >>> > > 12:49:33.473 [ApplicationMasterService] INFO >> >>> > > o.a.t.i.a.ApplicationMasterService - Runnable JarRunnable fully >> >>> > > provisioned with 1 instances. >> >>> > > 12:49:35.302 [zk-client-EventThread] INFO >> >>> > > o.a.t.i.TwillContainerLauncher - Container LiveNodeData updated: >> >>> > > >> >>> > > >> >>> > >> >>> >> {"data":{"containerId":"container_e29_1453498444043_0012_01_000002","host":"hdfs-ix03.se-ix.delta.prod"}} >> >>> > > 12:49:37.484 [ApplicationMasterService] INFO >> >>> > > o.a.t.i.a.ApplicationMasterService - Container >> >>> > > container_e29_1453498444043_0012_01_000002 completed with >> >>> > > COMPLETE:Exception from container-launch. >> >>> > > Container id: container_e29_1453498444043_0012_01_000002 >> >>> > > Exit code: 10 >> >>> > > Stack trace: ExitCodeException exitCode=10: >> >>> > > at org.apache.hadoop.util.Shell.runCommand(Shell.java:576) >> >>> > > at org.apache.hadoop.util.Shell.run(Shell.java:487) >> >>> > > at >> >>> > > >> >>> >> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:753) >> >>> > > at >> >>> > > >> >>> > >> >>> >> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:212) >> >>> > > at >> >>> > > >> >>> > >> >>> >> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302) >> >>> > > at >> >>> > > >> >>> > >> >>> >> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82) >> >>> > > at java.util.concurrent.FutureTask.run(FutureTask.java:266) >> >>> > > at >> >>> > > >> >>> > >> >>> >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) >> >>> > > at >> >>> > > >> >>> > >> >>> >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) >> >>> > > at java.lang.Thread.run(Thread.java:745) >> >>> > > >> >>> > > >> >>> > > Container exited with a non-zero exit code 10 >> >>> > > . >> >>> > > 12:49:37.488 [ApplicationMasterService] WARN >> >>> > > o.a.t.i.appmaster.RunningContainers - Container >> >>> > > container_e29_1453498444043_0012_01_000002 exited abnormally with >> >>> > > state COMPLETE, exit code 10. >> >>> > > 12:49:37.496 [ApplicationMasterService] INFO >> >>> > > o.a.t.i.a.ApplicationMasterService - All containers completed. >> >>> > > Shutting down application master. >> >>> > > 12:49:37.498 [ApplicationMasterService] INFO >> >>> > > o.a.t.i.a.ApplicationMasterService - Stop application master with >> >>> > > spec: >> >>> > > >> >>> > >> >>> >> {"name":"JarApp","runnables":{"JarRunnable":{"name":"JarRunnable","runnable":{"classname":"org.apache.twill.ext.BundledJarRunnable","name":"JarRunnable","arguments":{}},"resources":{"cores":1,"memorySize":512,"instances":1,"uplink":-1,"downlink":-1},"files":[{"name":"twill-app-1.0.0-SNAPSHOT.jar","uri":"hdfs://hdpcluster/user/stoffe/JarApp/be4bbf01-5e72-4147-b2eb-b84e19214b5b/twill-app-1.0.0-SNAPSHOT.jar.e09cf92c-56f7-42a8-84ac-53f2665afa1d.jar","lastModified":1453549766870,"size":7090,"archive":false,"pattern":null}]}},"orders":[{"names":["JarRunnable"],"type":"STARTED"}],"placementPolicies":[],"handler":{"classname":"org.apache.twill.internal.LogOnlyEventHandler","configs":{}}} >> >>> > > 12:49:37.500 [ApplicationMasterService] INFO >> >>> > > o.a.t.i.appmaster.RunningContainers - Stopping all instances of >> >>> > > JarRunnable >> >>> > > 12:49:37.500 [ApplicationMasterService] INFO >> >>> > > o.a.t.i.appmaster.RunningContainers - Terminated all instances of >> >>> > > JarRunnable >> >>> > > 12:49:37.512 [ApplicationMasterService] INFO >> >>> > > o.a.t.i.a.ApplicationMasterService - Application directory deleted: >> >>> > > >> >>> >> hdfs://hdpcluster/user/stoffe/JarApp/be4bbf01-5e72-4147-b2eb-b84e19214b5b >> >>> > > 12:49:37.512 [ApplicationMasterService] INFO >> >>> > > o.a.t.internal.AbstractTwillService - Remove live node >> >>> > > >> >>> > > >> >>> > >> >>> >> zookeeper01.se-ix.delta.prod/JarApp/instances/be4bbf01-5e72-4147-b2eb-b84e19214b5b >> >>> > > 12:49:37.516 [ApplicationMasterService] INFO >> >>> > > o.a.t.internal.AbstractTwillService - Service >> ApplicationMasterService >> >>> > > with runId be4bbf01-5e72-4147-b2eb-b84e19214b5b shutdown completed >> >>> > > 12:49:37.516 [main] INFO o.apache.twill.internal.ServiceMain - >> >>> > > Service ApplicationMasterService [TERMINATED] completed. >> >>> > > 12:49:39.676 [kafka-publisher] WARN >> o.a.t.i.k.c.SimpleKafkaPublisher >> >>> > > - Broker list is empty. No Kafka producer is created. >> >>> > > 12:49:40.037 [TwillZKPathService STOPPING] INFO >> >>> > > o.a.t.i.ServiceMain$TwillZKPathService - Removing container ZK >> path: >> >>> > > >> >>> >> zookeeper01.se-ix.delta.prod/JarApp/be4bbf01-5e72-4147-b2eb-b84e19214b5b >> >>> > > 12:49:40.248 [TrackerService STOPPING] INFO >> >>> > > o.a.t.i.appmaster.TrackerService - Tracker service stopped at >> >>> > > http://hdfs-ix03.se-ix.delta.prod:51793 >> >>> > > Main class completed. >> >>> > > Launcher completed >> >>> > > Cleanup directory tmp/twill.launcher-1453549768670-0 >> >>> > > >> >>> > > >> >>> > > >> >>> > > SLF4J: Class path contains multiple SLF4J bindings. >> >>> > > SLF4J: Found binding in >> >>> > > >> >>> > > >> >>> > >> >>> >> [jar:file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/logback-classic-1.0.9.jar!/org/slf4j/impl/StaticLoggerBinder.class] >> >>> > > SLF4J: Found binding in >> >>> > > >> >>> > > >> >>> > >> >>> >> [jar:file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class] >> >>> > > SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for >> an >> >>> > > explanation. >> >>> > > SLF4J: Actual binding is of type >> >>> > > [ch.qos.logback.classic.util.ContextSelectorStaticBinder] >> >>> > > 16/01/23 12:49:29 INFO impl.ContainerManagementProtocolProxy: >> >>> > > yarn.client.max-cached-nodemanagers-proxies : 0 >> >>> > > 16/01/23 12:49:30 INFO utils.VerifiableProperties: Verifying >> properties >> >>> > > 16/01/23 12:49:30 INFO utils.VerifiableProperties: Property >> log.dir is >> >>> > > overridden to >> >>> > > >> >>> > >> >>> >> /hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/kafka-logs >> >>> > > 16/01/23 12:49:30 INFO utils.VerifiableProperties: Property >> >>> > > default.replication.factor is overridden to 1 >> >>> > > 16/01/23 12:49:30 INFO utils.VerifiableProperties: Property port is >> >>> > > overridden to 58668 >> >>> > > 16/01/23 12:49:30 INFO utils.VerifiableProperties: Property >> >>> > > socket.request.max.bytes is overridden to 104857600 >> >>> > > 16/01/23 12:49:30 INFO utils.VerifiableProperties: Property >> >>> > > socket.send.buffer.bytes is overridden to 1048576 >> >>> > > 16/01/23 12:49:30 INFO utils.VerifiableProperties: Property >> >>> > > log.flush.interval.ms is overridden to 1000 >> >>> > > 16/01/23 12:49:30 INFO utils.VerifiableProperties: Property >> >>> > > zookeeper.connect is overridden to >> >>> > > >> >>> > > >> >>> > >> >>> >> zookeeper01.se-ix.delta.prod/JarApp/be4bbf01-5e72-4147-b2eb-b84e19214b5b/kafka >> >>> > > 16/01/23 12:49:30 INFO utils.VerifiableProperties: Property >> broker.id >> >>> > > is overridden to 1 >> >>> > > 16/01/23 12:49:30 INFO utils.VerifiableProperties: Property >> >>> > > log.retention.hours is overridden to 24 >> >>> > > 16/01/23 12:49:30 INFO utils.VerifiableProperties: Property >> >>> > > socket.receive.buffer.bytes is overridden to 1048576 >> >>> > > 16/01/23 12:49:30 INFO utils.VerifiableProperties: Property >> >>> > > zookeeper.connection.timeout.ms is overridden to 3000 >> >>> > > 16/01/23 12:49:30 INFO utils.VerifiableProperties: Property >> >>> > > num.partitions is overridden to 1 >> >>> > > 16/01/23 12:49:30 INFO utils.VerifiableProperties: Property >> >>> > > log.flush.interval.messages is overridden to 10000 >> >>> > > 16/01/23 12:49:30 INFO utils.VerifiableProperties: Property >> >>> > > log.segment.bytes is overridden to 536870912 >> >>> > > 16/01/23 12:49:30 INFO client.ConfiguredRMFailoverProxyProvider: >> >>> > > Failing over to rm2 >> >>> > > 16/01/23 12:49:30 INFO server.KafkaServer: [Kafka Server 1], >> Starting >> >>> > > 16/01/23 12:49:30 INFO log.LogManager: [Log Manager on Broker 1] >> Log >> >>> > > directory >> >>> > > >> >>> > >> >>> >> '/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/kafka-logs' >> >>> > > not found, creating it. >> >>> > > 16/01/23 12:49:30 INFO log.LogManager: [Log Manager on Broker 1] >> >>> > > Starting log cleaner every 600000 ms >> >>> > > 16/01/23 12:49:30 INFO log.LogManager: [Log Manager on Broker 1] >> >>> > > Starting log flusher every 3000 ms with the following overrides >> Map() >> >>> > > 16/01/23 12:49:30 INFO network.Acceptor: Awaiting socket >> connections >> >>> > > on 0.0.0.0:58668. >> >>> > > 16/01/23 12:49:30 INFO network.SocketServer: [Socket Server on >> Broker >> >>> > > 1], Started >> >>> > > 16/01/23 12:49:30 INFO server.KafkaZooKeeper: connecting to ZK: >> >>> > > >> >>> > > >> >>> > >> >>> >> zookeeper01.se-ix.delta.prod/JarApp/be4bbf01-5e72-4147-b2eb-b84e19214b5b/kafka >> >>> > > 16/01/23 12:49:30 INFO zkclient.ZkEventThread: Starting ZkClient >> event >> >>> > > thread. >> >>> > > 16/01/23 12:49:31 INFO zkclient.ZkClient: zookeeper state changed >> >>> > > (SyncConnected) >> >>> > > 16/01/23 12:49:31 INFO utils.ZkUtils$: Registered broker 1 at path >> >>> > > /brokers/ids/1 with address hdfs-ix03.se-ix.delta.prod:58668. >> >>> > > 16/01/23 12:49:31 INFO server.KafkaServer: [Kafka Server 1], >> >>> > > Connecting to ZK: >> >>> > > >> >>> > > >> >>> > >> >>> >> zookeeper01.se-ix.delta.prod/JarApp/be4bbf01-5e72-4147-b2eb-b84e19214b5b/kafka >> >>> > > 16/01/23 12:49:31 INFO utils.VerifiableProperties: Verifying >> properties >> >>> > > 16/01/23 12:49:31 INFO utils.VerifiableProperties: Property >> >>> > > metadata.broker.list is overridden to >> hdfs-ix03.se-ix.delta.prod:58668 >> >>> > > 16/01/23 12:49:31 INFO utils.VerifiableProperties: Property >> >>> > > request.required.acks is overridden to 1 >> >>> > > 16/01/23 12:49:31 INFO utils.VerifiableProperties: Property >> >>> > > partitioner.class is overridden to >> >>> > > org.apache.twill.internal.kafka.client.IntegerPartitioner >> >>> > > 16/01/23 12:49:31 INFO utils.VerifiableProperties: Property >> >>> > > compression.codec is overridden to snappy >> >>> > > 16/01/23 12:49:31 INFO utils.VerifiableProperties: Property >> >>> > > key.serializer.class is overridden to >> >>> > > org.apache.twill.internal.kafka.client.IntegerEncoder >> >>> > > 16/01/23 12:49:31 INFO utils.VerifiableProperties: Property >> >>> > > serializer.class is overridden to >> >>> > > org.apache.twill.internal.kafka.client.ByteBufferEncoder >> >>> > > 16/01/23 12:49:31 INFO utils.Mx4jLoader$: Will not load MX4J, >> >>> > > mx4j-tools.jar is not in the classpath >> >>> > > 16/01/23 12:49:31 INFO controller.KafkaController: [Controller 1]: >> >>> > > Controller starting up >> >>> > > 16/01/23 12:49:31 INFO server.ZookeeperLeaderElector: 1 >> successfully >> >>> > > elected as leader >> >>> > > 16/01/23 12:49:31 INFO controller.KafkaController: [Controller 1]: >> >>> > > Broker 1 starting become controller state transition >> >>> > > 16/01/23 12:49:31 INFO controller.KafkaController: [Controller 1]: >> >>> > > Controller 1 incremented epoch to 1 >> >>> > > 16/01/23 12:49:31 INFO client.ClientUtils$: Fetching metadata from >> >>> > > broker id:0,host:hdfs-ix03.se-ix.delta.prod,port:58668 with >> >>> > > correlation id 0 for 1 topic(s) Set(log) >> >>> > > 16/01/23 12:49:31 INFO producer.SyncProducer: Connected to >> >>> > > hdfs-ix03.se-ix.delta.prod:58668 for producing >> >>> > > 16/01/23 12:49:31 INFO controller.RequestSendThread: >> >>> > > [Controller-1-to-broker-1-send-thread], Starting >> >>> > > 16/01/23 12:49:31 INFO controller.KafkaController: [Controller 1]: >> >>> > > Currently active brokers in the cluster: Set(1) >> >>> > > 16/01/23 12:49:31 INFO controller.KafkaController: [Controller 1]: >> >>> > > Currently shutting brokers in the cluster: Set() >> >>> > > 16/01/23 12:49:31 INFO controller.KafkaController: [Controller 1]: >> >>> > > Current list of topics in the cluster: Set() >> >>> > > 16/01/23 12:49:31 INFO controller.ReplicaStateMachine: [Replica >> state >> >>> > > machine on controller 1]: No state transitions triggered since no >> >>> > > partitions are assigned to brokers 1 >> >>> > > 16/01/23 12:49:31 INFO controller.ReplicaStateMachine: [Replica >> state >> >>> > > machine on controller 1]: Invoking state change to OnlineReplica >> for >> >>> > > replicas >> >>> > > 16/01/23 12:49:31 INFO controller.ReplicaStateMachine: [Replica >> state >> >>> > > machine on controller 1]: Started replica state machine with >> initial >> >>> > > state -> Map() >> >>> > > 16/01/23 12:49:31 INFO controller.PartitionStateMachine: [Partition >> >>> > > state machine on Controller 1]: Started partition state machine >> with >> >>> > > initial state -> Map() >> >>> > > 16/01/23 12:49:31 INFO controller.KafkaController: [Controller 1]: >> >>> > > Broker 1 is ready to serve as the new controller with epoch 1 >> >>> > > 16/01/23 12:49:31 INFO controller.KafkaController: [Controller 1]: >> >>> > > Partitions being reassigned: Map() >> >>> > > 16/01/23 12:49:31 INFO controller.KafkaController: [Controller 1]: >> >>> > > Partitions already reassigned: List() >> >>> > > 16/01/23 12:49:31 INFO controller.KafkaController: [Controller 1]: >> >>> > > Resuming reassignment of partitions: Map() >> >>> > > 16/01/23 12:49:31 INFO controller.KafkaController: [Controller 1]: >> >>> > > Partitions undergoing preferred replica election: >> >>> > > 16/01/23 12:49:31 INFO admin.AdminUtils$: Topic creation { >> >>> > > "partitions":{ "0":[ 1 ] }, "version":1 } >> >>> > > 16/01/23 12:49:31 INFO controller.KafkaController: [Controller 1]: >> >>> > > Partitions that completed preferred replica election: >> >>> > > 16/01/23 12:49:31 INFO controller.KafkaController: [Controller 1]: >> >>> > > Resuming preferred replica election for partitions: >> >>> > > 16/01/23 12:49:31 INFO controller.KafkaController: [Controller 1]: >> >>> > > Starting preferred replica leader election for partitions >> >>> > > 16/01/23 12:49:31 INFO controller.PartitionStateMachine: [Partition >> >>> > > state machine on Controller 1]: Invoking state change to >> >>> > > OnlinePartition for partitions >> >>> > > 16/01/23 12:49:31 INFO controller.KafkaController: [Controller 1]: >> >>> > > Controller startup complete >> >>> > > 16/01/23 12:49:31 INFO server.KafkaApis: [KafkaApi-1] Auto >> creation of >> >>> > > topic log with 1 partitions and replication factor 1 is successful! >> >>> > > 16/01/23 12:49:31 INFO server.KafkaServer: [Kafka Server 1], >> Started >> >>> > > 16/01/23 12:49:31 INFO >> >>> > > server.ZookeeperLeaderElector$LeaderChangeListener: New leader is 1 >> >>> > > 16/01/23 12:49:31 INFO controller.ControllerEpochListener: >> >>> > > [ControllerEpochListener on 1]: Initialized controller epoch to 1 >> and >> >>> > > zk version 0 >> >>> > > 16/01/23 12:49:31 INFO producer.SyncProducer: Disconnecting from >> >>> > > hdfs-ix03.se-ix.delta.prod:58668 >> >>> > > 16/01/23 12:49:31 WARN producer.BrokerPartitionInfo: Error while >> >>> > > fetching metadata [{TopicMetadata for topic log -> >> >>> > > No partition metadata for topic log due to >> >>> > > kafka.common.LeaderNotAvailableException}] for topic [log]: class >> >>> > > kafka.common.LeaderNotAvailableException >> >>> > > 16/01/23 12:49:31 INFO client.ClientUtils$: Fetching metadata from >> >>> > > broker id:0,host:hdfs-ix03.se-ix.delta.prod,port:58668 with >> >>> > > correlation id 1 for 1 topic(s) Set(log) >> >>> > > 16/01/23 12:49:31 INFO producer.SyncProducer: Connected to >> >>> > > hdfs-ix03.se-ix.delta.prod:58668 for producing >> >>> > > 16/01/23 12:49:31 INFO >> >>> > > controller.PartitionStateMachine$TopicChangeListener: >> >>> > > [TopicChangeListener on Controller 1]: New topics: [Set(log)], >> deleted >> >>> > > topics: [Set()], new partition replica assignment [Map([log,0] -> >> >>> > > List(1))] >> >>> > > 16/01/23 12:49:31 INFO admin.AdminUtils$: Topic creation { >> >>> > > "partitions":{ "0":[ 1 ] }, "version":1 } >> >>> > > 16/01/23 12:49:31 INFO controller.KafkaController: [Controller 1]: >> New >> >>> > > topic creation callback for [log,0] >> >>> > > 16/01/23 12:49:31 INFO controller.KafkaController: [Controller 1]: >> New >> >>> > > partition creation callback for [log,0] >> >>> > > 16/01/23 12:49:31 INFO producer.SyncProducer: Disconnecting from >> >>> > > hdfs-ix03.se-ix.delta.prod:58668 >> >>> > > 16/01/23 12:49:31 INFO controller.PartitionStateMachine: [Partition >> >>> > > state machine on Controller 1]: Invoking state change to >> NewPartition >> >>> > > for partitions [log,0] >> >>> > > 16/01/23 12:49:31 WARN producer.BrokerPartitionInfo: Error while >> >>> > > fetching metadata [{TopicMetadata for topic log -> >> >>> > > No partition metadata for topic log due to >> >>> > > kafka.common.LeaderNotAvailableException}] for topic [log]: class >> >>> > > kafka.common.LeaderNotAvailableException >> >>> > > 16/01/23 12:49:31 INFO network.Processor: Closing socket >> connection to >> >>> > > /10.3.24.22. >> >>> > > 16/01/23 12:49:31 INFO network.Processor: Closing socket >> connection to >> >>> > > /10.3.24.22. >> >>> > > 16/01/23 12:49:31 ERROR async.DefaultEventHandler: Failed to >> collate >> >>> > > messages by topic, partition due to: Failed to fetch topic metadata >> >>> > > for topic: log >> >>> > > 16/01/23 12:49:31 INFO async.DefaultEventHandler: Back off for 100 >> ms >> >>> > > before retrying send. Remaining retries = 3 >> >>> > > 16/01/23 12:49:31 INFO controller.ReplicaStateMachine: [Replica >> state >> >>> > > machine on controller 1]: Invoking state change to NewReplica for >> >>> > > replicas PartitionAndReplica(log,0,1) >> >>> > > 16/01/23 12:49:31 INFO controller.PartitionStateMachine: [Partition >> >>> > > state machine on Controller 1]: Invoking state change to >> >>> > > OnlinePartition for partitions [log,0] >> >>> > > 16/01/23 12:49:31 INFO controller.ReplicaStateMachine: [Replica >> state >> >>> > > machine on controller 1]: Invoking state change to OnlineReplica >> for >> >>> > > replicas PartitionAndReplica(log,0,1) >> >>> > > 16/01/23 12:49:31 INFO server.ReplicaManager: [Replica Manager on >> >>> > > Broker 1]: Handling LeaderAndIsr request >> >>> > > >> >>> > > >> >>> > >> >>> >> Name:LeaderAndIsrRequest;Version:0;Controller:1;ControllerEpoch:1;CorrelationId:6;ClientId:id_1-host_null-port_58668;PartitionState:(log,0) >> >>> > > -> >> >>> > > >> >>> > >> >>> >> (LeaderAndIsrInfo:(Leader:1,ISR:1,LeaderEpoch:0,ControllerEpoch:1),ReplicationFactor:1),AllReplicas:1);Leaders:id:1,host:hdfs-ix03.se-ix.delta.prod,port:58668 >> >>> > > 16/01/23 12:49:31 INFO server.ReplicaFetcherManager: >> >>> > > [ReplicaFetcherManager on broker 1] Removing fetcher for partition >> >>> > > [log,0] >> >>> > > 16/01/23 12:49:31 INFO log.Log: [Kafka Log on Broker 1], Completed >> >>> > > load of log log-0 with log end offset 0 >> >>> > > 16/01/23 12:49:31 INFO log.LogManager: [Log Manager on Broker 1] >> >>> > > Created log for partition [log,0] in >> >>> > > >> >>> > > >> >>> > >> >>> >> /hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/kafka-logs. >> >>> > > 16/01/23 12:49:31 WARN server.HighwaterMarkCheckpoint: No >> >>> > > highwatermark file is found. Returning 0 as the highwatermark for >> >>> > > partition [log,0] >> >>> > > 16/01/23 12:49:31 INFO server.ReplicaManager: [Replica Manager on >> >>> > > Broker 1]: Handled leader and isr request >> >>> > > >> >>> > > >> >>> > >> >>> >> Name:LeaderAndIsrRequest;Version:0;Controller:1;ControllerEpoch:1;CorrelationId:6;ClientId:id_1-host_null-port_58668;PartitionState:(log,0) >> >>> > > -> >> >>> > > >> >>> > >> >>> >> (LeaderAndIsrInfo:(Leader:1,ISR:1,LeaderEpoch:0,ControllerEpoch:1),ReplicationFactor:1),AllReplicas:1);Leaders:id:1,host:hdfs-ix03.se-ix.delta.prod,port:58668 >> >>> > > 16/01/23 12:49:31 INFO client.ClientUtils$: Fetching metadata from >> >>> > > broker id:0,host:hdfs-ix03.se-ix.delta.prod,port:58668 with >> >>> > > correlation id 2 for 1 topic(s) Set(log) >> >>> > > 16/01/23 12:49:31 INFO producer.SyncProducer: Connected to >> >>> > > hdfs-ix03.se-ix.delta.prod:58668 for producing >> >>> > > 16/01/23 12:49:31 INFO admin.AdminUtils$: Topic creation { >> >>> > > "partitions":{ "0":[ 1 ] }, "version":1 } >> >>> > > 16/01/23 12:49:31 INFO producer.SyncProducer: Disconnecting from >> >>> > > hdfs-ix03.se-ix.delta.prod:58668 >> >>> > > 16/01/23 12:49:31 INFO network.Processor: Closing socket >> connection to >> >>> > > /10.3.24.22. >> >>> > > 16/01/23 12:49:31 WARN producer.BrokerPartitionInfo: Error while >> >>> > > fetching metadata [{TopicMetadata for topic log -> >> >>> > > No partition metadata for topic log due to >> >>> > > kafka.common.LeaderNotAvailableException}] for topic [log]: class >> >>> > > kafka.common.LeaderNotAvailableException >> >>> > > 16/01/23 12:49:31 INFO client.ClientUtils$: Fetching metadata from >> >>> > > broker id:0,host:hdfs-ix03.se-ix.delta.prod,port:58668 with >> >>> > > correlation id 3 for 1 topic(s) Set(log) >> >>> > > 16/01/23 12:49:31 INFO producer.SyncProducer: Connected to >> >>> > > hdfs-ix03.se-ix.delta.prod:58668 for producing >> >>> > > 16/01/23 12:49:31 INFO producer.SyncProducer: Disconnecting from >> >>> > > hdfs-ix03.se-ix.delta.prod:58668 >> >>> > > 16/01/23 12:49:31 INFO network.Processor: Closing socket >> connection to >> >>> > > /10.3.24.22. >> >>> > > 16/01/23 12:49:31 INFO producer.SyncProducer: Connected to >> >>> > > hdfs-ix03.se-ix.delta.prod:58668 for producing >> >>> > > 16/01/23 12:49:33 INFO impl.AMRMClientImpl: Received new token for >> : >> >>> > > hdfs-ix03.se-ix.delta.prod:45454 >> >>> > > 16/01/23 12:49:33 INFO impl.ContainerManagementProtocolProxy: >> Opening >> >>> > > proxy : hdfs-ix03.se-ix.delta.prod:45454 >> >>> > > 16/01/23 12:49:35 INFO network.Processor: Closing socket >> connection to >> >>> > > /10.3.24.22. >> >>> > > 16/01/23 12:49:35 INFO network.Processor: Closing socket >> connection to >> >>> > > /10.3.24.22. >> >>> > > 16/01/23 12:49:39 INFO server.KafkaServer: [Kafka Server 1], >> Shutting >> >>> > down >> >>> > > 16/01/23 12:49:39 INFO server.KafkaZooKeeper: Closing zookeeper >> >>> client... >> >>> > > 16/01/23 12:49:39 INFO zkclient.ZkEventThread: Terminate ZkClient >> event >> >>> > > thread. >> >>> > > 16/01/23 12:49:39 INFO network.SocketServer: [Socket Server on >> Broker >> >>> > > 1], Shutting down >> >>> > > 16/01/23 12:49:39 INFO network.SocketServer: [Socket Server on >> Broker >> >>> > > 1], Shutdown completed >> >>> > > 16/01/23 12:49:39 INFO server.KafkaRequestHandlerPool: [Kafka >> Request >> >>> > > Handler on Broker 1], shutting down >> >>> > > 16/01/23 12:49:39 INFO server.KafkaRequestHandlerPool: [Kafka >> Request >> >>> > > Handler on Broker 1], shutted down completely >> >>> > > 16/01/23 12:49:39 INFO utils.KafkaScheduler: Shutdown Kafka >> scheduler >> >>> > > 16/01/23 12:49:39 INFO server.ReplicaManager: [Replica Manager on >> >>> > > Broker 1]: Shut down >> >>> > > 16/01/23 12:49:39 INFO server.ReplicaFetcherManager: >> >>> > > [ReplicaFetcherManager on broker 1] shutting down >> >>> > > 16/01/23 12:49:39 INFO server.ReplicaFetcherManager: >> >>> > > [ReplicaFetcherManager on broker 1] shutdown completed >> >>> > > 16/01/23 12:49:39 INFO server.ReplicaManager: [Replica Manager on >> >>> > > Broker 1]: Shutted down completely >> >>> > > 16/01/23 12:49:40 INFO controller.RequestSendThread: >> >>> > > [Controller-1-to-broker-1-send-thread], Shutting down >> >>> > > 16/01/23 12:49:40 INFO controller.RequestSendThread: >> >>> > > [Controller-1-to-broker-1-send-thread], Stopped >> >>> > > 16/01/23 12:49:40 INFO controller.RequestSendThread: >> >>> > > [Controller-1-to-broker-1-send-thread], Shutdown completed >> >>> > > 16/01/23 12:49:40 INFO controller.KafkaController: [Controller 1]: >> >>> > > Controller shutdown complete >> >>> > > 16/01/23 12:49:40 INFO server.KafkaServer: [Kafka Server 1], Shut >> down >> >>> > > completed >> >>> > > 16/01/23 12:49:40 INFO impl.ContainerManagementProtocolProxy: >> Opening >> >>> > > proxy : hdfs-ix03.se-ix.delta.prod:45454 >> >>> > > 16/01/23 12:49:40 INFO impl.AMRMClientImpl: Waiting for >> application to >> >>> > > be successfully unregistered. >> >>> > > 16/01/23 12:49:40 INFO producer.SyncProducer: Disconnecting from >> >>> > > hdfs-ix03.se-ix.delta.prod:58668 >> >>> > > 16/01/23 12:49:40 WARN async.DefaultEventHandler: Failed to send >> >>> > > producer request with correlation id 35 to broker 1 with data for >> >>> > > partitions [log,0] >> >>> > > java.nio.channels.ClosedByInterruptException >> >>> > > at >> >>> > > >> >>> > >> >>> >> java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202) >> >>> > > at sun.nio.ch.SocketChannelImpl.poll(SocketChannelImpl.java:957) >> >>> > > at >> >>> > >> sun.nio.ch.SocketAdaptor$SocketInputStream.read(SocketAdaptor.java:204) >> >>> > > at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103) >> >>> > > at >> >>> > > >> >>> > >> >>> >> java.nio.channels.Channels$ReadableByteChannelImpl.read(Channels.java:385) >> >>> > > at kafka.utils.Utils$.read(Unknown Source) >> >>> > > at kafka.network.BoundedByteBufferReceive.readFrom(Unknown Source) >> >>> > > at kafka.network.Receive$class.readCompletely(Unknown Source) >> >>> > > at kafka.network.BoundedByteBufferReceive.readCompletely(Unknown >> >>> Source) >> >>> > > at kafka.network.BlockingChannel.receive(Unknown Source) >> >>> > > at kafka.producer.SyncProducer.liftedTree1$1(Unknown Source) >> >>> > > at >> >>> > >> kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(Unknown >> >>> > > Source) >> >>> > > at >> >>> > > >> >>> > >> >>> >> kafka.producer.SyncProducer$$anonfun$send$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(Unknown >> >>> > > Source) >> >>> > > at >> >>> > > >> >>> > >> >>> >> kafka.producer.SyncProducer$$anonfun$send$1$$anonfun$apply$mcV$sp$1.apply(Unknown >> >>> > > Source) >> >>> > > at >> >>> > > >> >>> > >> >>> >> kafka.producer.SyncProducer$$anonfun$send$1$$anonfun$apply$mcV$sp$1.apply(Unknown >> >>> > > Source) >> >>> > > at kafka.metrics.KafkaTimer.time(Unknown Source) >> >>> > > at kafka.producer.SyncProducer$$anonfun$send$1.apply$mcV$sp(Unknown >> >>> > Source) >> >>> > > at kafka.producer.SyncProducer$$anonfun$send$1.apply(Unknown >> Source) >> >>> > > at kafka.producer.SyncProducer$$anonfun$send$1.apply(Unknown >> Source) >> >>> > > at kafka.metrics.KafkaTimer.time(Unknown Source) >> >>> > > at kafka.producer.SyncProducer.send(Unknown Source) >> >>> > > at >> >>> > > >> >>> > >> >>> >> kafka.producer.async.DefaultEventHandler.kafka$producer$async$DefaultEventHandler$$send(Unknown >> >>> > > Source) >> >>> > > at >> >>> > > >> >>> > >> >>> >> kafka.producer.async.DefaultEventHandler$$anonfun$dispatchSerializedData$2.apply(Unknown >> >>> > > Source) >> >>> > > at >> >>> > > >> >>> > >> >>> >> kafka.producer.async.DefaultEventHandler$$anonfun$dispatchSerializedData$2.apply(Unknown >> >>> > > Source) >> >>> > > at >> >>> > > >> >>> > >> >>> >> scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:772) >> >>> > > at >> >>> > > >> >>> > >> >>> >> scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:98) >> >>> > > at >> >>> > > >> >>> > >> >>> >> scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:98) >> >>> > > at >> >>> > > >> >>> > >> >>> >> scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:226) >> >>> > > at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:39) >> >>> > > at scala.collection.mutable.HashMap.foreach(HashMap.scala:98) >> >>> > > at >> >>> > > >> >>> > >> >>> >> scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:771) >> >>> > > at >> >>> > >> kafka.producer.async.DefaultEventHandler.dispatchSerializedData(Unknown >> >>> > > Source) >> >>> > > at kafka.producer.async.DefaultEventHandler.handle(Unknown Source) >> >>> > > at kafka.producer.Producer.send(Unknown Source) >> >>> > > at kafka.javaapi.producer.Producer.send(Unknown Source) >> >>> > > at >> >>> > > >> >>> > >> >>> >> org.apache.twill.internal.kafka.client.SimpleKafkaPublisher$SimplePreparer.send(SimpleKafkaPublisher.java:122) >> >>> > > at >> >>> > > >> >>> > >> >>> >> org.apache.twill.internal.logging.KafkaAppender.doPublishLogs(KafkaAppender.java:268) >> >>> > > at >> >>> > > >> >>> > >> >>> >> org.apache.twill.internal.logging.KafkaAppender.publishLogs(KafkaAppender.java:228) >> >>> > > at >> >>> > > >> >>> > >> >>> >> org.apache.twill.internal.logging.KafkaAppender.access$700(KafkaAppender.java:66) >> >>> > > at >> >>> > > >> >>> > >> >>> >> org.apache.twill.internal.logging.KafkaAppender$2.run(KafkaAppender.java:280) >> >>> > > at >> >>> > >> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) >> >>> > > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) >> >>> > > at >> >>> > > >> >>> > >> >>> >> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) >> >>> > > at >> >>> > > >> >>> > >> >>> >> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) >> >>> > > at >> >>> > > >> >>> > >> >>> >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) >> >>> > > at >> >>> > > >> >>> > >> >>> >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) >> >>> > > at java.lang.Thread.run(Thread.java:745) >> >>> > > 16/01/23 12:49:40 INFO async.DefaultEventHandler: Back off for 100 >> ms >> >>> > > before retrying send. Remaining retries = 3 >> >>> > > 16/01/23 12:49:40 INFO producer.Producer: Shutting down producer >> >>> > > 16/01/23 12:49:40 INFO producer.ProducerPool: Closing all sync >> >>> producers >> >>> > > >> >>> > > >> >>> > > On Sat, Jan 23, 2016 at 1:22 AM, Terence Yim <[email protected]> >> wrote: >> >>> > > > Hi, >> >>> > > > >> >>> > > > It's due to a very old version of ASM library that bring it by >> >>> > > hadoop/yarn. >> >>> > > > Please add exclusion of asm library to all hadoop dependencies. >> >>> > > > >> >>> > > > <exclusion> >> >>> > > > <groupId>asm</groupId> >> >>> > > > <artifactId>asm</artifactId> >> >>> > > > </exclusion> >> >>> > > > >> >>> > > > Terence >> >>> > > > >> >>> > > > >> >>> > > > On Fri, Jan 22, 2016 at 2:34 PM, Kristoffer Sjögren < >> >>> [email protected]> >> >>> > > > wrote: >> >>> > > > >> >>> > > >> Further adding the following dependencies cause another >> exception. >> >>> > > >> >> >>> > > >> <dependency> >> >>> > > >> <groupId>com.google.guava</groupId> >> >>> > > >> <artifactId>guava</artifactId> >> >>> > > >> <version>13.0</version> >> >>> > > >> </dependency> >> >>> > > >> <dependency> >> >>> > > >> <groupId>org.apache.hadoop</groupId> >> >>> > > >> <artifactId>hadoop-hdfs</artifactId> >> >>> > > >> <version>2.7.1</version> >> >>> > > >> </dependency> >> >>> > > >> >> >>> > > >> Exception in thread " STARTING" >> >>> > > >> java.lang.IncompatibleClassChangeError: class >> >>> > > >> >> org.apache.twill.internal.utils.Dependencies$DependencyClassVisitor >> >>> > > >> has interface org.objectweb.asm.ClassVisitor as super class >> >>> > > >> at java.lang.ClassLoader.defineClass1(Native Method) >> >>> > > >> at java.lang.ClassLoader.defineClass(ClassLoader.java:760) >> >>> > > >> at >> >>> > > >> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) >> >>> > > >> at java.net.URLClassLoader.defineClass(URLClassLoader.java:467) >> >>> > > >> at java.net.URLClassLoader.access$100(URLClassLoader.java:73) >> >>> > > >> at java.net.URLClassLoader$1.run(URLClassLoader.java:368) >> >>> > > >> at java.net.URLClassLoader$1.run(URLClassLoader.java:362) >> >>> > > >> at java.security.AccessController.doPrivileged(Native Method) >> >>> > > >> at java.net.URLClassLoader.findClass(URLClassLoader.java:361) >> >>> > > >> at java.lang.ClassLoader.loadClass(ClassLoader.java:424) >> >>> > > >> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331) >> >>> > > >> at java.lang.ClassLoader.loadClass(ClassLoader.java:357) >> >>> > > >> at >> >>> > > >> >> >>> > > >> >>> > >> >>> >> org.apache.twill.internal.utils.Dependencies.findClassDependencies(Dependencies.java:86) >> >>> > > >> at >> >>> > > >> >> >>> > > >> >>> > >> >>> >> org.apache.twill.internal.ApplicationBundler.findDependencies(ApplicationBundler.java:198) >> >>> > > >> at >> >>> > > >> >> >>> > > >> >>> > >> >>> >> org.apache.twill.internal.ApplicationBundler.createBundle(ApplicationBundler.java:155) >> >>> > > >> at >> >>> > > >> >> >>> > > >> >>> > >> >>> >> org.apache.twill.internal.ApplicationBundler.createBundle(ApplicationBundler.java:126) >> >>> > > >> at >> >>> > > >> >> >>> > > >> >>> > >> >>> >> org.apache.twill.yarn.YarnTwillPreparer.createAppMasterJar(YarnTwillPreparer.java:402) >> >>> > > >> at >> >>> > > >> >> >>> > > >> >>> > >> >>> >> org.apache.twill.yarn.YarnTwillPreparer.access$200(YarnTwillPreparer.java:108) >> >>> > > >> at >> >>> > > >> >> >>> > > >> >>> > >> >>> >> org.apache.twill.yarn.YarnTwillPreparer$1.call(YarnTwillPreparer.java:299) >> >>> > > >> at >> >>> > > >> >> >>> > > >> >>> > >> >>> >> org.apache.twill.yarn.YarnTwillPreparer$1.call(YarnTwillPreparer.java:289) >> >>> > > >> at >> >>> > > >> >> >>> > > >> >>> > >> >>> >> org.apache.twill.yarn.YarnTwillController.doStartUp(YarnTwillController.java:97) >> >>> > > >> at >> >>> > > >> >> >>> > > >> >>> > >> >>> >> org.apache.twill.internal.AbstractZKServiceController.startUp(AbstractZKServiceController.java:76) >> >>> > > >> at >> >>> > > >> >> >>> > > >> >>> > >> >>> >> org.apache.twill.internal.AbstractExecutionServiceController$ServiceDelegate.startUp(AbstractExecutionServiceController.java:175) >> >>> > > >> at >> >>> > > >> >> >>> > > >> >>> > >> >>> >> com.google.common.util.concurrent.AbstractIdleService$1$1.run(AbstractIdleService.java:43) >> >>> > > >> at java.lang.Thread.run(Thread.java:745) >> >>> > > >> >> >>> > > >> On Fri, Jan 22, 2016 at 11:32 PM, Kristoffer Sjögren < >> >>> > [email protected]> >> >>> > > >> wrote: >> >>> > > >> > Add those dependencies fail with the following exception. >> >>> > > >> > >> >>> > > >> > Exception in thread "main" java.lang.AbstractMethodError: >> >>> > > >> > >> >>> > > >> >> >>> > > >> >>> > >> >>> >> org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider.getProxy()Lorg/apache/hadoop/io/retry/FailoverProxyProvider$ProxyInfo; >> >>> > > >> > at >> >>> > > >> >> >>> > > >> >>> > >> >>> >> org.apache.hadoop.io.retry.RetryInvocationHandler.<init>(RetryInvocationHandler.java:73) >> >>> > > >> > at >> >>> > > >> >> >>> > > >> >>> > >> >>> >> org.apache.hadoop.io.retry.RetryInvocationHandler.<init>(RetryInvocationHandler.java:64) >> >>> > > >> > at >> >>> org.apache.hadoop.io.retry.RetryProxy.create(RetryProxy.java:59) >> >>> > > >> > at >> >>> > > >> >> >>> > > >> >>> > >> >>> >> org.apache.hadoop.hdfs.NameNodeProxies.createProxy(NameNodeProxies.java:149) >> >>> > > >> > at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:569) >> >>> > > >> > at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:512) >> >>> > > >> > at >> >>> > > >> >> >>> > > >> >>> > >> >>> >> org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:142) >> >>> > > >> > at >> >>> > > >> org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2653) >> >>> > > >> > at >> org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:92) >> >>> > > >> > at >> >>> > > >> >> >>> > >> org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2687) >> >>> > > >> > at >> org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2669) >> >>> > > >> > at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:371) >> >>> > > >> > at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:170) >> >>> > > >> > at >> >>> > > >> >> >>> > > >> >>> > >> >>> >> org.apache.twill.yarn.YarnTwillRunnerService.createDefaultLocationFactory(YarnTwillRunnerService.java:615) >> >>> > > >> > at >> >>> > > >> >> >>> > > >> >>> > >> >>> >> org.apache.twill.yarn.YarnTwillRunnerService.<init>(YarnTwillRunnerService.java:149) >> >>> > > >> > at deephacks.BundledJarExample.main(BundledJarExample.java:70) >> >>> > > >> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >> >>> > > >> > at >> >>> > > >> >> >>> > > >> >>> > >> >>> >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) >> >>> > > >> > at >> >>> > > >> >> >>> > > >> >>> > >> >>> >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >> >>> > > >> > at java.lang.reflect.Method.invoke(Method.java:497) >> >>> > > >> > at >> >>> > > >> com.intellij.rt.execution.application.AppMain.main(AppMain.java:144) >> >>> > > >> > >> >>> > > >> > On Fri, Jan 22, 2016 at 10:58 PM, Terence Yim < >> [email protected]> >> >>> > > wrote: >> >>> > > >> >> Hi, >> >>> > > >> >> >> >>> > > >> >> If you run it from IDE, you and simply add a dependency on >> hadoop >> >>> > > with >> >>> > > >> >> version 2.7.1. E.g. if you are using Maven, you can add the >> >>> > > following to >> >>> > > >> >> your pom.xml dependencies section. >> >>> > > >> >> >> >>> > > >> >> <dependency> >> >>> > > >> >> <groupId>org.apache.hadoop</groupId> >> >>> > > >> >> <artifactId>hadoop-yarn-api</artifactId> >> >>> > > >> >> <version>2.7.1</version> >> >>> > > >> >> </dependency> >> >>> > > >> >> <dependency> >> >>> > > >> >> <groupId>org.apache.hadoop</groupId> >> >>> > > >> >> <artifactId>hadoop-yarn-common</artifactId> >> >>> > > >> >> <version>2.7.1</version> >> >>> > > >> >> </dependency> >> >>> > > >> >> <dependency> >> >>> > > >> >> <groupId>org.apache.hadoop</groupId> >> >>> > > >> >> <artifactId>hadoop-yarn-client</artifactId> >> >>> > > >> >> <version>2.7.1</version> >> >>> > > >> >> </dependency> >> >>> > > >> >> <dependency> >> >>> > > >> >> <groupId>org.apache.hadoop</groupId> >> >>> > > >> >> <artifactId>hadoop-common</artifactId> >> >>> > > >> >> <version>2.7.1</version> >> >>> > > >> >> </dependency> >> >>> > > >> >> >> >>> > > >> >> Terence >> >>> > > >> >> >> >>> > > >> >> On Fri, Jan 22, 2016 at 12:47 PM, Kristoffer Sjögren < >> >>> > > [email protected]> >> >>> > > >> >> wrote: >> >>> > > >> >> >> >>> > > >> >>> I run it from IDE right now, but would like to create a >> command >> >>> > line >> >>> > > >> >>> app eventually. >> >>> > > >> >>> >> >>> > > >> >>> I should clarify that the exception above is thrown on the >> YARN >> >>> > > node, >> >>> > > >> >>> not in the IDE. >> >>> > > >> >>> >> >>> > > >> >>> On Fri, Jan 22, 2016 at 9:32 PM, Terence Yim < >> [email protected]> >> >>> > > wrote: >> >>> > > >> >>> > Hi Kristoffer, >> >>> > > >> >>> > >> >>> > > >> >>> > The example itself shouldn't need any modification. >> However, >> >>> how >> >>> > > do >> >>> > > >> >>> > you run that class? Do you run it from IDE or from command >> >>> line >> >>> > > using >> >>> > > >> >>> > "java" command? >> >>> > > >> >>> > >> >>> > > >> >>> > Terence >> >>> > > >> >>> > >> >>> > > >> >>> > On Fri, Jan 22, 2016 at 12:16 PM, Kristoffer Sjögren < >> >>> > > >> [email protected]> >> >>> > > >> >>> wrote: >> >>> > > >> >>> >> Hi Terence, >> >>> > > >> >>> >> >> >>> > > >> >>> >> I'm quite new to Twill and not sure how to do that >> exactly. >> >>> > Could >> >>> > > >> you >> >>> > > >> >>> >> show me how to modify the following example to do the >> same? >> >>> > > >> >>> >> >> >>> > > >> >>> >> >> >>> > > >> >>> >> >>> > > >> >> >>> > > >> >>> > >> >>> >> https://github.com/apache/incubator-twill/blob/master/twill-examples/yarn/src/main/java/org/apache/twill/example/yarn/BundledJarExample.java >> >>> > > >> >>> >> >> >>> > > >> >>> >> On Fri, Jan 22, 2016 at 7:24 PM, Terence Yim < >> >>> [email protected] >> >>> > > >> >>> > > >> wrote: >> >>> > > >> >>> >>> Hi Kristoffer, >> >>> > > >> >>> >>> >> >>> > > >> >>> >>> Seems like the exception comes from the YARN class >> >>> > > >> "ConverterUtils". I >> >>> > > >> >>> >>> believe need to start the application with the version >> 2.7.1 >> >>> > > Hadoop >> >>> > > >> >>> >>> Jars. How to do start the twill application? Usually on >> a >> >>> > > cluster >> >>> > > >> with >> >>> > > >> >>> >>> hadoop installed, you can get all the hadoop jars in the >> >>> > > classpath >> >>> > > >> by >> >>> > > >> >>> >>> running this: >> >>> > > >> >>> >>> >> >>> > > >> >>> >>> export CP=`hadoop classpath` >> >>> > > >> >>> >>> java -cp .:$CP YourApp ... >> >>> > > >> >>> >>> >> >>> > > >> >>> >>> Assuming your app classes and Twill jars are in the >> current >> >>> > > >> directory. >> >>> > > >> >>> >>> >> >>> > > >> >>> >>> Terence >> >>> > > >> >>> >>> >> >>> > > >> >>> >>> On Fri, Jan 22, 2016 at 4:54 AM, Kristoffer Sjögren < >> >>> > > >> [email protected]> >> >>> > > >> >>> wrote: >> >>> > > >> >>> >>>> Here's the full stacktrace. >> >>> > > >> >>> >>>> >> >>> > > >> >>> >>>> Exception in thread "main" >> >>> > > >> java.lang.reflect.InvocationTargetException >> >>> > > >> >>> >>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native >> >>> > Method) >> >>> > > >> >>> >>>> at >> >>> > > >> >>> >> >>> > > >> >> >>> > > >> >>> > >> >>> >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) >> >>> > > >> >>> >>>> at >> >>> > > >> >>> >> >>> > > >> >> >>> > > >> >>> > >> >>> >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >> >>> > > >> >>> >>>> at java.lang.reflect.Method.invoke(Method.java:497) >> >>> > > >> >>> >>>> at >> >>> > > >> >> org.apache.twill.launcher.TwillLauncher.main(TwillLauncher.java:89) >> >>> > > >> >>> >>>> Caused by: java.lang.RuntimeException: >> >>> > > >> >>> >>>> java.lang.reflect.InvocationTargetException >> >>> > > >> >>> >>>> at >> >>> > > >> com.google.common.base.Throwables.propagate(Throwables.java:160) >> >>> > > >> >>> >>>> at >> >>> > > >> >>> >> >>> > > >> >> >>> > > >> >>> > >> >>> >> org.apache.twill.internal.yarn.VersionDetectYarnAMClientFactory.create(VersionDetectYarnAMClientFactory.java:61) >> >>> > > >> >>> >>>> at >> >>> > > >> >>> >> >>> > > >> >> >>> > > >> >>> > >> >>> >> org.apache.twill.internal.appmaster.ApplicationMasterMain.main(ApplicationMasterMain.java:77) >> >>> > > >> >>> >>>> ... 5 more >> >>> > > >> >>> >>>> Caused by: java.lang.reflect.InvocationTargetException >> >>> > > >> >>> >>>> at >> >>> > > sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native >> >>> > > >> >>> Method) >> >>> > > >> >>> >>>> at >> >>> > > >> >>> >> >>> > > >> >> >>> > > >> >>> > >> >>> >> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) >> >>> > > >> >>> >>>> at >> >>> > > >> >>> >> >>> > > >> >> >>> > > >> >>> > >> >>> >> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) >> >>> > > >> >>> >>>> at >> >>> > > java.lang.reflect.Constructor.newInstance(Constructor.java:422) >> >>> > > >> >>> >>>> at >> >>> > > >> >>> >> >>> > > >> >> >>> > > >> >>> > >> >>> >> org.apache.twill.internal.yarn.VersionDetectYarnAMClientFactory.create(VersionDetectYarnAMClientFactory.java:58) >> >>> > > >> >>> >>>> ... 6 more >> >>> > > >> >>> >>>> Caused by: java.lang.IllegalArgumentException: Invalid >> >>> > > >> ContainerId: >> >>> > > >> >>> >>>> container_e25_1453466340022_0004_01_000001 >> >>> > > >> >>> >>>> at >> >>> > > >> >>> >> >>> > > >> >> >>> > > >> >>> > >> >>> >> org.apache.hadoop.yarn.util.ConverterUtils.toContainerId(ConverterUtils.java:182) >> >>> > > >> >>> >>>> at >> >>> > > >> >>> >> >>> > > >> >> >>> > > >> >>> > >> >>> >> org.apache.twill.internal.yarn.AbstractYarnAMClient.<init>(AbstractYarnAMClient.java:83) >> >>> > > >> >>> >>>> at >> >>> > > >> >>> >> >>> > > >> >> >>> > > >> >>> > >> >>> >> org.apache.twill.internal.yarn.Hadoop21YarnAMClient.<init>(Hadoop21YarnAMClient.java:65) >> >>> > > >> >>> >>>> at >> >>> > > >> >>> >> >>> > > >> >> >>> > > >> >>> > >> >>> >> org.apache.twill.internal.yarn.Hadoop22YarnAMClient.<init>(Hadoop22YarnAMClient.java:34) >> >>> > > >> >>> >>>> ... 11 more >> >>> > > >> >>> >>>> Caused by: java.lang.NumberFormatException: For input >> >>> string: >> >>> > > >> "e25" >> >>> > > >> >>> >>>> at >> >>> > > >> >>> >> >>> > > >> >> >>> > > >> >>> > >> >>> >> java.lang.NumberFormatException.forInputString(NumberFormatException.java:65) >> >>> > > >> >>> >>>> at java.lang.Long.parseLong(Long.java:589) >> >>> > > >> >>> >>>> at java.lang.Long.parseLong(Long.java:631) >> >>> > > >> >>> >>>> at >> >>> > > >> >>> >> >>> > > >> >> >>> > > >> >>> > >> >>> >> org.apache.hadoop.yarn.util.ConverterUtils.toApplicationAttemptId(ConverterUtils.java:137) >> >>> > > >> >>> >>>> at >> >>> > > >> >>> >> >>> > > >> >> >>> > > >> >>> > >> >>> >> org.apache.hadoop.yarn.util.ConverterUtils.toContainerId(ConverterUtils.java:177) >> >>> > > >> >>> >>>> ... 14 more >> >>> > > >> >>> >>>> >> >>> > > >> >>> >>>> On Thu, Jan 21, 2016 at 10:59 PM, Kristoffer Sjögren < >> >>> > > >> >>> [email protected]> wrote: >> >>> > > >> >>> >>>>> Hi >> >>> > > >> >>> >>>>> >> >>> > > >> >>> >>>>> I'm trying the basic example [1] on yarn 2.7.1 but >> get an >> >>> > > >> exception >> >>> > > >> >>> as >> >>> > > >> >>> >>>>> soon as the application starts on the resource manager >> >>> that >> >>> > > >> tells me >> >>> > > >> >>> >>>>> the container id cannot be parsed. >> >>> > > >> >>> >>>>> >> >>> > > >> >>> >>>>> java.lang.IllegalArgumentException: Invalid >> containerId: >> >>> > > >> >>> >>>>> container_e04_1427159778706_0002_01_000001 >> >>> > > >> >>> >>>>> >> >>> > > >> >>> >>>>> I don't have the exact stacktrace but I recall it >> failing >> >>> in >> >>> > > >> >>> >>>>> ConverterUtils.toContainerId because it assumes that >> that >> >>> > the >> >>> > > >> first >> >>> > > >> >>> >>>>> token is an application attempt to be parsed as an >> >>> integer. >> >>> > > This >> >>> > > >> >>> class >> >>> > > >> >>> >>>>> resides in hadoop-yarn-common 2.3.0. >> >>> > > >> >>> >>>>> >> >>> > > >> >>> >>>>> Is there any way to either tweak the container id or >> make >> >>> > > twill >> >>> > > >> use >> >>> > > >> >>> >>>>> the 2.7.1 jar instead? >> >>> > > >> >>> >>>>> >> >>> > > >> >>> >>>>> Cheers, >> >>> > > >> >>> >>>>> -Kristoffer >> >>> > > >> >>> >>>>> >> >>> > > >> >>> >>>>> >> >>> > > >> >>> >>>>> [1] >> >>> > > >> >>> >> >>> > > >> >> >>> > > >> >>> > >> >>> >> https://github.com/apache/incubator-twill/blob/master/twill-examples/yarn/src/main/java/org/apache/twill/example/yarn/BundledJarExample.java >> >>> > > >> >>> >> >>> > > >> >> >>> > > >> >>> > >> >>> >>
