Since container 000002 could not be started successfully, you'll not find the logs in resource manager UI. You'll have to find the logs on the box where the container was launched.
If you look at App Master logs, you'll see a line like - 12:49:33.417 [ApplicationMasterService] INFO o.a.t.i.a.RunnableProcessLauncher - Launching in container container_e29_1453498444043_0012_01_000002 at *hdfs-ix03.se-ix.delta.prod*:45454, [$JAVA_HOME/bin/java -Djava.io.tmpdir=tmp -Dyarn.container=$YARN_CONTAINER_ID -Dtwill.runnable=$TWILL_APP_NAME.$TWILL_RUNNABLE_NAME -cp launcher.jar:$HADOOP_CONF_DIR -Xmx359m org.apache.twill.launcher.TwillLauncher container.jar org.apache.twill.internal.container.TwillContainerMain true 1><LOG_DIR>/stdout 2><LOG_DIR>/stderr] The stdout/stderr logs for container 000002 will be on the box where the container was launched (hdfs-ix03.se-ix.delta.prod in the above case). They should be in the haddop logs directory, which typically is /var/log/hadoop-yarn/container/<application-id>/<container-id/ Poorna. On Mon, Jan 25, 2016 at 6:15 AM, Kristoffer Sjögren <[email protected]> wrote: > I got a tip on the hadoop mailing list to set > yarn.nodemanager.delete.debug-delay-sec which prevented yarn from > deleting the app resources and logs immediately. > > However, the 000002 container logs is nowhere to be found even with > this property set? Are you sure that the container got a chance to > start? > > On Sun, Jan 24, 2016 at 12:55 PM, Kristoffer Sjögren <[email protected]> > wrote: > > I'm not sure where I can find those logs? There is no container or > > application with this id in the yarn UI. And there is no directory > > with that name on the machine that started the application. > > > > On Sat, Jan 23, 2016 at 11:17 PM, Poorna Chandra <[email protected]> wrote: > >> The logs pasted in your previous post are from the App Master - > >> container_e29_1453498444043_0012_01_000001. > >> > >> The App Master starts up fine now, and launches the application > container - > >> container_e29_1453498444043_0012_01_000002. It is the application > container > >> that dies on launch. We'll need the logs for the application container > to > >> see why is is dying. > >> > >> Poorna. > >> > >> On Sat, Jan 23, 2016 at 1:52 PM, Kristoffer Sjögren <[email protected]> > >> wrote: > >> > >>> I pasted both stdout and stderr in my previous post. > >>> Den 23 jan 2016 22:50 skrev "Poorna Chandra" <[email protected]>: > >>> > >>> > Hi Kristoffer, > >>> > > >>> > Looks like container_e29_1453498444043_0012_01_000002 could not be > >>> started > >>> > due to some issue. Can you attach the stdout and stderr logs for > >>> > container_e29_1453498444043_0012_01_000002? > >>> > > >>> > Poorna. > >>> > > >>> > > >>> > On Sat, Jan 23, 2016 at 3:53 AM, Kristoffer Sjögren < > [email protected]> > >>> > wrote: > >>> > > >>> > > Yes, that almost worked. Now the application starts on Yarn and > after > >>> > > a while an exception is thrown and the application exits with code > 10. > >>> > > > >>> > > > >>> > > Application > >>> > > > >>> > > About > >>> > > Jobs > >>> > > > >>> > > Tools > >>> > > > >>> > > Log Type: stdout > >>> > > > >>> > > Log Upload Time: Sat Jan 23 12:49:41 +0100 2016 > >>> > > > >>> > > Log Length: 21097 > >>> > > > >>> > > UnJar appMaster.jar to tmp/twill.launcher-1453549768670-0 > >>> > > Launch class > >>> (org.apache.twill.internal.appmaster.ApplicationMasterMain) > >>> > > with classpath: > >>> > > > >>> > > > >>> > > >>> > [file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/, > >>> > > > >>> > > > >>> > > >>> > file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/classes, > >>> > > > >>> > > > >>> > > >>> > file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/resources, > >>> > > > >>> > > > >>> > > >>> > file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/commons-cli-1.2.jar, > >>> > > > >>> > > > >>> > > >>> > file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/scala-library-2.10.1.jar, > >>> > > > >>> > > > >>> > > >>> > file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/logback-classic-1.0.9.jar, > >>> > > > >>> > > > >>> > > >>> > file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/commons-math3-3.1.1.jar, > >>> > > > >>> > > > >>> > > >>> > file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/logback-core-1.0.9.jar, > >>> > > > >>> > > > >>> > > >>> > file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/xmlenc-0.52.jar, > >>> > > > >>> > > > >>> > > >>> > file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/jsch-0.1.42.jar, > >>> > > > >>> > > > >>> > > >>> > file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/httpclient-4.1.2.jar, > >>> > > > >>> > > > >>> > > >>> > file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/commons-configuration-1.6.jar, > >>> > > > >>> > > > >>> > > >>> > file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/metrics-core-2.2.0.jar, > >>> > > > >>> > > > >>> > > >>> > file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/jetty-6.1.26.jar, > >>> > > > >>> > > > >>> > > >>> > file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/hadoop-yarn-api-2.7.1.jar, > >>> > > > >>> > > > >>> > > >>> > file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/hadoop-annotations-2.7.1.jar, > >>> > > > >>> > > > >>> > > >>> > file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/guice-3.0.jar, > >>> > > > >>> > > > >>> > > >>> > file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/commons-net-3.1.jar, > >>> > > > >>> > > > >>> > > >>> > file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/jetty-util-6.1.26.jar, > >>> > > > >>> > > > >>> > > >>> > file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/kafka_2.10-0.8.0.jar, > >>> > > > >>> > > > >>> > > >>> > file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/twill-api-0.6.0-incubating.jar, > >>> > > > >>> > > > >>> > > >>> > file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/slf4j-api-1.7.10.jar, > >>> > > > >>> > > > >>> > > >>> > file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/paranamer-2.3.jar, > >>> > > > >>> > > > >>> > > >>> > file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/protobuf-java-2.5.0.jar, > >>> > > > >>> > > > >>> > > >>> > file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/apacheds-kerberos-codec-2.0.0-M15.jar, > >>> > > > >>> > > > >>> > > >>> > file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/avro-1.7.4.jar, > >>> > > > >>> > > > >>> > > >>> > file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/commons-compress-1.4.1.jar, > >>> > > > >>> > > > >>> > > >>> > file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/hadoop-auth-2.7.1.jar, > >>> > > > >>> > > > >>> > > >>> > file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/zookeeper-3.4.6.jar, > >>> > > > >>> > > > >>> > > >>> > file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/jersey-core-1.9.jar, > >>> > > > >>> > > > >>> > > >>> > file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/curator-client-2.7.1.jar, > >>> > > > >>> > > > >>> > > >>> > file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/twill-zookeeper-0.6.0-incubating.jar, > >>> > > > >>> > > > >>> > > >>> > file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/jersey-client-1.9.jar, > >>> > > > >>> > > > >>> > > >>> > file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/gson-2.2.4.jar, > >>> > > > >>> > > > >>> > > >>> > file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/hadoop-yarn-common-2.7.1.jar, > >>> > > > >>> > > > >>> > > >>> > file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/hadoop-hdfs-2.7.1.jar, > >>> > > > >>> > > > >>> > > >>> > file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/api-asn1-api-1.0.0-M20.jar, > >>> > > > >>> > > > >>> > > >>> > file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/twill-discovery-core-0.6.0-incubating.jar, > >>> > > > >>> > > > >>> > > >>> > file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/commons-collections-3.2.1.jar, > >>> > > > >>> > > > >>> > > >>> > file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/netty-3.7.0.Final.jar, > >>> > > > >>> > > > >>> > > >>> > file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/slf4j-log4j12-1.7.10.jar, > >>> > > > >>> > > > >>> > > >>> > file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/hadoop-common-2.7.1.jar, > >>> > > > >>> > > > >>> > > >>> > file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/jackson-mapper-asl-1.9.13.jar, > >>> > > > >>> > > > >>> > > >>> > file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/zkclient-0.3.jar, > >>> > > > >>> > > > >>> > > >>> > file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/jackson-jaxrs-1.9.13.jar, > >>> > > > >>> > > > >>> > > >>> > file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/jackson-xc-1.9.13.jar, > >>> > > > >>> > > > >>> > > >>> > file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/jsr305-3.0.0.jar, > >>> > > > >>> > > > >>> > > >>> > file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/snappy-java-1.0.4.1.jar, > >>> > > > >>> > > > >>> > > >>> > file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/log4j-1.2.17.jar, > >>> > > > >>> > > > >>> > > >>> > file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/commons-codec-1.4.jar, > >>> > > > >>> > > > >>> > > >>> > file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/asm-all-5.0.2.jar, > >>> > > > >>> > > > >>> > > >>> > file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/netty-all-4.0.23.Final.jar, > >>> > > > >>> > > > >>> > > >>> > file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/servlet-api-2.5.jar, > >>> > > > >>> > > > >>> > > >>> > file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/guava-13.0.jar, > >>> > > > >>> > > > >>> > > >>> > file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/jopt-simple-3.2.jar, > >>> > > > >>> > > > >>> > > >>> > file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/curator-framework-2.7.1.jar, > >>> > > > >>> > > > >>> > > >>> > file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/hadoop-yarn-client-2.7.1.jar, > >>> > > > >>> > > > >>> > > >>> > file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/commons-httpclient-3.1.jar, > >>> > > > >>> > > > >>> > > >>> > file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/twill-discovery-api-0.6.0-incubating.jar, > >>> > > > >>> > > > >>> > > >>> > file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/commons-lang-2.6.jar, > >>> > > > >>> > > > >>> > > >>> > file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/httpcore-4.1.2.jar, > >>> > > > >>> > > > >>> > > >>> > file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/twill-yarn-0.6.0-incubating.jar, > >>> > > > >>> > > > >>> > > >>> > file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/api-util-1.0.0-M20.jar, > >>> > > > >>> > > > >>> > > >>> > file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/htrace-core-3.1.0-incubating.jar, > >>> > > > >>> > > > >>> > > >>> > file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/twill-common-0.6.0-incubating.jar, > >>> > > > >>> > > > >>> > > >>> > file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/commons-io-2.4.jar, > >>> > > > >>> > > > >>> > > >>> > file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/jersey-server-1.9.jar, > >>> > > > >>> > > > >>> > > >>> > file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/apacheds-i18n-2.0.0-M15.jar, > >>> > > > >>> > > > >>> > > >>> > file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/commons-logging-1.1.3.jar, > >>> > > > >>> > > > >>> > > >>> > file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/twill-core-0.6.0-incubating.jar, > >>> > > > >>> > > > >>> > > >>> > file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/jackson-core-asl-1.9.13.jar, > >>> > > > >>> > > > >>> > > >>> > file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/javax.inject-1.jar] > >>> > > Launching main: public static void > >>> > > > >>> > > > >>> > > >>> > org.apache.twill.internal.appmaster.ApplicationMasterMain.main(java.lang.String[]) > >>> > > throws java.lang.Exception [] > >>> > > 12:49:29.586 [main] DEBUG o.a.h.s.a.util.KerberosName - Kerberos > krb5 > >>> > > configuration not found, setting default realm to empty > >>> > > 12:49:30.083 [main] DEBUG o.a.h.h.p.d.s.DataTransferSaslUtil - > >>> > > DataTransferProtocol not using SaslPropertiesResolver, no QOP > found in > >>> > > configuration for dfs.data.transfer.protection > >>> > > 12:49:30.552 [main] INFO o.apache.twill.internal.ServiceMain - > >>> > > Starting service ApplicationMasterService [NEW]. > >>> > > 12:49:30.600 [kafka-publisher] WARN > o.a.t.i.k.c.SimpleKafkaPublisher > >>> > > - Broker list is empty. No Kafka producer is created. > >>> > > 12:49:30.704 [TrackerService STARTING] INFO > >>> > > o.a.t.i.appmaster.TrackerService - Tracker service started at > >>> > > http://hdfs-ix03.se-ix.delta.prod:51793 > >>> > > 12:49:30.922 [TwillZKPathService STARTING] INFO > >>> > > o.a.t.i.ServiceMain$TwillZKPathService - Creating container ZK > path: > >>> > > > >>> > zookeeper01.se-ix.delta.prod/JarApp/be4bbf01-5e72-4147-b2eb-b84e19214b5b > >>> > > 12:49:31.102 [kafka-publisher] INFO > o.a.t.i.k.c.SimpleKafkaPublisher > >>> > > - Update Kafka producer broker list: > hdfs-ix03.se-ix.delta.prod:58668 > >>> > > 12:49:31.288 [ApplicationMasterService] INFO > >>> > > o.a.t.internal.AbstractTwillService - Create live node > >>> > > > >>> > > > >>> > > >>> > zookeeper01.se-ix.delta.prod/JarApp/instances/be4bbf01-5e72-4147-b2eb-b84e19214b5b > >>> > > 12:49:31.308 [ApplicationMasterService] INFO > >>> > > o.a.t.i.a.ApplicationMasterService - Start application master with > >>> > > spec: > >>> > > > >>> > > >>> > {"name":"JarApp","runnables":{"JarRunnable":{"name":"JarRunnable","runnable":{"classname":"org.apache.twill.ext.BundledJarRunnable","name":"JarRunnable","arguments":{}},"resources":{"cores":1,"memorySize":512,"instances":1,"uplink":-1,"downlink":-1},"files":[{"name":"twill-app-1.0.0-SNAPSHOT.jar","uri":"hdfs://hdpcluster/user/stoffe/JarApp/be4bbf01-5e72-4147-b2eb-b84e19214b5b/twill-app-1.0.0-SNAPSHOT.jar.e09cf92c-56f7-42a8-84ac-53f2665afa1d.jar","lastModified":1453549766870,"size":7090,"archive":false,"pattern":null}]}},"orders":[{"names":["JarRunnable"],"type":"STARTED"}],"placementPolicies":[],"handler":{"classname":"org.apache.twill.internal.LogOnlyEventHandler","configs":{}}} > >>> > > 12:49:31.318 [main] INFO o.apache.twill.internal.ServiceMain - > >>> > > Service ApplicationMasterService [RUNNING] started. > >>> > > 12:49:31.344 [ApplicationMasterService] INFO > >>> > > o.a.t.i.a.ApplicationMasterService - Request 1 container with > >>> > > capability <memory:512, vCores:1> for runnable JarRunnable > >>> > > 12:49:33.368 [ApplicationMasterService] INFO > >>> > > o.a.t.i.a.ApplicationMasterService - Got container > >>> > > container_e29_1453498444043_0012_01_000002 > >>> > > 12:49:33.369 [ApplicationMasterService] INFO > >>> > > o.a.t.i.a.ApplicationMasterService - Starting runnable JarRunnable > >>> > > with > >>> > > > >>> > > >>> > RunnableProcessLauncher{container=org.apache.twill.internal.yarn.Hadoop21YarnContainerInfo@5e82cebd > >>> > > } > >>> > > 12:49:33.417 [ApplicationMasterService] INFO > >>> > > o.a.t.i.a.RunnableProcessLauncher - Launching in container > >>> > > container_e29_1453498444043_0012_01_000002 at > >>> > > hdfs-ix03.se-ix.delta.prod:45454, [$JAVA_HOME/bin/java > >>> > > -Djava.io.tmpdir=tmp -Dyarn.container=$YARN_CONTAINER_ID > >>> > > -Dtwill.runnable=$TWILL_APP_NAME.$TWILL_RUNNABLE_NAME -cp > >>> > > launcher.jar:$HADOOP_CONF_DIR -Xmx359m > >>> > > org.apache.twill.launcher.TwillLauncher container.jar > >>> > > org.apache.twill.internal.container.TwillContainerMain true > >>> > > 1><LOG_DIR>/stdout 2><LOG_DIR>/stderr] > >>> > > 12:49:33.473 [ApplicationMasterService] INFO > >>> > > o.a.t.i.a.ApplicationMasterService - Runnable JarRunnable fully > >>> > > provisioned with 1 instances. > >>> > > 12:49:35.302 [zk-client-EventThread] INFO > >>> > > o.a.t.i.TwillContainerLauncher - Container LiveNodeData updated: > >>> > > > >>> > > > >>> > > >>> > {"data":{"containerId":"container_e29_1453498444043_0012_01_000002","host":"hdfs-ix03.se-ix.delta.prod"}} > >>> > > 12:49:37.484 [ApplicationMasterService] INFO > >>> > > o.a.t.i.a.ApplicationMasterService - Container > >>> > > container_e29_1453498444043_0012_01_000002 completed with > >>> > > COMPLETE:Exception from container-launch. > >>> > > Container id: container_e29_1453498444043_0012_01_000002 > >>> > > Exit code: 10 > >>> > > Stack trace: ExitCodeException exitCode=10: > >>> > > at org.apache.hadoop.util.Shell.runCommand(Shell.java:576) > >>> > > at org.apache.hadoop.util.Shell.run(Shell.java:487) > >>> > > at > >>> > > > >>> > org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:753) > >>> > > at > >>> > > > >>> > > >>> > org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:212) > >>> > > at > >>> > > > >>> > > >>> > org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302) > >>> > > at > >>> > > > >>> > > >>> > org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82) > >>> > > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > >>> > > at > >>> > > > >>> > > >>> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > >>> > > at > >>> > > > >>> > > >>> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > >>> > > at java.lang.Thread.run(Thread.java:745) > >>> > > > >>> > > > >>> > > Container exited with a non-zero exit code 10 > >>> > > . > >>> > > 12:49:37.488 [ApplicationMasterService] WARN > >>> > > o.a.t.i.appmaster.RunningContainers - Container > >>> > > container_e29_1453498444043_0012_01_000002 exited abnormally with > >>> > > state COMPLETE, exit code 10. > >>> > > 12:49:37.496 [ApplicationMasterService] INFO > >>> > > o.a.t.i.a.ApplicationMasterService - All containers completed. > >>> > > Shutting down application master. > >>> > > 12:49:37.498 [ApplicationMasterService] INFO > >>> > > o.a.t.i.a.ApplicationMasterService - Stop application master with > >>> > > spec: > >>> > > > >>> > > >>> > {"name":"JarApp","runnables":{"JarRunnable":{"name":"JarRunnable","runnable":{"classname":"org.apache.twill.ext.BundledJarRunnable","name":"JarRunnable","arguments":{}},"resources":{"cores":1,"memorySize":512,"instances":1,"uplink":-1,"downlink":-1},"files":[{"name":"twill-app-1.0.0-SNAPSHOT.jar","uri":"hdfs://hdpcluster/user/stoffe/JarApp/be4bbf01-5e72-4147-b2eb-b84e19214b5b/twill-app-1.0.0-SNAPSHOT.jar.e09cf92c-56f7-42a8-84ac-53f2665afa1d.jar","lastModified":1453549766870,"size":7090,"archive":false,"pattern":null}]}},"orders":[{"names":["JarRunnable"],"type":"STARTED"}],"placementPolicies":[],"handler":{"classname":"org.apache.twill.internal.LogOnlyEventHandler","configs":{}}} > >>> > > 12:49:37.500 [ApplicationMasterService] INFO > >>> > > o.a.t.i.appmaster.RunningContainers - Stopping all instances of > >>> > > JarRunnable > >>> > > 12:49:37.500 [ApplicationMasterService] INFO > >>> > > o.a.t.i.appmaster.RunningContainers - Terminated all instances of > >>> > > JarRunnable > >>> > > 12:49:37.512 [ApplicationMasterService] INFO > >>> > > o.a.t.i.a.ApplicationMasterService - Application directory deleted: > >>> > > > >>> > hdfs://hdpcluster/user/stoffe/JarApp/be4bbf01-5e72-4147-b2eb-b84e19214b5b > >>> > > 12:49:37.512 [ApplicationMasterService] INFO > >>> > > o.a.t.internal.AbstractTwillService - Remove live node > >>> > > > >>> > > > >>> > > >>> > zookeeper01.se-ix.delta.prod/JarApp/instances/be4bbf01-5e72-4147-b2eb-b84e19214b5b > >>> > > 12:49:37.516 [ApplicationMasterService] INFO > >>> > > o.a.t.internal.AbstractTwillService - Service > ApplicationMasterService > >>> > > with runId be4bbf01-5e72-4147-b2eb-b84e19214b5b shutdown completed > >>> > > 12:49:37.516 [main] INFO o.apache.twill.internal.ServiceMain - > >>> > > Service ApplicationMasterService [TERMINATED] completed. > >>> > > 12:49:39.676 [kafka-publisher] WARN > o.a.t.i.k.c.SimpleKafkaPublisher > >>> > > - Broker list is empty. No Kafka producer is created. > >>> > > 12:49:40.037 [TwillZKPathService STOPPING] INFO > >>> > > o.a.t.i.ServiceMain$TwillZKPathService - Removing container ZK > path: > >>> > > > >>> > zookeeper01.se-ix.delta.prod/JarApp/be4bbf01-5e72-4147-b2eb-b84e19214b5b > >>> > > 12:49:40.248 [TrackerService STOPPING] INFO > >>> > > o.a.t.i.appmaster.TrackerService - Tracker service stopped at > >>> > > http://hdfs-ix03.se-ix.delta.prod:51793 > >>> > > Main class completed. > >>> > > Launcher completed > >>> > > Cleanup directory tmp/twill.launcher-1453549768670-0 > >>> > > > >>> > > > >>> > > > >>> > > SLF4J: Class path contains multiple SLF4J bindings. > >>> > > SLF4J: Found binding in > >>> > > > >>> > > > >>> > > >>> > [jar:file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/logback-classic-1.0.9.jar!/org/slf4j/impl/StaticLoggerBinder.class] > >>> > > SLF4J: Found binding in > >>> > > > >>> > > > >>> > > >>> > [jar:file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class] > >>> > > SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for > an > >>> > > explanation. > >>> > > SLF4J: Actual binding is of type > >>> > > [ch.qos.logback.classic.util.ContextSelectorStaticBinder] > >>> > > 16/01/23 12:49:29 INFO impl.ContainerManagementProtocolProxy: > >>> > > yarn.client.max-cached-nodemanagers-proxies : 0 > >>> > > 16/01/23 12:49:30 INFO utils.VerifiableProperties: Verifying > properties > >>> > > 16/01/23 12:49:30 INFO utils.VerifiableProperties: Property > log.dir is > >>> > > overridden to > >>> > > > >>> > > >>> > /hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/kafka-logs > >>> > > 16/01/23 12:49:30 INFO utils.VerifiableProperties: Property > >>> > > default.replication.factor is overridden to 1 > >>> > > 16/01/23 12:49:30 INFO utils.VerifiableProperties: Property port is > >>> > > overridden to 58668 > >>> > > 16/01/23 12:49:30 INFO utils.VerifiableProperties: Property > >>> > > socket.request.max.bytes is overridden to 104857600 > >>> > > 16/01/23 12:49:30 INFO utils.VerifiableProperties: Property > >>> > > socket.send.buffer.bytes is overridden to 1048576 > >>> > > 16/01/23 12:49:30 INFO utils.VerifiableProperties: Property > >>> > > log.flush.interval.ms is overridden to 1000 > >>> > > 16/01/23 12:49:30 INFO utils.VerifiableProperties: Property > >>> > > zookeeper.connect is overridden to > >>> > > > >>> > > > >>> > > >>> > zookeeper01.se-ix.delta.prod/JarApp/be4bbf01-5e72-4147-b2eb-b84e19214b5b/kafka > >>> > > 16/01/23 12:49:30 INFO utils.VerifiableProperties: Property > broker.id > >>> > > is overridden to 1 > >>> > > 16/01/23 12:49:30 INFO utils.VerifiableProperties: Property > >>> > > log.retention.hours is overridden to 24 > >>> > > 16/01/23 12:49:30 INFO utils.VerifiableProperties: Property > >>> > > socket.receive.buffer.bytes is overridden to 1048576 > >>> > > 16/01/23 12:49:30 INFO utils.VerifiableProperties: Property > >>> > > zookeeper.connection.timeout.ms is overridden to 3000 > >>> > > 16/01/23 12:49:30 INFO utils.VerifiableProperties: Property > >>> > > num.partitions is overridden to 1 > >>> > > 16/01/23 12:49:30 INFO utils.VerifiableProperties: Property > >>> > > log.flush.interval.messages is overridden to 10000 > >>> > > 16/01/23 12:49:30 INFO utils.VerifiableProperties: Property > >>> > > log.segment.bytes is overridden to 536870912 > >>> > > 16/01/23 12:49:30 INFO client.ConfiguredRMFailoverProxyProvider: > >>> > > Failing over to rm2 > >>> > > 16/01/23 12:49:30 INFO server.KafkaServer: [Kafka Server 1], > Starting > >>> > > 16/01/23 12:49:30 INFO log.LogManager: [Log Manager on Broker 1] > Log > >>> > > directory > >>> > > > >>> > > >>> > '/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/kafka-logs' > >>> > > not found, creating it. > >>> > > 16/01/23 12:49:30 INFO log.LogManager: [Log Manager on Broker 1] > >>> > > Starting log cleaner every 600000 ms > >>> > > 16/01/23 12:49:30 INFO log.LogManager: [Log Manager on Broker 1] > >>> > > Starting log flusher every 3000 ms with the following overrides > Map() > >>> > > 16/01/23 12:49:30 INFO network.Acceptor: Awaiting socket > connections > >>> > > on 0.0.0.0:58668. > >>> > > 16/01/23 12:49:30 INFO network.SocketServer: [Socket Server on > Broker > >>> > > 1], Started > >>> > > 16/01/23 12:49:30 INFO server.KafkaZooKeeper: connecting to ZK: > >>> > > > >>> > > > >>> > > >>> > zookeeper01.se-ix.delta.prod/JarApp/be4bbf01-5e72-4147-b2eb-b84e19214b5b/kafka > >>> > > 16/01/23 12:49:30 INFO zkclient.ZkEventThread: Starting ZkClient > event > >>> > > thread. > >>> > > 16/01/23 12:49:31 INFO zkclient.ZkClient: zookeeper state changed > >>> > > (SyncConnected) > >>> > > 16/01/23 12:49:31 INFO utils.ZkUtils$: Registered broker 1 at path > >>> > > /brokers/ids/1 with address hdfs-ix03.se-ix.delta.prod:58668. > >>> > > 16/01/23 12:49:31 INFO server.KafkaServer: [Kafka Server 1], > >>> > > Connecting to ZK: > >>> > > > >>> > > > >>> > > >>> > zookeeper01.se-ix.delta.prod/JarApp/be4bbf01-5e72-4147-b2eb-b84e19214b5b/kafka > >>> > > 16/01/23 12:49:31 INFO utils.VerifiableProperties: Verifying > properties > >>> > > 16/01/23 12:49:31 INFO utils.VerifiableProperties: Property > >>> > > metadata.broker.list is overridden to > hdfs-ix03.se-ix.delta.prod:58668 > >>> > > 16/01/23 12:49:31 INFO utils.VerifiableProperties: Property > >>> > > request.required.acks is overridden to 1 > >>> > > 16/01/23 12:49:31 INFO utils.VerifiableProperties: Property > >>> > > partitioner.class is overridden to > >>> > > org.apache.twill.internal.kafka.client.IntegerPartitioner > >>> > > 16/01/23 12:49:31 INFO utils.VerifiableProperties: Property > >>> > > compression.codec is overridden to snappy > >>> > > 16/01/23 12:49:31 INFO utils.VerifiableProperties: Property > >>> > > key.serializer.class is overridden to > >>> > > org.apache.twill.internal.kafka.client.IntegerEncoder > >>> > > 16/01/23 12:49:31 INFO utils.VerifiableProperties: Property > >>> > > serializer.class is overridden to > >>> > > org.apache.twill.internal.kafka.client.ByteBufferEncoder > >>> > > 16/01/23 12:49:31 INFO utils.Mx4jLoader$: Will not load MX4J, > >>> > > mx4j-tools.jar is not in the classpath > >>> > > 16/01/23 12:49:31 INFO controller.KafkaController: [Controller 1]: > >>> > > Controller starting up > >>> > > 16/01/23 12:49:31 INFO server.ZookeeperLeaderElector: 1 > successfully > >>> > > elected as leader > >>> > > 16/01/23 12:49:31 INFO controller.KafkaController: [Controller 1]: > >>> > > Broker 1 starting become controller state transition > >>> > > 16/01/23 12:49:31 INFO controller.KafkaController: [Controller 1]: > >>> > > Controller 1 incremented epoch to 1 > >>> > > 16/01/23 12:49:31 INFO client.ClientUtils$: Fetching metadata from > >>> > > broker id:0,host:hdfs-ix03.se-ix.delta.prod,port:58668 with > >>> > > correlation id 0 for 1 topic(s) Set(log) > >>> > > 16/01/23 12:49:31 INFO producer.SyncProducer: Connected to > >>> > > hdfs-ix03.se-ix.delta.prod:58668 for producing > >>> > > 16/01/23 12:49:31 INFO controller.RequestSendThread: > >>> > > [Controller-1-to-broker-1-send-thread], Starting > >>> > > 16/01/23 12:49:31 INFO controller.KafkaController: [Controller 1]: > >>> > > Currently active brokers in the cluster: Set(1) > >>> > > 16/01/23 12:49:31 INFO controller.KafkaController: [Controller 1]: > >>> > > Currently shutting brokers in the cluster: Set() > >>> > > 16/01/23 12:49:31 INFO controller.KafkaController: [Controller 1]: > >>> > > Current list of topics in the cluster: Set() > >>> > > 16/01/23 12:49:31 INFO controller.ReplicaStateMachine: [Replica > state > >>> > > machine on controller 1]: No state transitions triggered since no > >>> > > partitions are assigned to brokers 1 > >>> > > 16/01/23 12:49:31 INFO controller.ReplicaStateMachine: [Replica > state > >>> > > machine on controller 1]: Invoking state change to OnlineReplica > for > >>> > > replicas > >>> > > 16/01/23 12:49:31 INFO controller.ReplicaStateMachine: [Replica > state > >>> > > machine on controller 1]: Started replica state machine with > initial > >>> > > state -> Map() > >>> > > 16/01/23 12:49:31 INFO controller.PartitionStateMachine: [Partition > >>> > > state machine on Controller 1]: Started partition state machine > with > >>> > > initial state -> Map() > >>> > > 16/01/23 12:49:31 INFO controller.KafkaController: [Controller 1]: > >>> > > Broker 1 is ready to serve as the new controller with epoch 1 > >>> > > 16/01/23 12:49:31 INFO controller.KafkaController: [Controller 1]: > >>> > > Partitions being reassigned: Map() > >>> > > 16/01/23 12:49:31 INFO controller.KafkaController: [Controller 1]: > >>> > > Partitions already reassigned: List() > >>> > > 16/01/23 12:49:31 INFO controller.KafkaController: [Controller 1]: > >>> > > Resuming reassignment of partitions: Map() > >>> > > 16/01/23 12:49:31 INFO controller.KafkaController: [Controller 1]: > >>> > > Partitions undergoing preferred replica election: > >>> > > 16/01/23 12:49:31 INFO admin.AdminUtils$: Topic creation { > >>> > > "partitions":{ "0":[ 1 ] }, "version":1 } > >>> > > 16/01/23 12:49:31 INFO controller.KafkaController: [Controller 1]: > >>> > > Partitions that completed preferred replica election: > >>> > > 16/01/23 12:49:31 INFO controller.KafkaController: [Controller 1]: > >>> > > Resuming preferred replica election for partitions: > >>> > > 16/01/23 12:49:31 INFO controller.KafkaController: [Controller 1]: > >>> > > Starting preferred replica leader election for partitions > >>> > > 16/01/23 12:49:31 INFO controller.PartitionStateMachine: [Partition > >>> > > state machine on Controller 1]: Invoking state change to > >>> > > OnlinePartition for partitions > >>> > > 16/01/23 12:49:31 INFO controller.KafkaController: [Controller 1]: > >>> > > Controller startup complete > >>> > > 16/01/23 12:49:31 INFO server.KafkaApis: [KafkaApi-1] Auto > creation of > >>> > > topic log with 1 partitions and replication factor 1 is successful! > >>> > > 16/01/23 12:49:31 INFO server.KafkaServer: [Kafka Server 1], > Started > >>> > > 16/01/23 12:49:31 INFO > >>> > > server.ZookeeperLeaderElector$LeaderChangeListener: New leader is 1 > >>> > > 16/01/23 12:49:31 INFO controller.ControllerEpochListener: > >>> > > [ControllerEpochListener on 1]: Initialized controller epoch to 1 > and > >>> > > zk version 0 > >>> > > 16/01/23 12:49:31 INFO producer.SyncProducer: Disconnecting from > >>> > > hdfs-ix03.se-ix.delta.prod:58668 > >>> > > 16/01/23 12:49:31 WARN producer.BrokerPartitionInfo: Error while > >>> > > fetching metadata [{TopicMetadata for topic log -> > >>> > > No partition metadata for topic log due to > >>> > > kafka.common.LeaderNotAvailableException}] for topic [log]: class > >>> > > kafka.common.LeaderNotAvailableException > >>> > > 16/01/23 12:49:31 INFO client.ClientUtils$: Fetching metadata from > >>> > > broker id:0,host:hdfs-ix03.se-ix.delta.prod,port:58668 with > >>> > > correlation id 1 for 1 topic(s) Set(log) > >>> > > 16/01/23 12:49:31 INFO producer.SyncProducer: Connected to > >>> > > hdfs-ix03.se-ix.delta.prod:58668 for producing > >>> > > 16/01/23 12:49:31 INFO > >>> > > controller.PartitionStateMachine$TopicChangeListener: > >>> > > [TopicChangeListener on Controller 1]: New topics: [Set(log)], > deleted > >>> > > topics: [Set()], new partition replica assignment [Map([log,0] -> > >>> > > List(1))] > >>> > > 16/01/23 12:49:31 INFO admin.AdminUtils$: Topic creation { > >>> > > "partitions":{ "0":[ 1 ] }, "version":1 } > >>> > > 16/01/23 12:49:31 INFO controller.KafkaController: [Controller 1]: > New > >>> > > topic creation callback for [log,0] > >>> > > 16/01/23 12:49:31 INFO controller.KafkaController: [Controller 1]: > New > >>> > > partition creation callback for [log,0] > >>> > > 16/01/23 12:49:31 INFO producer.SyncProducer: Disconnecting from > >>> > > hdfs-ix03.se-ix.delta.prod:58668 > >>> > > 16/01/23 12:49:31 INFO controller.PartitionStateMachine: [Partition > >>> > > state machine on Controller 1]: Invoking state change to > NewPartition > >>> > > for partitions [log,0] > >>> > > 16/01/23 12:49:31 WARN producer.BrokerPartitionInfo: Error while > >>> > > fetching metadata [{TopicMetadata for topic log -> > >>> > > No partition metadata for topic log due to > >>> > > kafka.common.LeaderNotAvailableException}] for topic [log]: class > >>> > > kafka.common.LeaderNotAvailableException > >>> > > 16/01/23 12:49:31 INFO network.Processor: Closing socket > connection to > >>> > > /10.3.24.22. > >>> > > 16/01/23 12:49:31 INFO network.Processor: Closing socket > connection to > >>> > > /10.3.24.22. > >>> > > 16/01/23 12:49:31 ERROR async.DefaultEventHandler: Failed to > collate > >>> > > messages by topic, partition due to: Failed to fetch topic metadata > >>> > > for topic: log > >>> > > 16/01/23 12:49:31 INFO async.DefaultEventHandler: Back off for 100 > ms > >>> > > before retrying send. Remaining retries = 3 > >>> > > 16/01/23 12:49:31 INFO controller.ReplicaStateMachine: [Replica > state > >>> > > machine on controller 1]: Invoking state change to NewReplica for > >>> > > replicas PartitionAndReplica(log,0,1) > >>> > > 16/01/23 12:49:31 INFO controller.PartitionStateMachine: [Partition > >>> > > state machine on Controller 1]: Invoking state change to > >>> > > OnlinePartition for partitions [log,0] > >>> > > 16/01/23 12:49:31 INFO controller.ReplicaStateMachine: [Replica > state > >>> > > machine on controller 1]: Invoking state change to OnlineReplica > for > >>> > > replicas PartitionAndReplica(log,0,1) > >>> > > 16/01/23 12:49:31 INFO server.ReplicaManager: [Replica Manager on > >>> > > Broker 1]: Handling LeaderAndIsr request > >>> > > > >>> > > > >>> > > >>> > Name:LeaderAndIsrRequest;Version:0;Controller:1;ControllerEpoch:1;CorrelationId:6;ClientId:id_1-host_null-port_58668;PartitionState:(log,0) > >>> > > -> > >>> > > > >>> > > >>> > (LeaderAndIsrInfo:(Leader:1,ISR:1,LeaderEpoch:0,ControllerEpoch:1),ReplicationFactor:1),AllReplicas:1);Leaders:id:1,host:hdfs-ix03.se-ix.delta.prod,port:58668 > >>> > > 16/01/23 12:49:31 INFO server.ReplicaFetcherManager: > >>> > > [ReplicaFetcherManager on broker 1] Removing fetcher for partition > >>> > > [log,0] > >>> > > 16/01/23 12:49:31 INFO log.Log: [Kafka Log on Broker 1], Completed > >>> > > load of log log-0 with log end offset 0 > >>> > > 16/01/23 12:49:31 INFO log.LogManager: [Log Manager on Broker 1] > >>> > > Created log for partition [log,0] in > >>> > > > >>> > > > >>> > > >>> > /hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/kafka-logs. > >>> > > 16/01/23 12:49:31 WARN server.HighwaterMarkCheckpoint: No > >>> > > highwatermark file is found. Returning 0 as the highwatermark for > >>> > > partition [log,0] > >>> > > 16/01/23 12:49:31 INFO server.ReplicaManager: [Replica Manager on > >>> > > Broker 1]: Handled leader and isr request > >>> > > > >>> > > > >>> > > >>> > Name:LeaderAndIsrRequest;Version:0;Controller:1;ControllerEpoch:1;CorrelationId:6;ClientId:id_1-host_null-port_58668;PartitionState:(log,0) > >>> > > -> > >>> > > > >>> > > >>> > (LeaderAndIsrInfo:(Leader:1,ISR:1,LeaderEpoch:0,ControllerEpoch:1),ReplicationFactor:1),AllReplicas:1);Leaders:id:1,host:hdfs-ix03.se-ix.delta.prod,port:58668 > >>> > > 16/01/23 12:49:31 INFO client.ClientUtils$: Fetching metadata from > >>> > > broker id:0,host:hdfs-ix03.se-ix.delta.prod,port:58668 with > >>> > > correlation id 2 for 1 topic(s) Set(log) > >>> > > 16/01/23 12:49:31 INFO producer.SyncProducer: Connected to > >>> > > hdfs-ix03.se-ix.delta.prod:58668 for producing > >>> > > 16/01/23 12:49:31 INFO admin.AdminUtils$: Topic creation { > >>> > > "partitions":{ "0":[ 1 ] }, "version":1 } > >>> > > 16/01/23 12:49:31 INFO producer.SyncProducer: Disconnecting from > >>> > > hdfs-ix03.se-ix.delta.prod:58668 > >>> > > 16/01/23 12:49:31 INFO network.Processor: Closing socket > connection to > >>> > > /10.3.24.22. > >>> > > 16/01/23 12:49:31 WARN producer.BrokerPartitionInfo: Error while > >>> > > fetching metadata [{TopicMetadata for topic log -> > >>> > > No partition metadata for topic log due to > >>> > > kafka.common.LeaderNotAvailableException}] for topic [log]: class > >>> > > kafka.common.LeaderNotAvailableException > >>> > > 16/01/23 12:49:31 INFO client.ClientUtils$: Fetching metadata from > >>> > > broker id:0,host:hdfs-ix03.se-ix.delta.prod,port:58668 with > >>> > > correlation id 3 for 1 topic(s) Set(log) > >>> > > 16/01/23 12:49:31 INFO producer.SyncProducer: Connected to > >>> > > hdfs-ix03.se-ix.delta.prod:58668 for producing > >>> > > 16/01/23 12:49:31 INFO producer.SyncProducer: Disconnecting from > >>> > > hdfs-ix03.se-ix.delta.prod:58668 > >>> > > 16/01/23 12:49:31 INFO network.Processor: Closing socket > connection to > >>> > > /10.3.24.22. > >>> > > 16/01/23 12:49:31 INFO producer.SyncProducer: Connected to > >>> > > hdfs-ix03.se-ix.delta.prod:58668 for producing > >>> > > 16/01/23 12:49:33 INFO impl.AMRMClientImpl: Received new token for > : > >>> > > hdfs-ix03.se-ix.delta.prod:45454 > >>> > > 16/01/23 12:49:33 INFO impl.ContainerManagementProtocolProxy: > Opening > >>> > > proxy : hdfs-ix03.se-ix.delta.prod:45454 > >>> > > 16/01/23 12:49:35 INFO network.Processor: Closing socket > connection to > >>> > > /10.3.24.22. > >>> > > 16/01/23 12:49:35 INFO network.Processor: Closing socket > connection to > >>> > > /10.3.24.22. > >>> > > 16/01/23 12:49:39 INFO server.KafkaServer: [Kafka Server 1], > Shutting > >>> > down > >>> > > 16/01/23 12:49:39 INFO server.KafkaZooKeeper: Closing zookeeper > >>> client... > >>> > > 16/01/23 12:49:39 INFO zkclient.ZkEventThread: Terminate ZkClient > event > >>> > > thread. > >>> > > 16/01/23 12:49:39 INFO network.SocketServer: [Socket Server on > Broker > >>> > > 1], Shutting down > >>> > > 16/01/23 12:49:39 INFO network.SocketServer: [Socket Server on > Broker > >>> > > 1], Shutdown completed > >>> > > 16/01/23 12:49:39 INFO server.KafkaRequestHandlerPool: [Kafka > Request > >>> > > Handler on Broker 1], shutting down > >>> > > 16/01/23 12:49:39 INFO server.KafkaRequestHandlerPool: [Kafka > Request > >>> > > Handler on Broker 1], shutted down completely > >>> > > 16/01/23 12:49:39 INFO utils.KafkaScheduler: Shutdown Kafka > scheduler > >>> > > 16/01/23 12:49:39 INFO server.ReplicaManager: [Replica Manager on > >>> > > Broker 1]: Shut down > >>> > > 16/01/23 12:49:39 INFO server.ReplicaFetcherManager: > >>> > > [ReplicaFetcherManager on broker 1] shutting down > >>> > > 16/01/23 12:49:39 INFO server.ReplicaFetcherManager: > >>> > > [ReplicaFetcherManager on broker 1] shutdown completed > >>> > > 16/01/23 12:49:39 INFO server.ReplicaManager: [Replica Manager on > >>> > > Broker 1]: Shutted down completely > >>> > > 16/01/23 12:49:40 INFO controller.RequestSendThread: > >>> > > [Controller-1-to-broker-1-send-thread], Shutting down > >>> > > 16/01/23 12:49:40 INFO controller.RequestSendThread: > >>> > > [Controller-1-to-broker-1-send-thread], Stopped > >>> > > 16/01/23 12:49:40 INFO controller.RequestSendThread: > >>> > > [Controller-1-to-broker-1-send-thread], Shutdown completed > >>> > > 16/01/23 12:49:40 INFO controller.KafkaController: [Controller 1]: > >>> > > Controller shutdown complete > >>> > > 16/01/23 12:49:40 INFO server.KafkaServer: [Kafka Server 1], Shut > down > >>> > > completed > >>> > > 16/01/23 12:49:40 INFO impl.ContainerManagementProtocolProxy: > Opening > >>> > > proxy : hdfs-ix03.se-ix.delta.prod:45454 > >>> > > 16/01/23 12:49:40 INFO impl.AMRMClientImpl: Waiting for > application to > >>> > > be successfully unregistered. > >>> > > 16/01/23 12:49:40 INFO producer.SyncProducer: Disconnecting from > >>> > > hdfs-ix03.se-ix.delta.prod:58668 > >>> > > 16/01/23 12:49:40 WARN async.DefaultEventHandler: Failed to send > >>> > > producer request with correlation id 35 to broker 1 with data for > >>> > > partitions [log,0] > >>> > > java.nio.channels.ClosedByInterruptException > >>> > > at > >>> > > > >>> > > >>> > java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202) > >>> > > at sun.nio.ch.SocketChannelImpl.poll(SocketChannelImpl.java:957) > >>> > > at > >>> > > sun.nio.ch.SocketAdaptor$SocketInputStream.read(SocketAdaptor.java:204) > >>> > > at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103) > >>> > > at > >>> > > > >>> > > >>> > java.nio.channels.Channels$ReadableByteChannelImpl.read(Channels.java:385) > >>> > > at kafka.utils.Utils$.read(Unknown Source) > >>> > > at kafka.network.BoundedByteBufferReceive.readFrom(Unknown Source) > >>> > > at kafka.network.Receive$class.readCompletely(Unknown Source) > >>> > > at kafka.network.BoundedByteBufferReceive.readCompletely(Unknown > >>> Source) > >>> > > at kafka.network.BlockingChannel.receive(Unknown Source) > >>> > > at kafka.producer.SyncProducer.liftedTree1$1(Unknown Source) > >>> > > at > >>> > > kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(Unknown > >>> > > Source) > >>> > > at > >>> > > > >>> > > >>> > kafka.producer.SyncProducer$$anonfun$send$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(Unknown > >>> > > Source) > >>> > > at > >>> > > > >>> > > >>> > kafka.producer.SyncProducer$$anonfun$send$1$$anonfun$apply$mcV$sp$1.apply(Unknown > >>> > > Source) > >>> > > at > >>> > > > >>> > > >>> > kafka.producer.SyncProducer$$anonfun$send$1$$anonfun$apply$mcV$sp$1.apply(Unknown > >>> > > Source) > >>> > > at kafka.metrics.KafkaTimer.time(Unknown Source) > >>> > > at kafka.producer.SyncProducer$$anonfun$send$1.apply$mcV$sp(Unknown > >>> > Source) > >>> > > at kafka.producer.SyncProducer$$anonfun$send$1.apply(Unknown > Source) > >>> > > at kafka.producer.SyncProducer$$anonfun$send$1.apply(Unknown > Source) > >>> > > at kafka.metrics.KafkaTimer.time(Unknown Source) > >>> > > at kafka.producer.SyncProducer.send(Unknown Source) > >>> > > at > >>> > > > >>> > > >>> > kafka.producer.async.DefaultEventHandler.kafka$producer$async$DefaultEventHandler$$send(Unknown > >>> > > Source) > >>> > > at > >>> > > > >>> > > >>> > kafka.producer.async.DefaultEventHandler$$anonfun$dispatchSerializedData$2.apply(Unknown > >>> > > Source) > >>> > > at > >>> > > > >>> > > >>> > kafka.producer.async.DefaultEventHandler$$anonfun$dispatchSerializedData$2.apply(Unknown > >>> > > Source) > >>> > > at > >>> > > > >>> > > >>> > scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:772) > >>> > > at > >>> > > > >>> > > >>> > scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:98) > >>> > > at > >>> > > > >>> > > >>> > scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:98) > >>> > > at > >>> > > > >>> > > >>> > scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:226) > >>> > > at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:39) > >>> > > at scala.collection.mutable.HashMap.foreach(HashMap.scala:98) > >>> > > at > >>> > > > >>> > > >>> > scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:771) > >>> > > at > >>> > > kafka.producer.async.DefaultEventHandler.dispatchSerializedData(Unknown > >>> > > Source) > >>> > > at kafka.producer.async.DefaultEventHandler.handle(Unknown Source) > >>> > > at kafka.producer.Producer.send(Unknown Source) > >>> > > at kafka.javaapi.producer.Producer.send(Unknown Source) > >>> > > at > >>> > > > >>> > > >>> > org.apache.twill.internal.kafka.client.SimpleKafkaPublisher$SimplePreparer.send(SimpleKafkaPublisher.java:122) > >>> > > at > >>> > > > >>> > > >>> > org.apache.twill.internal.logging.KafkaAppender.doPublishLogs(KafkaAppender.java:268) > >>> > > at > >>> > > > >>> > > >>> > org.apache.twill.internal.logging.KafkaAppender.publishLogs(KafkaAppender.java:228) > >>> > > at > >>> > > > >>> > > >>> > org.apache.twill.internal.logging.KafkaAppender.access$700(KafkaAppender.java:66) > >>> > > at > >>> > > > >>> > > >>> > org.apache.twill.internal.logging.KafkaAppender$2.run(KafkaAppender.java:280) > >>> > > at > >>> > > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > >>> > > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) > >>> > > at > >>> > > > >>> > > >>> > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) > >>> > > at > >>> > > > >>> > > >>> > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) > >>> > > at > >>> > > > >>> > > >>> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > >>> > > at > >>> > > > >>> > > >>> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > >>> > > at java.lang.Thread.run(Thread.java:745) > >>> > > 16/01/23 12:49:40 INFO async.DefaultEventHandler: Back off for 100 > ms > >>> > > before retrying send. Remaining retries = 3 > >>> > > 16/01/23 12:49:40 INFO producer.Producer: Shutting down producer > >>> > > 16/01/23 12:49:40 INFO producer.ProducerPool: Closing all sync > >>> producers > >>> > > > >>> > > > >>> > > On Sat, Jan 23, 2016 at 1:22 AM, Terence Yim <[email protected]> > wrote: > >>> > > > Hi, > >>> > > > > >>> > > > It's due to a very old version of ASM library that bring it by > >>> > > hadoop/yarn. > >>> > > > Please add exclusion of asm library to all hadoop dependencies. > >>> > > > > >>> > > > <exclusion> > >>> > > > <groupId>asm</groupId> > >>> > > > <artifactId>asm</artifactId> > >>> > > > </exclusion> > >>> > > > > >>> > > > Terence > >>> > > > > >>> > > > > >>> > > > On Fri, Jan 22, 2016 at 2:34 PM, Kristoffer Sjögren < > >>> [email protected]> > >>> > > > wrote: > >>> > > > > >>> > > >> Further adding the following dependencies cause another > exception. > >>> > > >> > >>> > > >> <dependency> > >>> > > >> <groupId>com.google.guava</groupId> > >>> > > >> <artifactId>guava</artifactId> > >>> > > >> <version>13.0</version> > >>> > > >> </dependency> > >>> > > >> <dependency> > >>> > > >> <groupId>org.apache.hadoop</groupId> > >>> > > >> <artifactId>hadoop-hdfs</artifactId> > >>> > > >> <version>2.7.1</version> > >>> > > >> </dependency> > >>> > > >> > >>> > > >> Exception in thread " STARTING" > >>> > > >> java.lang.IncompatibleClassChangeError: class > >>> > > >> > org.apache.twill.internal.utils.Dependencies$DependencyClassVisitor > >>> > > >> has interface org.objectweb.asm.ClassVisitor as super class > >>> > > >> at java.lang.ClassLoader.defineClass1(Native Method) > >>> > > >> at java.lang.ClassLoader.defineClass(ClassLoader.java:760) > >>> > > >> at > >>> > > > java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) > >>> > > >> at java.net.URLClassLoader.defineClass(URLClassLoader.java:467) > >>> > > >> at java.net.URLClassLoader.access$100(URLClassLoader.java:73) > >>> > > >> at java.net.URLClassLoader$1.run(URLClassLoader.java:368) > >>> > > >> at java.net.URLClassLoader$1.run(URLClassLoader.java:362) > >>> > > >> at java.security.AccessController.doPrivileged(Native Method) > >>> > > >> at java.net.URLClassLoader.findClass(URLClassLoader.java:361) > >>> > > >> at java.lang.ClassLoader.loadClass(ClassLoader.java:424) > >>> > > >> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331) > >>> > > >> at java.lang.ClassLoader.loadClass(ClassLoader.java:357) > >>> > > >> at > >>> > > >> > >>> > > > >>> > > >>> > org.apache.twill.internal.utils.Dependencies.findClassDependencies(Dependencies.java:86) > >>> > > >> at > >>> > > >> > >>> > > > >>> > > >>> > org.apache.twill.internal.ApplicationBundler.findDependencies(ApplicationBundler.java:198) > >>> > > >> at > >>> > > >> > >>> > > > >>> > > >>> > org.apache.twill.internal.ApplicationBundler.createBundle(ApplicationBundler.java:155) > >>> > > >> at > >>> > > >> > >>> > > > >>> > > >>> > org.apache.twill.internal.ApplicationBundler.createBundle(ApplicationBundler.java:126) > >>> > > >> at > >>> > > >> > >>> > > > >>> > > >>> > org.apache.twill.yarn.YarnTwillPreparer.createAppMasterJar(YarnTwillPreparer.java:402) > >>> > > >> at > >>> > > >> > >>> > > > >>> > > >>> > org.apache.twill.yarn.YarnTwillPreparer.access$200(YarnTwillPreparer.java:108) > >>> > > >> at > >>> > > >> > >>> > > > >>> > > >>> > org.apache.twill.yarn.YarnTwillPreparer$1.call(YarnTwillPreparer.java:299) > >>> > > >> at > >>> > > >> > >>> > > > >>> > > >>> > org.apache.twill.yarn.YarnTwillPreparer$1.call(YarnTwillPreparer.java:289) > >>> > > >> at > >>> > > >> > >>> > > > >>> > > >>> > org.apache.twill.yarn.YarnTwillController.doStartUp(YarnTwillController.java:97) > >>> > > >> at > >>> > > >> > >>> > > > >>> > > >>> > org.apache.twill.internal.AbstractZKServiceController.startUp(AbstractZKServiceController.java:76) > >>> > > >> at > >>> > > >> > >>> > > > >>> > > >>> > org.apache.twill.internal.AbstractExecutionServiceController$ServiceDelegate.startUp(AbstractExecutionServiceController.java:175) > >>> > > >> at > >>> > > >> > >>> > > > >>> > > >>> > com.google.common.util.concurrent.AbstractIdleService$1$1.run(AbstractIdleService.java:43) > >>> > > >> at java.lang.Thread.run(Thread.java:745) > >>> > > >> > >>> > > >> On Fri, Jan 22, 2016 at 11:32 PM, Kristoffer Sjögren < > >>> > [email protected]> > >>> > > >> wrote: > >>> > > >> > Add those dependencies fail with the following exception. > >>> > > >> > > >>> > > >> > Exception in thread "main" java.lang.AbstractMethodError: > >>> > > >> > > >>> > > >> > >>> > > > >>> > > >>> > org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider.getProxy()Lorg/apache/hadoop/io/retry/FailoverProxyProvider$ProxyInfo; > >>> > > >> > at > >>> > > >> > >>> > > > >>> > > >>> > org.apache.hadoop.io.retry.RetryInvocationHandler.<init>(RetryInvocationHandler.java:73) > >>> > > >> > at > >>> > > >> > >>> > > > >>> > > >>> > org.apache.hadoop.io.retry.RetryInvocationHandler.<init>(RetryInvocationHandler.java:64) > >>> > > >> > at > >>> org.apache.hadoop.io.retry.RetryProxy.create(RetryProxy.java:59) > >>> > > >> > at > >>> > > >> > >>> > > > >>> > > >>> > org.apache.hadoop.hdfs.NameNodeProxies.createProxy(NameNodeProxies.java:149) > >>> > > >> > at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:569) > >>> > > >> > at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:512) > >>> > > >> > at > >>> > > >> > >>> > > > >>> > > >>> > org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:142) > >>> > > >> > at > >>> > > > org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2653) > >>> > > >> > at > org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:92) > >>> > > >> > at > >>> > > >> > >>> > > org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2687) > >>> > > >> > at > org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2669) > >>> > > >> > at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:371) > >>> > > >> > at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:170) > >>> > > >> > at > >>> > > >> > >>> > > > >>> > > >>> > org.apache.twill.yarn.YarnTwillRunnerService.createDefaultLocationFactory(YarnTwillRunnerService.java:615) > >>> > > >> > at > >>> > > >> > >>> > > > >>> > > >>> > org.apache.twill.yarn.YarnTwillRunnerService.<init>(YarnTwillRunnerService.java:149) > >>> > > >> > at deephacks.BundledJarExample.main(BundledJarExample.java:70) > >>> > > >> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > >>> > > >> > at > >>> > > >> > >>> > > > >>> > > >>> > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > >>> > > >> > at > >>> > > >> > >>> > > > >>> > > >>> > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > >>> > > >> > at java.lang.reflect.Method.invoke(Method.java:497) > >>> > > >> > at > >>> > > > com.intellij.rt.execution.application.AppMain.main(AppMain.java:144) > >>> > > >> > > >>> > > >> > On Fri, Jan 22, 2016 at 10:58 PM, Terence Yim < > [email protected]> > >>> > > wrote: > >>> > > >> >> Hi, > >>> > > >> >> > >>> > > >> >> If you run it from IDE, you and simply add a dependency on > hadoop > >>> > > with > >>> > > >> >> version 2.7.1. E.g. if you are using Maven, you can add the > >>> > > following to > >>> > > >> >> your pom.xml dependencies section. > >>> > > >> >> > >>> > > >> >> <dependency> > >>> > > >> >> <groupId>org.apache.hadoop</groupId> > >>> > > >> >> <artifactId>hadoop-yarn-api</artifactId> > >>> > > >> >> <version>2.7.1</version> > >>> > > >> >> </dependency> > >>> > > >> >> <dependency> > >>> > > >> >> <groupId>org.apache.hadoop</groupId> > >>> > > >> >> <artifactId>hadoop-yarn-common</artifactId> > >>> > > >> >> <version>2.7.1</version> > >>> > > >> >> </dependency> > >>> > > >> >> <dependency> > >>> > > >> >> <groupId>org.apache.hadoop</groupId> > >>> > > >> >> <artifactId>hadoop-yarn-client</artifactId> > >>> > > >> >> <version>2.7.1</version> > >>> > > >> >> </dependency> > >>> > > >> >> <dependency> > >>> > > >> >> <groupId>org.apache.hadoop</groupId> > >>> > > >> >> <artifactId>hadoop-common</artifactId> > >>> > > >> >> <version>2.7.1</version> > >>> > > >> >> </dependency> > >>> > > >> >> > >>> > > >> >> Terence > >>> > > >> >> > >>> > > >> >> On Fri, Jan 22, 2016 at 12:47 PM, Kristoffer Sjögren < > >>> > > [email protected]> > >>> > > >> >> wrote: > >>> > > >> >> > >>> > > >> >>> I run it from IDE right now, but would like to create a > command > >>> > line > >>> > > >> >>> app eventually. > >>> > > >> >>> > >>> > > >> >>> I should clarify that the exception above is thrown on the > YARN > >>> > > node, > >>> > > >> >>> not in the IDE. > >>> > > >> >>> > >>> > > >> >>> On Fri, Jan 22, 2016 at 9:32 PM, Terence Yim < > [email protected]> > >>> > > wrote: > >>> > > >> >>> > Hi Kristoffer, > >>> > > >> >>> > > >>> > > >> >>> > The example itself shouldn't need any modification. > However, > >>> how > >>> > > do > >>> > > >> >>> > you run that class? Do you run it from IDE or from command > >>> line > >>> > > using > >>> > > >> >>> > "java" command? > >>> > > >> >>> > > >>> > > >> >>> > Terence > >>> > > >> >>> > > >>> > > >> >>> > On Fri, Jan 22, 2016 at 12:16 PM, Kristoffer Sjögren < > >>> > > >> [email protected]> > >>> > > >> >>> wrote: > >>> > > >> >>> >> Hi Terence, > >>> > > >> >>> >> > >>> > > >> >>> >> I'm quite new to Twill and not sure how to do that > exactly. > >>> > Could > >>> > > >> you > >>> > > >> >>> >> show me how to modify the following example to do the > same? > >>> > > >> >>> >> > >>> > > >> >>> >> > >>> > > >> >>> > >>> > > >> > >>> > > > >>> > > >>> > https://github.com/apache/incubator-twill/blob/master/twill-examples/yarn/src/main/java/org/apache/twill/example/yarn/BundledJarExample.java > >>> > > >> >>> >> > >>> > > >> >>> >> On Fri, Jan 22, 2016 at 7:24 PM, Terence Yim < > >>> [email protected] > >>> > > > >>> > > >> wrote: > >>> > > >> >>> >>> Hi Kristoffer, > >>> > > >> >>> >>> > >>> > > >> >>> >>> Seems like the exception comes from the YARN class > >>> > > >> "ConverterUtils". I > >>> > > >> >>> >>> believe need to start the application with the version > 2.7.1 > >>> > > Hadoop > >>> > > >> >>> >>> Jars. How to do start the twill application? Usually on > a > >>> > > cluster > >>> > > >> with > >>> > > >> >>> >>> hadoop installed, you can get all the hadoop jars in the > >>> > > classpath > >>> > > >> by > >>> > > >> >>> >>> running this: > >>> > > >> >>> >>> > >>> > > >> >>> >>> export CP=`hadoop classpath` > >>> > > >> >>> >>> java -cp .:$CP YourApp ... > >>> > > >> >>> >>> > >>> > > >> >>> >>> Assuming your app classes and Twill jars are in the > current > >>> > > >> directory. > >>> > > >> >>> >>> > >>> > > >> >>> >>> Terence > >>> > > >> >>> >>> > >>> > > >> >>> >>> On Fri, Jan 22, 2016 at 4:54 AM, Kristoffer Sjögren < > >>> > > >> [email protected]> > >>> > > >> >>> wrote: > >>> > > >> >>> >>>> Here's the full stacktrace. > >>> > > >> >>> >>>> > >>> > > >> >>> >>>> Exception in thread "main" > >>> > > >> java.lang.reflect.InvocationTargetException > >>> > > >> >>> >>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native > >>> > Method) > >>> > > >> >>> >>>> at > >>> > > >> >>> > >>> > > >> > >>> > > > >>> > > >>> > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > >>> > > >> >>> >>>> at > >>> > > >> >>> > >>> > > >> > >>> > > > >>> > > >>> > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > >>> > > >> >>> >>>> at java.lang.reflect.Method.invoke(Method.java:497) > >>> > > >> >>> >>>> at > >>> > > >> > org.apache.twill.launcher.TwillLauncher.main(TwillLauncher.java:89) > >>> > > >> >>> >>>> Caused by: java.lang.RuntimeException: > >>> > > >> >>> >>>> java.lang.reflect.InvocationTargetException > >>> > > >> >>> >>>> at > >>> > > >> com.google.common.base.Throwables.propagate(Throwables.java:160) > >>> > > >> >>> >>>> at > >>> > > >> >>> > >>> > > >> > >>> > > > >>> > > >>> > org.apache.twill.internal.yarn.VersionDetectYarnAMClientFactory.create(VersionDetectYarnAMClientFactory.java:61) > >>> > > >> >>> >>>> at > >>> > > >> >>> > >>> > > >> > >>> > > > >>> > > >>> > org.apache.twill.internal.appmaster.ApplicationMasterMain.main(ApplicationMasterMain.java:77) > >>> > > >> >>> >>>> ... 5 more > >>> > > >> >>> >>>> Caused by: java.lang.reflect.InvocationTargetException > >>> > > >> >>> >>>> at > >>> > > sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native > >>> > > >> >>> Method) > >>> > > >> >>> >>>> at > >>> > > >> >>> > >>> > > >> > >>> > > > >>> > > >>> > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) > >>> > > >> >>> >>>> at > >>> > > >> >>> > >>> > > >> > >>> > > > >>> > > >>> > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > >>> > > >> >>> >>>> at > >>> > > java.lang.reflect.Constructor.newInstance(Constructor.java:422) > >>> > > >> >>> >>>> at > >>> > > >> >>> > >>> > > >> > >>> > > > >>> > > >>> > org.apache.twill.internal.yarn.VersionDetectYarnAMClientFactory.create(VersionDetectYarnAMClientFactory.java:58) > >>> > > >> >>> >>>> ... 6 more > >>> > > >> >>> >>>> Caused by: java.lang.IllegalArgumentException: Invalid > >>> > > >> ContainerId: > >>> > > >> >>> >>>> container_e25_1453466340022_0004_01_000001 > >>> > > >> >>> >>>> at > >>> > > >> >>> > >>> > > >> > >>> > > > >>> > > >>> > org.apache.hadoop.yarn.util.ConverterUtils.toContainerId(ConverterUtils.java:182) > >>> > > >> >>> >>>> at > >>> > > >> >>> > >>> > > >> > >>> > > > >>> > > >>> > org.apache.twill.internal.yarn.AbstractYarnAMClient.<init>(AbstractYarnAMClient.java:83) > >>> > > >> >>> >>>> at > >>> > > >> >>> > >>> > > >> > >>> > > > >>> > > >>> > org.apache.twill.internal.yarn.Hadoop21YarnAMClient.<init>(Hadoop21YarnAMClient.java:65) > >>> > > >> >>> >>>> at > >>> > > >> >>> > >>> > > >> > >>> > > > >>> > > >>> > org.apache.twill.internal.yarn.Hadoop22YarnAMClient.<init>(Hadoop22YarnAMClient.java:34) > >>> > > >> >>> >>>> ... 11 more > >>> > > >> >>> >>>> Caused by: java.lang.NumberFormatException: For input > >>> string: > >>> > > >> "e25" > >>> > > >> >>> >>>> at > >>> > > >> >>> > >>> > > >> > >>> > > > >>> > > >>> > java.lang.NumberFormatException.forInputString(NumberFormatException.java:65) > >>> > > >> >>> >>>> at java.lang.Long.parseLong(Long.java:589) > >>> > > >> >>> >>>> at java.lang.Long.parseLong(Long.java:631) > >>> > > >> >>> >>>> at > >>> > > >> >>> > >>> > > >> > >>> > > > >>> > > >>> > org.apache.hadoop.yarn.util.ConverterUtils.toApplicationAttemptId(ConverterUtils.java:137) > >>> > > >> >>> >>>> at > >>> > > >> >>> > >>> > > >> > >>> > > > >>> > > >>> > org.apache.hadoop.yarn.util.ConverterUtils.toContainerId(ConverterUtils.java:177) > >>> > > >> >>> >>>> ... 14 more > >>> > > >> >>> >>>> > >>> > > >> >>> >>>> On Thu, Jan 21, 2016 at 10:59 PM, Kristoffer Sjögren < > >>> > > >> >>> [email protected]> wrote: > >>> > > >> >>> >>>>> Hi > >>> > > >> >>> >>>>> > >>> > > >> >>> >>>>> I'm trying the basic example [1] on yarn 2.7.1 but > get an > >>> > > >> exception > >>> > > >> >>> as > >>> > > >> >>> >>>>> soon as the application starts on the resource manager > >>> that > >>> > > >> tells me > >>> > > >> >>> >>>>> the container id cannot be parsed. > >>> > > >> >>> >>>>> > >>> > > >> >>> >>>>> java.lang.IllegalArgumentException: Invalid > containerId: > >>> > > >> >>> >>>>> container_e04_1427159778706_0002_01_000001 > >>> > > >> >>> >>>>> > >>> > > >> >>> >>>>> I don't have the exact stacktrace but I recall it > failing > >>> in > >>> > > >> >>> >>>>> ConverterUtils.toContainerId because it assumes that > that > >>> > the > >>> > > >> first > >>> > > >> >>> >>>>> token is an application attempt to be parsed as an > >>> integer. > >>> > > This > >>> > > >> >>> class > >>> > > >> >>> >>>>> resides in hadoop-yarn-common 2.3.0. > >>> > > >> >>> >>>>> > >>> > > >> >>> >>>>> Is there any way to either tweak the container id or > make > >>> > > twill > >>> > > >> use > >>> > > >> >>> >>>>> the 2.7.1 jar instead? > >>> > > >> >>> >>>>> > >>> > > >> >>> >>>>> Cheers, > >>> > > >> >>> >>>>> -Kristoffer > >>> > > >> >>> >>>>> > >>> > > >> >>> >>>>> > >>> > > >> >>> >>>>> [1] > >>> > > >> >>> > >>> > > >> > >>> > > > >>> > > >>> > https://github.com/apache/incubator-twill/blob/master/twill-examples/yarn/src/main/java/org/apache/twill/example/yarn/BundledJarExample.java > >>> > > >> >>> > >>> > > >> > >>> > > > >>> > > >>> >
