Hm... interesting. What did you see in the application master's logs? I saw that the remaining processes running are SamzaAppMasters.
On Tue, Sep 22, 2015 at 1:05 AM, Jordi Blasi Uribarri <jbl...@nextel.es> wrote: > Hi, > > I have two machines running yarn and samza. They are samza 0.9.1 and > hadoop 2.6.0. > > I run the kill-all.sh I recently wrote and calls the kill-yarn-job.sh. > This is the output: > > java version "1.7.0_79" > OpenJDK Runtime Environment (IcedTea 2.5.6) (7u79-2.5.6-1~deb7u1) > OpenJDK 64-Bit Server VM (build 24.79-b02, mixed mode) > SLF4J: Class path contains multiple SLF4J bindings. > SLF4J: Found binding in > [jar:file:/opt/jobs/lib/redirect-0.0.1.jar!/org/slf4j/impl/StaticLoggerBinder.class] > SLF4J: Found binding in > [jar:file:/opt/jobs/lib/samzafroga-0.0.1-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class] > SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an > explanation. > java version "1.7.0_79" > OpenJDK Runtime Environment (IcedTea 2.5.6) (7u79-2.5.6-1~deb7u1) > OpenJDK 64-Bit Server VM (build 24.79-b02, mixed mode) > /usr/lib/jvm/java-7-openjdk-amd64/bin/java > -Dlog4j.configuration=file:bin/log4j-console.xml -Dsamza.log.dir=/opt/jobs > -Djava.io.tmpdir=/opt/jobs/tmp -Xmx768M -XX:+PrintGCDateStamps > -Xloggc:/opt/jobs/gc.log -XX:+UseGCLogFileRotation > -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=10241024 -d64 -cp > /opt/hadoop-2.6.0/conf:/opt/jobs/lib/redirect-0.0.1.jar:/opt/jobs/lib/samzafroga-0.0.1-jar-with-dependencies.jar > org.apache.hadoop.yarn.client.cli.ApplicationCLI application -kill > application_1442908447829_0001 > 2015-09-22 10:02:46 RMProxy [INFO] Connecting to ResourceManager at > kfk-samza01/192.168.15.92:8032 > SLF4J: Class path contains multiple SLF4J bindings. > SLF4J: Found binding in > [jar:file:/opt/jobs/lib/redirect-0.0.1.jar!/org/slf4j/impl/StaticLoggerBinder.class] > SLF4J: Found binding in > [jar:file:/opt/jobs/lib/samzafroga-0.0.1-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class] > SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an > explanation. > 2015-09-22 10:02:46 NativeCodeLoader [WARN] Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > Killing application application_1442908447829_0001 > 2015-09-22 10:02:47 YarnClientImpl [INFO] Killed application > application_1442908447829_0001 > java version "1.7.0_79" > OpenJDK Runtime Environment (IcedTea 2.5.6) (7u79-2.5.6-1~deb7u1) > OpenJDK 64-Bit Server VM (build 24.79-b02, mixed mode) > /usr/lib/jvm/java-7-openjdk-amd64/bin/java > -Dlog4j.configuration=file:bin/log4j-console.xml -Dsamza.log.dir=/opt/jobs > -Djava.io.tmpdir=/opt/jobs/tmp -Xmx768M -XX:+PrintGCDateStamps > -Xloggc:/opt/jobs/gc.log -XX:+UseGCLogFileRotation > -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=10241024 -d64 -cp > /opt/hadoop-2.6.0/conf:/opt/jobs/lib/redirect-0.0.1.jar:/opt/jobs/lib/samzafroga-0.0.1-jar-with-dependencies.jar > org.apache.hadoop.yarn.client.cli.ApplicationCLI application -kill > application_1442908447829_0002 > 2015-09-22 10:02:49 RMProxy [INFO] Connecting to ResourceManager at > kfk-samza01/192.168.15.92:8032 > SLF4J: Class path contains multiple SLF4J bindings. > SLF4J: Found binding in > [jar:file:/opt/jobs/lib/redirect-0.0.1.jar!/org/slf4j/impl/StaticLoggerBinder.class] > SLF4J: Found binding in > [jar:file:/opt/jobs/lib/samzafroga-0.0.1-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class] > SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an > explanation. > 2015-09-22 10:02:49 NativeCodeLoader [WARN] Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > Killing application application_1442908447829_0002 > 2015-09-22 10:02:50 YarnClientImpl [INFO] Killed application > application_1442908447829_0002 > > When I run ps -fe |grep java I see this: > > root 9542 1 3 09:54 pts/0 00:00:21 > /usr/lib/jvm/java-7-openjdk-amd64/bin/java -Dproc_resourcemanager -Xmx1000m > -Dhadoop.log.dir=/opt/hadoop-2.6.0/logs > -Dyarn.log.dir=/opt/hadoop-2.6.0/logs > -Dhadoop.log.file=yarn-root-resourcemanager-kfk-samza01.log > -Dyarn.log.file=yarn-root-resourcemanager-kfk-samza01.log > -Dyarn.home.dir=/opt/hadoop-2.6.0 -Dhadoop.home.dir=/opt/hadoop-2.6.0 > -Dhadoop.root.logger=INFO,RFA -Dyarn.root.logger=INFO,RFA > -Djava.library.path=/opt/hadoop-2.6.0/lib/native -classpath > /opt/hadoop-2.6.0/conf:/opt/hadoop-2.6.0/conf:/opt/hadoop-2.6.0/conf:/opt/hadoop-2.6.0/share/hadoop/common/lib/*:/opt/hadoop-2.6.0/share/hadoop/common/*:/opt/hadoop-2.6.0/share/hadoop/hdfs:/opt/hadoop-2.6.0/share/hadoop/hdfs/lib/*:/opt/hadoop-2.6.0/share/hadoop/hdfs/*:/opt/hadoop-2.6.0/share/hadoop/yarn/lib/*:/opt/hadoop-2.6.0/share/hadoop/yarn/*:/opt/hadoop-2.6.0/share/hadoop/mapreduce/lib/*:/opt/hadoop-2.6.0/share/hadoop/mapreduce/*:/opt/hadoop-2.6.0/share/hadoop/yarn/*:/opt/hadoop-2.6.0/share/hadoop/yarn/lib/*:/opt/hadoop-2.6.0/conf/rm-config/log4j.properties > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager > root 9814 1 4 09:54 ? 00:00:24 > /usr/lib/jvm/java-7-openjdk-amd64/bin/java -Dproc_nodemanager -Xmx1000m > -server -Dhadoop.log.dir=/opt/hadoop-2.6.0/logs > -Dyarn.log.dir=/opt/hadoop-2.6.0/logs > -Dhadoop.log.file=yarn-root-nodemanager-kfk-samza01.log > -Dyarn.log.file=yarn-root-nodemanager-kfk-samza01.log > -Dyarn.home.dir=/opt/hadoop-2.6.0 -Dhadoop.home.dir=/opt/hadoop-2.6.0 > -Dhadoop.root.logger=INFO,RFA -Dyarn.root.logger=INFO,RFA > -Djava.library.path=/opt/hadoop-2.6.0/lib/native -classpath > /opt/hadoop-2.6.0/conf:/opt/hadoop-2.6.0/conf:/opt/hadoop-2.6.0/conf:/opt/hadoop-2.6.0/share/hadoop/common/lib/*:/opt/hadoop-2.6.0/share/hadoop/common/*:/opt/hadoop-2.6.0/share/hadoop/hdfs:/opt/hadoop-2.6.0/share/hadoop/hdfs/lib/*:/opt/hadoop-2.6.0/share/hadoop/hdfs/*:/opt/hadoop-2.6.0/share/hadoop/yarn/lib/*:/opt/hadoop-2.6.0/share/hadoop/yarn/*:/opt/hadoop-2.6.0/share/hadoop/mapreduce/lib/*:/opt/hadoop-2.6.0/share/hadoop/mapreduce/*:/opt/hadoop-2.6.0/share/hadoop/yarn/*:/opt/hadoop-2.6.0/share/hadoop/yarn/lib/*:/opt/hadoop-2.6.0/conf/nm-config/log4j.properties > org.apache.hadoop.yarn.server.nodemanager.NodeManager > root 10271 10268 0 09:54 ? 00:00:05 > /usr/lib/jvm/java-7-openjdk-amd64/bin/java -server > -Dsamza.container.name=samza-application-master > -Dsamza.log.dir=/opt/hadoop-2.6.0/logs/userlogs/application_1442908447829_0002/container_1442908447829_0002_01_000001 > -Djava.io.tmpdir=/tmp/hadoop-root/nm-local-dir/usercache/root/appcache/application_1442908447829_0002/container_1442908447829_0002_01_000001/__package/tmp > -Xmx768M -XX:+PrintGCDateStamps > -Xloggc:/opt/hadoop-2.6.0/logs/userlogs/application_1442908447829_0002/container_1442908447829_0002_01_000001/gc.log > -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 > -XX:GCLogFileSize=10241024 -d64 -cp > /opt/hadoop-2.6.0/conf:/tmp/hadoop-root/nm-local-dir/usercache/root/appcache/application_1442908447829_0002/container_1442908447829_0002_01_000001/__package/lib/jackson-annotations-2.6.0.jar:/tmp/hadoop-root/nm-local-dir/usercache/root/appcache/application_1442908447829_0002/container_1442908447829_0002_01_000001/__package/lib/jackson-core-2.6.0.jar:/tmp/hadoop-root/nm-local-dir/usercache/root/appcache/application_1442908447829_0002/container_1442908447829_0002_01_000001/__package/lib/jackson-databind-2.6.0.jar:/tmp/hadoop-root/nm-local-dir/usercache/root/appcache/application_1442908447829_0002/container_1442908447829_0002_01_000001/__package/lib/jackson-dataformat-smile-2.6.0.jar:/tmp/hadoop-root/nm-local-dir/usercache/root/appcache/application_1442908447829_0002/container_1442908447829_0002_01_000001/__package/lib/jackson-jaxrs-json-provider-2.6.0.jar:/tmp/hadoop-root/nm-local-dir/usercache/root/appcache/application_1442908447829_0002/container_1442908447829_0002_01_000001/__package/lib/jackson-module-jaxb-annotations-2.6.0.jar:/tmp/hadoop-root/nm-local-dir/usercache/root/appcache/application_1442908447829_0002/container_1442908447829_0002_01_000001/__package/lib/nxtBroker-0.0.1.jar:/tmp/hadoop-root/nm-local-dir/usercache/root/appcache/application_1442908447829_0002/container_1442908447829_0002_01_000001/__package/lib/nxtBroker-0.0.1-jar-with-dependencies.jar > org.apache.samza.job.yarn.SamzaAppMaster > root 10346 10344 0 09:54 ? 00:00:04 > /usr/lib/jvm/java-7-openjdk-amd64/bin/java -server > -Dsamza.container.name=samza-application-master > -Dsamza.log.dir=/opt/hadoop-2.6.0/logs/userlogs/application_1442908447829_0001/container_1442908447829_0001_01_000001 > -Djava.io.tmpdir=/tmp/hadoop-root/nm-local-dir/usercache/root/appcache/application_1442908447829_0001/container_1442908447829_0001_01_000001/__package/tmp > -Xmx768M -XX:+PrintGCDateStamps > -Xloggc:/opt/hadoop-2.6.0/logs/userlogs/application_1442908447829_0001/container_1442908447829_0001_01_000001/gc.log > -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 > -XX:GCLogFileSize=10241024 -d64 -cp > /opt/hadoop-2.6.0/conf:/tmp/hadoop-root/nm-local-dir/usercache/root/appcache/application_1442908447829_0001/container_1442908447829_0001_01_000001/__package/lib/jackson-annotations-2.6.0.jar:/tmp/hadoop-root/nm-local-dir/usercache/root/appcache/application_1442908447829_0001/container_1442908447829_0001_01_000001/__package/lib/jackson-core-2.6.0.jar:/tmp/hadoop-root/nm-local-dir/usercache/root/appcache/application_1442908447829_0001/container_1442908447829_0001_01_000001/__package/lib/jackson-databind-2.6.0.jar:/tmp/hadoop-root/nm-local-dir/usercache/root/appcache/application_1442908447829_0001/container_1442908447829_0001_01_000001/__package/lib/jackson-dataformat-smile-2.6.0.jar:/tmp/hadoop-root/nm-local-dir/usercache/root/appcache/application_1442908447829_0001/container_1442908447829_0001_01_000001/__package/lib/jackson-jaxrs-json-provider-2.6.0.jar:/tmp/hadoop-root/nm-local-dir/usercache/root/appcache/application_1442908447829_0001/container_1442908447829_0001_01_000001/__package/lib/jackson-module-jaxb-annotations-2.6.0.jar:/tmp/hadoop-root/nm-local-dir/usercache/root/appcache/application_1442908447829_0001/container_1442908447829_0001_01_000001/__package/lib/nxtBroker-0.0.1.jar:/tmp/hadoop-root/nm-local-dir/usercache/root/appcache/application_1442908447829_0001/container_1442908447829_0001_01_000001/__package/lib/nxtBroker-0.0.1-jar-with-dependencies.jar > org.apache.samza.job.yarn.SamzaAppMaster > > As you can see the proceses are still there. In the web application the > appear as KILLED. > > Thanks, > > Jordi > > -----Mensaje original----- > De: Yan Fang [mailto:yanfang...@gmail.com] > Enviado el: martes, 22 de septiembre de 2015 9:59 > Para: dev@samza.apache.org > Asunto: Re: process killing > > Hi Jordi, > > 1. Are you running the job in one machine yarn? or in a cluster? > > 2. what kind of the java process do you see after killing the yarn > application? Because usually, when we run kill-yarn-job applicationId, we > do kill all the processes (this is actually done by the Yarn). > > 3. Which version of Samza and Yarn are you using ? This matters sometimes. > > Thanks, > > Fang, Yan > yanfang...@gmail.com > > On Tue, Sep 22, 2015 at 3:42 PM, Jordi Blasi Uribarri <jbl...@nextel.es> > wrote: > > > Hi, > > > > I am currently developing solution using samza and in the development > > process I need to constantly change the code and test in the system. > > What I am seeing is that most of the times I kill a job using the > > kill-yarn-job script the job gets killed according to the web > > interface but I see the java process running. I also have seen that > > the job was actually been executed, as I got messages in the far end > > of the application. I have been manually killing these processes (kill > -9 ) but I have some questions: > > > > > > - Is there a reason for the processes not to be killed. It was > > not a matter of time as I could find them hours later. > > > > - I don’t know if there should be any other action performed to > > completely clean the information or killing the process the hard way > > is enough. > > > > - I am finding some memory consumption problems that I don’t > know > > if they are related with this. Maybe I will describe them in another > > message. > > > > Thnaks, > > > > Jordi > > ________________________________ > > Jordi Blasi Uribarri > > Área I+D+i > > > > jbl...@nextel.es > > Oficina Bilbao > > > > [http://www.nextel.es/wp-content/uploads/Firma_Nextel_2015.png] > > >
