[
https://issues.apache.org/jira/browse/GEARPUMP-185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15452397#comment-15452397
]
Kam Kasravi commented on GEARPUMP-185:
--------------------------------------
The problem (and resolution) was that the services process CLASSPATH did not
include gear.conf (gear.conf was not copied into the conf subdirectory) when
this process was installed in TAP. Once gear.conf was copied into this
directory everything works as expected where the gear.conf used to build
gearpump commands (output/target/pack/bin) is used by all commands. I think a
reasonable improvement is a WARNING emitted to console if gear.conf is not
found.
> Yarn kills Apache Gearpump's worker if the memory usage is too high
> -------------------------------------------------------------------
>
> Key: GEARPUMP-185
> URL: https://issues.apache.org/jira/browse/GEARPUMP-185
> Project: Apache Gearpump
> Issue Type: Bug
> Environment: yarn
> Reporter: Piotr Grabuszynski
> Assignee: Kam Kasravi
> Fix For: 0.8.0
>
>
> h3. Description
> If the traffic is to big (memory usage is height) Yarn kills Gearpump's
> worker.
> Clean YARN log (only the part with processes, bigger piece below):
> {code}
> Dump of the process-tree for container_e05_1468841606343_0128_01_000003 :
> |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS)
> SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
> |- 2702 2164 2159 2159 (java) 137198 34820 8656748544 199840
> /usr/java/jdk1.8.0_72/jre/bin/java -Dgearpump.username=vcap
> -Dgearpump.applicationId=1 -Dgearpump.executorId=0
> -Dgearpump.master.starttime=2016-07-20-08-36
> -Dgearpump.log.application.dir=logs -Djava.net.preferIPv4Stack=true
> -Dgearpump.config.file=/tmp/gearpump6797336735445779094.conf ...
> |- 2646 2164 2159 2159 (java) 4268 464 5847887872 70626
> /usr/java/jdk1.8.0_72/jre/bin/java -Dgearpump.username=vcap
> -Dgearpump.applicationId=1 -Dgearpump.executorId=-1
> -Dgearpump.master.starttime=2016-07-20-08-36
> -Dgearpump.log.application.dir=logs -Djava.net.preferIPv4Stack=true
> -Dgearpump.config.file=/tmp/gearpump941454978115257558.conf ...
> |- 2159 2157 2159 2159 (bash) 0 0 108654592 301 /bin/bash -c
> /usr/java/jdk1.8.0_72/bin/java -Xmx512m -cp
> conf:pack/gearpump-2.11-0.8.0/conf:pack/gearpump-2.11-0.8.0/lib/daemon/*:pack/gearpump-2.11-0.8.0/lib/*:/etc/hadoop/conf.cloudera.YARN:/var/run/cloudera-scm-agent/process/195-yarn-NODEMANAGER
> ...
> |- 2165 2159 2159 2159 (tee) 38 396 103337984 164 /usr/bin/tee -a
> /var/log/hadoop-yarn/container/application_1468841606343_0128/container_e05_1468841606343_0128_01_000003/stderr
>
> |- 2164 2159 2159 2159 (java) 4915 815 2367729664 78249
> /usr/java/jdk1.8.0_72/bin/java -Xmx512m -cp
> conf:pack/gearpump-2.11-0.8.0/conf:pack/gearpump-2.11-0.8.0/lib/daemon/*:pack/gearpump-2.11-0.8.0/lib/*:/etc/hadoop/conf.cloudera.YARN:/var/run/cloudera-scm-agent/process/195-yarn-NODEMANAGER
> ...
> |- 6417 2164 2159 2159 (java) 3900 1182 5893246976 125356
> /usr/java/jdk1.8.0_72/jre/bin/java -Dgearpump.username=vcap
> -Dgearpump.applicationId=2 -Dgearpump.executorId=0
> -Dgearpump.master.starttime=2016-07-20-08-36
> -Dgearpump.log.application.dir=logs -Djava.net.preferIPv4Stack=true ...
> |- 6360 2164 2159 2159 (java) 966 58 5845979136 54720
> /usr/java/jdk1.8.0_72/jre/bin/java -Dgearpump.username=vcap
> -Dgearpump.applicationId=2 -Dgearpump.executorId=-1
> -Dgearpump.master.starttime=2016-07-20-08-36
> -Dgearpump.log.application.dir=logs -Djava.net.preferIPv4Stack=true ...
> {code}
> *As we can see there are no "-Xmx" or "-Xsx" parameters near executors'
> processes.*
> In standalone mode, we can see both:
> {code}
> // MASTER
> GER\pgr+ 48280 5.3 3.1 5406960 261356 pts/13 Sl+ 09:28 0:10
> /usr/lib/jvm/java-8-oracle/bin/java -server -Djava.net.preferIPv4Stack=true
> -DlogFilename=master
> -Dgearpump.home=/home/local/GER/pgrabusz/Documents/gearpump-sources/gearpump-2.11-0.8.0
> -Djava.rmi.server.hostname=localhost ...
> // WORKER
> GER\pgr+ 48417 7.7 1.8 5392212 154728 pts/27 Sl+ 09:29 0:05
> /usr/lib/jvm/java-8-oracle/bin/java -server -Djava.net.preferIPv4Stack=true
> -DlogFilename=worker
> -Dgearpump.home=/home/local/GER/pgrabusz/Documents/gearpump-sources/gearpump-2.11-0.8.0
> -Djava.rmi.server.hostname=localhost ...
> // APP MASTER
> GER\pgr+ 48707 15.0 3.1 4572132 255776 pts/27 Sl+ 09:33 0:11
> /usr/lib/jvm/java-8-oracle/jre/bin/java -server -Xms512M -Xmx1024M -Xss1M
> -XX:+HeapDumpOnOutOfMemoryError -XX:+UseConcMarkSweepGC
> -XX:CMSInitiatingOccupancyFraction=80 -XX:+UseParNewGC -XX:NewRatio=3
> -Djava.rmi.server.hostname=localhost -Dgearpump.username=GER\pgrabusz
> -Dgearpump.applicationId=1 -Dgearpump.executorId=-1 ...
> // APP EXECUTOR 0
> GER\pgr+ 48761 132 5.2 7268592 427200 pts/27 Sl+ 09:33 2:31
> /usr/lib/jvm/java-8-oracle/jre/bin/java -server -Xms512M -Xmx1024M -Xss1M
> -XX:+HeapDumpOnOutOfMemoryError -XX:+UseConcMarkSweepGC
> -XX:CMSInitiatingOccupancyFraction=80 -XX:+UseParNewGC -XX:NewRatio=3 [...]
> -Dgearpump.applicationId=1 -Dgearpump.executorId=0 ...
> {code}
> h3. Steps to Reproduce
> To generate trafic I've used such application:
> {code}
> kafka source
> / \
> kafka sink hbase sink
> {code}
> Where kafka source write to the same topic as kafka source reads.
> Send some message to topic.
> Worker will disappear after few minutes (1-40 min depends on how much memory
> worker has);
> h3. Actual Results
> Worker has been killed
> h3. Additional information
> Increasing value of env WORKERS_MEMORY_LIMIT in gearpump-broker doesn't work.
> It increases only running time.
> It might be a problem with some java properties (hip size or sth). It is also
> possible there's some memory lick f.e. in hbase sink.
> Bigger piece of YARN log:
> {code}
> 2016-07-20 09:04:22,217 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
> Memory usage of ProcessTree 27737 for container-id
> container_e05_1468841606343_0021_01_000002: 285.7 MB of 1 GB physical memory
> used; 2.4 GB of 2.1 GB virtual memory used
> 2016-07-20 09:04:22,270 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
> Memory usage of ProcessTree 2046 for container-id
> container_e05_1468841606343_0128_01_000001: 356.7 MB of 1 GB physical memory
> used; 2.5 GB of 2.1 GB virtual memory used
> 2016-07-20 09:04:22,317 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
> Memory usage of ProcessTree 2159 for container-id
> container_e05_1468841606343_0128_01_000003: 2.0 GB of 2 GB physical memory
> used; 26.8 GB of 4.2 GB virtual memory used
> 2016-07-20 09:04:22,378 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
> Memory usage of ProcessTree 2116 for container-id
> container_e05_1468841606343_0128_01_000002: 348.0 MB of 1 GB physical memory
> used; 2.4 GB of 2.1 GB virtual memory used
> 2016-07-20 09:04:25,424 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
> Memory usage of ProcessTree 27737 for container-id
> container_e05_1468841606343_0021_01_000002: 285.7 MB of 1 GB physical memory
> used; 2.4 GB of 2.1 GB virtual memory used
> 2016-07-20 09:04:25,494 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
> Memory usage of ProcessTree 2046 for container-id
> container_e05_1468841606343_0128_01_000001: 356.7 MB of 1 GB physical memory
> used; 2.5 GB of 2.1 GB virtual memory used
> 2016-07-20 09:04:25,538 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
> Memory usage of ProcessTree 2159 for container-id
> container_e05_1468841606343_0128_01_000003: 2.0 GB of 2 GB physical memory
> used; 26.8 GB of 4.2 GB virtual memory used
> 2016-07-20 09:04:25,539 WARN
> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
> Process tree for container: container_e05_1468841606343_0128_01_000003 has
> processes older than 1 iteration running over the configured limit.
> Limit=2147483648, current usage = 2167832576
> 2016-07-20 09:04:25,547 WARN
> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
> Container [pid=2159,containerID=container_e05_1468841606343_0128_01_000003]
> is running beyond physical memory limits. Current usage: 2.0 GB of 2 GB
> physical memory used; 26.8 GB of 4.2 GB virtual memory used. Killing
> container.
> Dump of the process-tree for container_e05_1468841606343_0128_01_000003 :
> |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS)
> SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
> |- 2702 2164 2159 2159 (java) 137198 34820 8656748544 199840
> /usr/java/jdk1.8.0_72/jre/bin/java -Dgearpump.username=vcap
> -Dgearpump.applicationId=1 -Dgearpump.executorId=0
> -Dgearpump.master.starttime=2016-07-20-08-36
> -Dgearpump.log.application.dir=logs -Djava.net.preferIPv4Stack=true
> -Dgearpump.config.file=/tmp/gearpump6797336735445779094.conf -cp
> conf:pack/gearpump-2.11-0.8.0/conf:pack/gearpump-2.11-0.8.0/lib/netty-3.8.0.Final.jar:pack/gearpump-2.11-0.8.0/lib/gearpump-streaming_2.11-0.8.0.jar:pack/gearpump-2.11-0.8.0/lib/gearpump-shaded-guava-16.0.1.jar:pack/gearpump-2.11-0.8.0/lib/gearpump-shaded-metrics-graphite-3.0.2.jar:pack/gearpump-2.11-0.8.0/lib/scala-reflect-2.11.8.jar:pack/gearpump-2.11-0.8.0/lib/slf4j-log4j12-1.7.7.jar:pack/gearpump-2.11-0.8.0/lib/log4j-1.2.17.jar:pack/gearpump-2.11-0.8.0/lib/akka-slf4j_2.11-2.4.2.jar:pack/gearpump-2.11-0.8.0/lib/scala-parser-combinators_2.11-1.0.4.jar:pack/gearpump-2.11-0.8.0/lib/akka-actor_2.11-2.4.2.jar:pack/gearpump-2.11-0.8.0/lib/scala-java8-compat_2.11-0.7.0.jar:pack/gearpump-2.11-0.8.0/lib/slf4j-api-1.7.16.jar:pack/gearpump-2.11-0.8.0/lib/akka-agent_2.11-2.4.2.jar:pack/gearpump-2.11-0.8.0/lib/scala-library-2.11.8.jar:pack/gearpump-2.11-0.8.0/lib/commons-lang-2.6.jar:pack/gearpump-2.11-0.8.0/lib/akka-remote_2.11-2.4.2.jar:pack/gearpump-2.11-0.8.0/lib/config-1.3.0.jar:pack/gearpump-2.11-0.8.0/lib/akka-protobuf_2.11-2.4.2.jar:pack/gearpump-2.11-0.8.0/lib/gearpump-core_2.11-0.8.0.jar:pack/gearpump-2.11-0.8.0/lib/akka-kernel_2.11-2.4.2.jar:pack/gearpump-2.11-0.8.0/lib/scala-stm_2.11-0.7.jar:pack/gearpump-2.11-0.8.0/lib/uncommons-maths-1.2.2a.jar:pack/gearpump-2.11-0.8.0/lib/gearpump-shaded-gs-collections-6.2.0.jar:pack/gearpump-2.11-0.8.0/lib/gearpump-shaded-akka-kryo_2.11-0.3.2.jar:/etc/hadoop/conf.cloudera.YARN:/var/run/cloudera-scm-agent/process/195-yarn-NODEMANAGER:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/hadoop-aws-2.6.0-cdh5.5.1.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-format.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/hadoop-common-tests.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-scrooge_2.10.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-encoding.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-hadoop.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-format-javadoc.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/hadoop-common-2.6.0-cdh5.5.1.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-generator.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-thrift.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-jackson.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-hadoop-bundle.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-column.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-protobuf.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/hadoop-common.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-pig.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/hadoop-annotations.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-scala_2.10.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/hadoop-auth.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-test-hadoop2.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-common.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-pig-bundle.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/hadoop-annotations-2.6.0-cdh5.5.1.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-format-sources.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/hadoop-aws.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/hadoop-common-2.6.0-cdh5.5.1-tests.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/hadoop-nfs-2.6.0-cdh5.5.1.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/
> |- 2646 2164 2159 2159 (java) 4268 464 5847887872 70626
> /usr/java/jdk1.8.0_72/jre/bin/java -Dgearpump.username=vcap
> -Dgearpump.applicationId=1 -Dgearpump.executorId=-1
> -Dgearpump.master.starttime=2016-07-20-08-36
> -Dgearpump.log.application.dir=logs -Djava.net.preferIPv4Stack=true
> -Dgearpump.config.file=/tmp/gearpump941454978115257558.conf -cp
> conf:pack/gearpump-2.11-0.8.0/conf:pack/gearpump-2.11-0.8.0/lib/netty-3.8.0.Final.jar:pack/gearpump-2.11-0.8.0/lib/gearpump-streaming_2.11-0.8.0.jar:pack/gearpump-2.11-0.8.0/lib/gearpump-shaded-guava-16.0.1.jar:pack/gearpump-2.11-0.8.0/lib/gearpump-shaded-metrics-graphite-3.0.2.jar:pack/gearpump-2.11-0.8.0/lib/scala-reflect-2.11.8.jar:pack/gearpump-2.11-0.8.0/lib/slf4j-log4j12-1.7.7.jar:pack/gearpump-2.11-0.8.0/lib/log4j-1.2.17.jar:pack/gearpump-2.11-0.8.0/lib/akka-slf4j_2.11-2.4.2.jar:pack/gearpump-2.11-0.8.0/lib/scala-parser-combinators_2.11-1.0.4.jar:pack/gearpump-2.11-0.8.0/lib/akka-actor_2.11-2.4.2.jar:pack/gearpump-2.11-0.8.0/lib/scala-java8-compat_2.11-0.7.0.jar:pack/gearpump-2.11-0.8.0/lib/slf4j-api-1.7.16.jar:pack/gearpump-2.11-0.8.0/lib/akka-agent_2.11-2.4.2.jar:pack/gearpump-2.11-0.8.0/lib/scala-library-2.11.8.jar:pack/gearpump-2.11-0.8.0/lib/commons-lang-2.6.jar:pack/gearpump-2.11-0.8.0/lib/akka-remote_2.11-2.4.2.jar:pack/gearpump-2.11-0.8.0/lib/config-1.3.0.jar:pack/gearpump-2.11-0.8.0/lib/akka-protobuf_2.11-2.4.2.jar:pack/gearpump-2.11-0.8.0/lib/gearpump-core_2.11-0.8.0.jar:pack/gearpump-2.11-0.8.0/lib/akka-kernel_2.11-2.4.2.jar:pack/gearpump-2.11-0.8.0/lib/scala-stm_2.11-0.7.jar:pack/gearpump-2.11-0.8.0/lib/uncommons-maths-1.2.2a.jar:pack/gearpump-2.11-0.8.0/lib/gearpump-shaded-gs-collections-6.2.0.jar:pack/gearpump-2.11-0.8.0/lib/gearpump-shaded-akka-kryo_2.11-0.3.2.jar:/etc/hadoop/conf.cloudera.YARN:/var/run/cloudera-scm-agent/process/195-yarn-NODEMANAGER:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/hadoop-aws-2.6.0-cdh5.5.1.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-format.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/hadoop-common-tests.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-scrooge_2.10.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-encoding.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-hadoop.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-format-javadoc.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/hadoop-common-2.6.0-cdh5.5.1.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-generator.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-thrift.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-jackson.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-hadoop-bundle.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-column.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-protobuf.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/hadoop-common.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-pig.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/hadoop-annotations.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-scala_2.10.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/hadoop-auth.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-test-hadoop2.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-common.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-pig-bundle.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/hadoop-annotations-2.6.0-cdh5.5.1.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-format-sources.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/hadoop-aws.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/hadoop-common-2.6.0-cdh5.5.1-tests.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/hadoop-nfs-2.6.0-cdh5.5.1.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/
> |- 2159 2157 2159 2159 (bash) 0 0 108654592 301 /bin/bash -c
> /usr/java/jdk1.8.0_72/bin/java -Xmx512m -cp
> conf:pack/gearpump-2.11-0.8.0/conf:pack/gearpump-2.11-0.8.0/lib/daemon/*:pack/gearpump-2.11-0.8.0/lib/*:/etc/hadoop/conf.cloudera.YARN:/var/run/cloudera-scm-agent/process/195-yarn-NODEMANAGER:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/*:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/lib/*:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop-hdfs/*:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop-hdfs/lib/*:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop-yarn/*:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop-yarn/lib/*
> -Dgearpump.cluster.masters.0=cdh-worker-2.node.envname.consul:54946
> -Dgearpump.log.daemon.dir=/var/log/hadoop-yarn/container/application_1468841606343_0128/container_e05_1468841606343_0128_01_000003
>
> -Dgearpump.worker-resource-manager-container-id=container_e05_1468841606343_0128_01_000003
>
> -Dgearpump.home=/data1/hadoop/yarn/nm/usercache/cf/appcache/application_1468841606343_0128/container_e05_1468841606343_0128_01_000003/pack/gearpump-2.11-0.8.0
>
> -Dgearpump.log.application.dir=/var/log/hadoop-yarn/container/application_1468841606343_0128/container_e05_1468841606343_0128_01_000003
> -Dgearpump.hostname=cdh-worker-2.node.envname.consul
> io.gearpump.cluster.main.Worker 2>&1 | /usr/bin/tee -a
> /var/log/hadoop-yarn/container/application_1468841606343_0128/container_e05_1468841606343_0128_01_000003/stderr
>
> |- 2165 2159 2159 2159 (tee) 38 396 103337984 164 /usr/bin/tee -a
> /var/log/hadoop-yarn/container/application_1468841606343_0128/container_e05_1468841606343_0128_01_000003/stderr
>
> |- 2164 2159 2159 2159 (java) 4915 815 2367729664 78249
> /usr/java/jdk1.8.0_72/bin/java -Xmx512m -cp
> conf:pack/gearpump-2.11-0.8.0/conf:pack/gearpump-2.11-0.8.0/lib/daemon/*:pack/gearpump-2.11-0.8.0/lib/*:/etc/hadoop/conf.cloudera.YARN:/var/run/cloudera-scm-agent/process/195-yarn-NODEMANAGER:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/*:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/lib/*:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop-hdfs/*:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop-hdfs/lib/*:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop-yarn/*:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop-yarn/lib/*
> -Dgearpump.cluster.masters.0=cdh-worker-2.node.envname.consul:54946
> -Dgearpump.log.daemon.dir=/var/log/hadoop-yarn/container/application_1468841606343_0128/container_e05_1468841606343_0128_01_000003
>
> -Dgearpump.worker-resource-manager-container-id=container_e05_1468841606343_0128_01_000003
>
> -Dgearpump.home=/data1/hadoop/yarn/nm/usercache/cf/appcache/application_1468841606343_0128/container_e05_1468841606343_0128_01_000003/pack/gearpump-2.11-0.8.0
>
> -Dgearpump.log.application.dir=/var/log/hadoop-yarn/container/application_1468841606343_0128/container_e05_1468841606343_0128_01_000003
> -Dgearpump.hostname=cdh-worker-2.node.envname.consul
> io.gearpump.cluster.main.Worker
> |- 6417 2164 2159 2159 (java) 3900 1182 5893246976 125356
> /usr/java/jdk1.8.0_72/jre/bin/java -Dgearpump.username=vcap
> -Dgearpump.applicationId=2 -Dgearpump.executorId=0
> -Dgearpump.master.starttime=2016-07-20-08-36
> -Dgearpump.log.application.dir=logs -Djava.net.preferIPv4Stack=true
> -Dgearpump.config.file=/tmp/gearpump7648852439078424047.conf -cp
> conf:pack/gearpump-2.11-0.8.0/conf:pack/gearpump-2.11-0.8.0/lib/netty-3.8.0.Final.jar:pack/gearpump-2.11-0.8.0/lib/gearpump-streaming_2.11-0.8.0.jar:pack/gearpump-2.11-0.8.0/lib/gearpump-shaded-guava-16.0.1.jar:pack/gearpump-2.11-0.8.0/lib/gearpump-shaded-metrics-graphite-3.0.2.jar:pack/gearpump-2.11-0.8.0/lib/scala-reflect-2.11.8.jar:pack/gearpump-2.11-0.8.0/lib/slf4j-log4j12-1.7.7.jar:pack/gearpump-2.11-0.8.0/lib/log4j-1.2.17.jar:pack/gearpump-2.11-0.8.0/lib/akka-slf4j_2.11-2.4.2.jar:pack/gearpump-2.11-0.8.0/lib/scala-parser-combinators_2.11-1.0.4.jar:pack/gearpump-2.11-0.8.0/lib/akka-actor_2.11-2.4.2.jar:pack/gearpump-2.11-0.8.0/lib/scala-java8-compat_2.11-0.7.0.jar:pack/gearpump-2.11-0.8.0/lib/slf4j-api-1.7.16.jar:pack/gearpump-2.11-0.8.0/lib/akka-agent_2.11-2.4.2.jar:pack/gearpump-2.11-0.8.0/lib/scala-library-2.11.8.jar:pack/gearpump-2.11-0.8.0/lib/commons-lang-2.6.jar:pack/gearpump-2.11-0.8.0/lib/akka-remote_2.11-2.4.2.jar:pack/gearpump-2.11-0.8.0/lib/config-1.3.0.jar:pack/gearpump-2.11-0.8.0/lib/akka-protobuf_2.11-2.4.2.jar:pack/gearpump-2.11-0.8.0/lib/gearpump-core_2.11-0.8.0.jar:pack/gearpump-2.11-0.8.0/lib/akka-kernel_2.11-2.4.2.jar:pack/gearpump-2.11-0.8.0/lib/scala-stm_2.11-0.7.jar:pack/gearpump-2.11-0.8.0/lib/uncommons-maths-1.2.2a.jar:pack/gearpump-2.11-0.8.0/lib/gearpump-shaded-gs-collections-6.2.0.jar:pack/gearpump-2.11-0.8.0/lib/gearpump-shaded-akka-kryo_2.11-0.3.2.jar:/etc/hadoop/conf.cloudera.YARN:/var/run/cloudera-scm-agent/process/195-yarn-NODEMANAGER:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/hadoop-aws-2.6.0-cdh5.5.1.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-format.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/hadoop-common-tests.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-scrooge_2.10.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-encoding.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-hadoop.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-format-javadoc.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/hadoop-common-2.6.0-cdh5.5.1.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-generator.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-thrift.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-jackson.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-hadoop-bundle.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-column.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-protobuf.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/hadoop-common.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-pig.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/hadoop-annotations.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-scala_2.10.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/hadoop-auth.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-test-hadoop2.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-common.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-pig-bundle.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/hadoop-annotations-2.6.0-cdh5.5.1.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-format-sources.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/hadoop-aws.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/hadoop-common-2.6.0-cdh5.5.1-tests.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/hadoop-nfs-2.6.0-cdh5.5.1.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/
> |- 6360 2164 2159 2159 (java) 966 58 5845979136 54720
> /usr/java/jdk1.8.0_72/jre/bin/java -Dgearpump.username=vcap
> -Dgearpump.applicationId=2 -Dgearpump.executorId=-1
> -Dgearpump.master.starttime=2016-07-20-08-36
> -Dgearpump.log.application.dir=logs -Djava.net.preferIPv4Stack=true
> -Dgearpump.config.file=/tmp/gearpump7311147404608349128.conf -cp
> conf:pack/gearpump-2.11-0.8.0/conf:pack/gearpump-2.11-0.8.0/lib/netty-3.8.0.Final.jar:pack/gearpump-2.11-0.8.0/lib/gearpump-streaming_2.11-0.8.0.jar:pack/gearpump-2.11-0.8.0/lib/gearpump-shaded-guava-16.0.1.jar:pack/gearpump-2.11-0.8.0/lib/gearpump-shaded-metrics-graphite-3.0.2.jar:pack/gearpump-2.11-0.8.0/lib/scala-reflect-2.11.8.jar:pack/gearpump-2.11-0.8.0/lib/slf4j-log4j12-1.7.7.jar:pack/gearpump-2.11-0.8.0/lib/log4j-1.2.17.jar:pack/gearpump-2.11-0.8.0/lib/akka-slf4j_2.11-2.4.2.jar:pack/gearpump-2.11-0.8.0/lib/scala-parser-combinators_2.11-1.0.4.jar:pack/gearpump-2.11-0.8.0/lib/akka-actor_2.11-2.4.2.jar:pack/gearpump-2.11-0.8.0/lib/scala-java8-compat_2.11-0.7.0.jar:pack/gearpump-2.11-0.8.0/lib/slf4j-api-1.7.16.jar:pack/gearpump-2.11-0.8.0/lib/akka-agent_2.11-2.4.2.jar:pack/gearpump-2.11-0.8.0/lib/scala-library-2.11.8.jar:pack/gearpump-2.11-0.8.0/lib/commons-lang-2.6.jar:pack/gearpump-2.11-0.8.0/lib/akka-remote_2.11-2.4.2.jar:pack/gearpump-2.11-0.8.0/lib/config-1.3.0.jar:pack/gearpump-2.11-0.8.0/lib/akka-protobuf_2.11-2.4.2.jar:pack/gearpump-2.11-0.8.0/lib/gearpump-core_2.11-0.8.0.jar:pack/gearpump-2.11-0.8.0/lib/akka-kernel_2.11-2.4.2.jar:pack/gearpump-2.11-0.8.0/lib/scala-stm_2.11-0.7.jar:pack/gearpump-2.11-0.8.0/lib/uncommons-maths-1.2.2a.jar:pack/gearpump-2.11-0.8.0/lib/gearpump-shaded-gs-collections-6.2.0.jar:pack/gearpump-2.11-0.8.0/lib/gearpump-shaded-akka-kryo_2.11-0.3.2.jar:/etc/hadoop/conf.cloudera.YARN:/var/run/cloudera-scm-agent/process/195-yarn-NODEMANAGER:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/hadoop-aws-2.6.0-cdh5.5.1.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-format.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/hadoop-common-tests.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-scrooge_2.10.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-encoding.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-hadoop.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-format-javadoc.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/hadoop-common-2.6.0-cdh5.5.1.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-generator.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-thrift.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-jackson.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-hadoop-bundle.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-column.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-protobuf.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/hadoop-common.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-pig.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/hadoop-annotations.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-scala_2.10.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/hadoop-auth.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-test-hadoop2.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-common.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-pig-bundle.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/hadoop-annotations-2.6.0-cdh5.5.1.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-format-sources.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/hadoop-aws.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/hadoop-common-2.6.0-cdh5.5.1-tests.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/hadoop-nfs-2.6.0-cdh5.5.1.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib
> 2016-07-20 09:04:25,548 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container:
> Container container_e05_1468841606343_0128_01_000003 transitioned from
> RUNNING to KILLING
> 2016-07-20 09:04:25,548 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch:
> Cleaning up container container_e05_1468841606343_0128_01_000003
> 2016-07-20 09:04:25,548 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
> Removed ProcessTree with root 2159
> 2016-07-20 09:04:25,584 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
> Memory usage of ProcessTree 2116 for container-id
> container_e05_1468841606343_0128_01_000002: 348.0 MB of 1 GB physical memory
> used; 2.4 GB of 2.1 GB virtual memory used
> 2016-07-20 09:04:25,621 WARN
> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Exit code
> from container container_e05_1468841606343_0128_01_000003 is : 143
> 2016-07-20 09:04:25,655 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container:
> Container container_e05_1468841606343_0128_01_000003 transitioned from
> KILLING to CONTAINER_CLEANEDUP_AFTER_KILL
> 2016-07-20 09:04:25,655 INFO
> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting
> absolute path :
> /data1/hadoop/yarn/nm/usercache/cf/appcache/application_1468841606343_0128/container_e05_1468841606343_0128_01_000003
> 2016-07-20 09:04:25,656 INFO
> org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=cf
> OPERATION=Container Finished - Killed TARGET=ContainerImpl
> RESULT=SUCCESS APPID=application_1468841606343_0128
> CONTAINERID=container_e05_1468841606343_0128_01_000003
> 2016-07-20 09:04:25,656 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container:
> Container container_e05_1468841606343_0128_01_000003 transitioned from
> CONTAINER_CLEANEDUP_AFTER_KILL to DONE
> 2016-07-20 09:04:25,656 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application:
> Removing container_e05_1468841606343_0128_01_000003 from application
> application_1468841606343_0128
> 2016-07-20 09:04:25,656 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl:
> Considering container container_e05_1468841606343_0128_01_000003 for
> log-aggregation
> 2016-07-20 09:04:25,656 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got
> event CONTAINER_STOP for appId application_1468841606343_0128
> 2016-07-20 09:04:25,656 INFO
> org.apache.spark.network.yarn.YarnShuffleService: Stopping container
> container_e05_1468841606343_0128_01_000003
> 2016-07-20 09:04:27,703 INFO
> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Removed
> completed containers from NM context:
> [container_e05_1468841606343_0128_01_000003]
> 2016-07-20 09:04:28,584 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
> Stopping resource-monitoring for container_e05_1468841606343_0128_01_000003
> 2016-07-20 09:04:28,594 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
> Memory usage of ProcessTree 27737 for container-id
> container_e05_1468841606343_0021_01_000002: 285.7 MB of 1 GB physical memory
> used; 2.4 GB of 2.1 GB virtual memory used
> 2016-07-20 09:04:28,603 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
> Memory usage of ProcessTree 2046 for container-id
> container_e05_1468841606343_0128_01_000001: 356.7 MB of 1 GB physical memory
> used; 2.5 GB of 2.1 GB virtual memory used
> 2016-07-20 09:04:28,613 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
> Memory usage of ProcessTree 2116 for container-id
> container_e05_1468841606343_0128_01_000002: 348.1 MB of 1 GB physical memory
> used; 2.4 GB of 2.1 GB virtual memory used
> 2016-07-20 09:04:31,623 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
> Memory usage of ProcessTree 27737 for container-id
> container_e05_1468841606343_0021_01_000002: 285.7 MB of 1 GB physical memory
> used; 2.4 GB of 2.1 GB virtual memory used
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
