yarn kill container due to running beyond physical memory limits [ How can i debug memory issue ]

[prateekarora]

Hi

I am running flink 1.0.2 with Yarn .

After running application for some time , Yarn kill my container due to
running beyond physical memory limits .

how can i debug memory issue ?

below are the logs :

Container container_1463184272818_0165_01_000012 is completed with
diagnostics: Container
[pid=19349,containerID=container_1463184272818_0165_01_000012] is running
beyond physical memory limits. Current usage: 6.0 GB of 6 GB physical memory
used; 9.1 GB of 12.6 GB virtual memory used. Killing container.

Dump of the process-tree for container_1463184272818_0165_01_000012 :
        |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS)
SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
        |- 19356 19349 19349 19349 (java) 39350 9110 9711140864 1581168
/usr/lib/jvm/java-7-oracle-cloudera/bin/java -Xms4608m -Xmx4608m
-XX:MaxDirectMemorySize=4608m
-Djava.library.path=/home/nativelibraries/native_lib/
-Dlog.file=/var/log/hadoop-yarn/container/application_1463184272818_0165/container_1463184272818_0165_01_000012/taskmanager.log
-Dlogback.configurationFile=file:logback.xml
-Dlog4j.configuration=file:log4j.properties
org.apache.flink.yarn.YarnTaskManagerRunner --configDir .
        |- 19349 19345 19349 19349 (bash) 0 0 11456512 359 /bin/bash -c
/usr/lib/jvm/java-7-oracle-cloudera/bin/java -Xms4608m -Xmx4608m
-XX:MaxDirectMemorySize=4608m
-Djava.library.path=/home/nativelibraries/native_lib/
-Dlog.file=/var/log/hadoop-yarn/container/application_1463184272818_0165/container_1463184272818_0165_01_000012/taskmanager.log
-Dlogback.configurationFile=file:logback.xml
-Dlog4j.configuration=file:log4j.properties
org.apache.flink.yarn.YarnTaskManagerRunner --configDir . 1>
/var/log/hadoop-yarn/container/application_1463184272818_0165/container_1463184272818_0165_01_000012/taskmanager.out
2>
/var/log/hadoop-yarn/container/application_1463184272818_0165/container_1463184272818_0165_01_000012/taskmanager.err

Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143



--
View this message in context: 
http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/yarn-kill-container-due-to-running-beyond-physical-memory-limits-How-can-i-debug-memory-issue-tp7296.html
Sent from the Apache Flink User Mailing List archive. mailing list archive at 
Nabble.com.