You can also change the default value of yarn.nodemanager.vmem-pmem-ratio On Sat, Jun 20, 2015 at 12:39 AM, Drake민영근 <[email protected]> wrote:
> Hi, > > You should disable vmem check. See this: > http://blog.cloudera.com/blog/2014/04/apache-hadoop-yarn-avoiding-6-time-consuming-gotchas/ > > > Thanks. > > 2015년 6월 17일 수요일, Naganarasimha G R (Naga)<[email protected]>님이 > 작성한 메시지: > > Hi, >> From the logs its pretty clear its due to >> *"Current usage: 576.2 MB of 2 GB physical memory used; 4.2 GB of 4.2 GB >> virtual memory used. Killing container."* >> Please increase the value yarn.nodemanager.vmem-pmem-ratio from the >> default value 2 to something like 4 or 8 based on ur app and system. >> >> + Naga >> ------------------------------ >> *From:* Arbi Akhina [[email protected]] >> *Sent:* Wednesday, June 17, 2015 17:19 >> *To:* [email protected] >> *Subject:* YARN container killed as running beyond memory limits >> >> Hi, I've a YARN application that submits containers. In the >> AplicationMaster logs I see that the container is killed. Here is the logs: >> >> Jun 17, 2015 1:31:27 PM com.heavenize.modules.RMCallbackHandler >> onContainersCompleted >> INFO: container 'container_1434471275225_0007_01_000002' status is >> ContainerStatus: [ContainerId: container_1434471275225_0007_01_000002, >> State: COMPLETE, Diagnostics: Container >> [pid=4069,containerID=container_1434471275225_0007_01_000002] is running >> beyond virtual memory limits. Current usage: 576.2 MB of 2 GB physical >> memory used; 4.2 GB of 4.2 GB virtual memory used. Killing container. >> Dump of the process-tree for container_1434471275225_0007_01_000002 : >> |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) >> SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE >> |- 4094 4093 4069 4069 (java) 2932 94 2916065280 122804 >> /usr/lib/jvm/java-7-openjdk-amd64/bin/java -Xms512m -Xmx2048m >> -XX:MaxPermSize=250m -XX:+UseConcMarkSweepGC >> -Dosmoze.path=/tmp/hadoop-hadoop/nm-local-dir/usercache/hadoop/appcache/container_1434471275225_0007_01_000002/Osmoze >> -Dspring.profiles.active=webServer -jar >> /tmp/hadoop-hadoop/nm-local-dir/usercache/hadoop/appcache/container_1434471275225_0007_01_000002/heavenize-modules.jar >> |- 4093 4073 4069 4069 (sh) 0 0 4550656 164 /bin/sh >> /tmp/hadoop-hadoop/nm-local-dir/usercache/hadoop/appcache/container_1434471275225_0007_01_000002/startup.sh >> |- 4073 4069 4069 4069 (java) 249 34 1577267200 24239 >> /usr/lib/jvm/java-7-openjdk-amd64/bin/java >> com.heavenize.yarn.task.ModulesManager -containerId >> container_1434471275225_0007_01_000002 -port 5369 -exe >> hdfs://hadoop-server/user/hadoop/heavenize/heavenize-modules.jar -conf >> hdfs://hadoop-server/user/hadoop/heavenize/config.zip >> |- 4069 1884 4069 4069 (bash) 0 0 12730368 304 /bin/bash -c >> /usr/lib/jvm/java-7-openjdk-amd64/bin/java >> com.heavenize.yarn.task.ModulesManager -containerId >> container_1434471275225_0007_01_000002 -port 5369 -exe >> hdfs://hadoop-server/user/hadoop/heavenize/heavenize-modules.jar -conf >> hdfs://hadoop-server/user/hadoop/heavenize/config.zip 1> >> /usr/local/hadoop/logs/userlogs/application_1434471275225_0007/container_1434471275225_0007_01_000002/stdout >> 2> >> /usr/local/hadoop/logs/userlogs/application_1434471275225_0007/container_1434471275225_0007_01_000002/stderr >> >> >> I don't see any memory excess, any idea where this error comes from? >> There is no errors in the container, it just stop logging as a result >> of being killed. >> > > > -- > Drake 민영근 Ph.D > kt NexR > >
