Re: Running beyond physical memory limits
The setting to increase is spark.yarn.executor.memoryOverhead On Wed, Apr 15, 2015 at 6:35 AM, Brahma Reddy Battula brahmareddy.batt...@huawei.com wrote: Hello Sean Owen, Thanks for your reply..Ill increase overhead memory and check it.. Bytheway ,Any difference between 1.1 and 1.2 makes, this issue..? Since It was passing spark 1.1 and throwing following error in 1.2...( this makes me doubt full) Thanks Regards Brahma Reddy Battula From: Sean Owen [so...@cloudera.com] Sent: Wednesday, April 15, 2015 2:49 PM To: Brahma Reddy Battula Cc: Akhil Das; user@spark.apache.org Subject: Re: Running beyond physical memory limits This is not related to executor memory, but the extra overhead subtracted from the executor's size in order to avoid using more than the physical memory that YARN allows. That is, if you declare a 32G executor YARN lets you use 32G physical memory but your JVM heap must be significantly less than 32G max. This is the overhead factor that is subtracted for you, and it seems to need to be bigger in your case. On Wed, Apr 15, 2015 at 10:16 AM, Brahma Reddy Battula brahmareddy.batt...@huawei.com wrote: Thanks lot for your reply.. There is no issue with spark1.1..Following issue came when I upgrade to spark2.0...Hence I did not decrease spark.executor.memory... I mean to say, used same config for spark1.1 and spark1.2.. Is there any issue with spark1.2..? Or Yarn will lead this..? And why executor will not release memory, if there are tasks running..? Thanks Regards Brahma Reddy Battula From: Akhil Das [ak...@sigmoidanalytics.com] Sent: Wednesday, April 15, 2015 2:35 PM To: Brahma Reddy Battula Cc: user@spark.apache.org Subject: Re: Running beyond physical memory limits Did you try reducing your spark.executor.memory? Thanks Best Regards On Wed, Apr 15, 2015 at 2:29 PM, Brahma Reddy Battula brahmareddy.batt...@huawei.com wrote: Hello Sparkers I am newbie to spark and need help.. We are using spark 1.2, we are getting the following error and executor is getting killed..I seen SPARK-1930 and it should be in 1.2.. Any pointer to following error, like what might lead this error.. 2015-04-15 11:55:39,697 | WARN | Container Monitor | Container [pid=126843,containerID=container_1429065217137_0012_01_-411041790] is running beyond physical memory limits. Current usage: 26.0 GB of 26 GB physical memory used; 26.7 GB of 260 GB virtual memory used. Killing container. Dump of the process-tree for container_1429065217137_0012_01_-411041790 : |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE |- 126872 126843 126843 126843 (java) 2049457 22816 28673892352 6824864 /opt/huawei/Bigdata/jdk1.7.0_76//bin/java -server -XX:OnOutOfMemoryError=kill %p -Xms24576m -Xmx24576m -Dlog4j.configuration=file:/opt/huawei/Bigdata/DataSight_FM_BasePlatform_V100R001C00_Spark/spark/conf/log4j-executor.properties -Djava.library.path=/opt/huawei/Bigdata/DataSight_FM_BasePlatform_V100R001C00_Hadoop//hadoop/lib/native -Djava.io.tmpdir=/srv/BigData/hadoop/data4/nm/localdir/usercache/ossuser/appcache/application_1429065217137_0012/container_1429065217137_0012_01_-411041790/tmp -Dspark.driver.port=23204 -Dspark.random.port.max=23999 -Dspark.akka.threads=32 -Dspark.akka.frameSize=10 -Dspark.akka.timeout=100 -Dspark.ui.port=23000 -Dspark.random.port.min=23000 -Dspark.yarn.app.container.log.dir=/srv/BigData/hadoop/data5/nm/containerlogs/application_1429065217137_0012/container_1429065217137_0012_01_-411041790 org.apache.spark.executor.CoarseGrainedExecutorBackend akka.tcp://sparkDriver@172.57.1.61:23204/user/CoarseGrainedScheduler 3 hadoopc1h11 10 application_1429065217137_0012 |- 126843 76960 126843 126843 (bash) 0 0 11603968 331 /bin/bash -c /opt/huawei/Bigdata/jdk1.7.0_76//bin/java -server -XX:OnOutOfMemoryError='kill %p' -Xms24576m -Xmx24576m -Dlog4j.configuration=file:/opt/huawei/Bigdata/DataSight_FM_BasePlatform_V100R001C00_Spark/spark/conf/log4j-executor.properties -Djava.library.path=/opt/huawei/Bigdata/DataSight_FM_BasePlatform_V100R001C00_Hadoop//hadoop/lib/native -Djava.io.tmpdir=/srv/BigData/hadoop/data4/nm/localdir/usercache/ossuser/appcache/application_1429065217137_0012/container_1429065217137_0012_01_-411041790/tmp '-Dspark.driver.port=23204' '-Dspark.random.port.max=23999' '-Dspark.akka.threads=32' '-Dspark.akka.frameSize=10' '-Dspark.akka.timeout=100' '-Dspark.ui.port=23000' '-Dspark.random.port.min=23000' -Dspark.yarn.app.container.log.dir=/srv/BigData/hadoop/data5/nm/containerlogs/application_1429065217137_0012/container_1429065217137_0012_01_-411041790 org.apache.spark.executor.CoarseGrainedExecutorBackend akka.tcp://sparkDriver@172.57.1.61:23204/user
RE: Running beyond physical memory limits
Thanks lot for your reply.. There is no issue with spark1.1..Following issue came when I upgrade to spark2.0...Hence I did not decrease spark.executor.memory... I mean to say, used same config for spark1.1 and spark1.2.. Is there any issue with spark1.2..? Or Yarn will lead this..? And why executor will not release memory, if there are tasks running..? Thanks Regards Brahma Reddy Battula From: Akhil Das [ak...@sigmoidanalytics.com] Sent: Wednesday, April 15, 2015 2:35 PM To: Brahma Reddy Battula Cc: user@spark.apache.org Subject: Re: Running beyond physical memory limits Did you try reducing your spark.executor.memory? Thanks Best Regards On Wed, Apr 15, 2015 at 2:29 PM, Brahma Reddy Battula brahmareddy.batt...@huawei.commailto:brahmareddy.batt...@huawei.com wrote: Hello Sparkers I am newbie to spark and need help.. We are using spark 1.2, we are getting the following error and executor is getting killed..I seen SPARK-1930 and it should be in 1.2.. Any pointer to following error, like what might lead this error.. 2015-04-15 11:55:39,697 | WARN | Container Monitor | Container [pid=126843,containerID=container_1429065217137_0012_01_-411041790] is running beyond physical memory limits. Current usage: 26.0 GB of 26 GB physical memory used; 26.7 GB of 260 GB virtual memory used. Killing container. Dump of the process-tree for container_1429065217137_0012_01_-411041790 : |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE |- 126872 126843 126843 126843 (java) 2049457 22816 28673892352 6824864 /opt/huawei/Bigdata/jdk1.7.0_76//bin/java -server -XX:OnOutOfMemoryError=kill %p -Xms24576m -Xmx24576m -Dlog4j.configuration=file:/opt/huawei/Bigdata/DataSight_FM_BasePlatform_V100R001C00_Spark/spark/conf/log4j-executor.properties -Djava.library.path=/opt/huawei/Bigdata/DataSight_FM_BasePlatform_V100R001C00_Hadoop//hadoop/lib/native -Djava.io.tmpdir=/srv/BigData/hadoop/data4/nm/localdir/usercache/ossuser/appcache/application_1429065217137_0012/container_1429065217137_0012_01_-411041790/tmp -Dspark.driver.port=23204 -Dspark.random.port.max=23999 -Dspark.akka.threads=32 -Dspark.akka.frameSize=10 -Dspark.akka.timeout=100 -Dspark.ui.port=23000 -Dspark.random.port.min=23000 -Dspark.yarn.app.container.log.dir=/srv/BigData/hadoop/data5/nm/containerlogs/application_1429065217137_0012/container_1429065217137_0012_01_-411041790 org.apache.spark.executor.CoarseGrainedExecutorBackend akka.tcp://sparkDriver@172.57.1.61:23204/user/CoarseGrainedSchedulerhttp://sparkDriver@172.57.1.61:23204/user/CoarseGrainedScheduler 3 hadoopc1h11 10 application_1429065217137_0012 |- 126843 76960 126843 126843 (bash) 0 0 11603968 331 /bin/bash -c /opt/huawei/Bigdata/jdk1.7.0_76//bin/java -server -XX:OnOutOfMemoryError='kill %p' -Xms24576m -Xmx24576m -Dlog4j.configuration=file:/opt/huawei/Bigdata/DataSight_FM_BasePlatform_V100R001C00_Spark/spark/conf/log4j-executor.properties -Djava.library.path=/opt/huawei/Bigdata/DataSight_FM_BasePlatform_V100R001C00_Hadoop//hadoop/lib/native -Djava.io.tmpdir=/srv/BigData/hadoop/data4/nm/localdir/usercache/ossuser/appcache/application_1429065217137_0012/container_1429065217137_0012_01_-411041790/tmp '-Dspark.driver.port=23204' '-Dspark.random.port.max=23999' '-Dspark.akka.threads=32' '-Dspark.akka.frameSize=10' '-Dspark.akka.timeout=100' '-Dspark.ui.port=23000' '-Dspark.random.port.min=23000' -Dspark.yarn.app.container.log.dir=/srv/BigData/hadoop/data5/nm/containerlogs/application_1429065217137_0012/container_1429065217137_0012_01_-411041790 org.apache.spark.executor.CoarseGrainedExecutorBackend akka.tcp://sparkDriver@172.57.1.61:23204/user/CoarseGrainedSchedulerhttp://sparkDriver@172.57.1.61:23204/user/CoarseGrainedScheduler 3 hadoopc1h11 10 application_1429065217137_0012 1 /srv/BigData/hadoop/data5/nm/containerlogs/application_1429065217137_0012/container_1429065217137_0012_01_-411041790/stdout 2 /srv/BigData/hadoop/data5/nm/containerlogs/application_1429065217137_0012/container_1429065217137_0012_01_-411041790/stderr | org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl$MonitoringThread.run(ContainersMonitorImpl.java:447) And some doubts 1) why executor will not release memory, if there are tasks running..? 2) is there issue from hadoop which will lead this error..? Any help , will be appreciated... Thanks Regards Brahma Reddy Battula
Re: Running beyond physical memory limits
This is not related to executor memory, but the extra overhead subtracted from the executor's size in order to avoid using more than the physical memory that YARN allows. That is, if you declare a 32G executor YARN lets you use 32G physical memory but your JVM heap must be significantly less than 32G max. This is the overhead factor that is subtracted for you, and it seems to need to be bigger in your case. On Wed, Apr 15, 2015 at 10:16 AM, Brahma Reddy Battula brahmareddy.batt...@huawei.com wrote: Thanks lot for your reply.. There is no issue with spark1.1..Following issue came when I upgrade to spark2.0...Hence I did not decrease spark.executor.memory... I mean to say, used same config for spark1.1 and spark1.2.. Is there any issue with spark1.2..? Or Yarn will lead this..? And why executor will not release memory, if there are tasks running..? Thanks Regards Brahma Reddy Battula From: Akhil Das [ak...@sigmoidanalytics.com] Sent: Wednesday, April 15, 2015 2:35 PM To: Brahma Reddy Battula Cc: user@spark.apache.org Subject: Re: Running beyond physical memory limits Did you try reducing your spark.executor.memory? Thanks Best Regards On Wed, Apr 15, 2015 at 2:29 PM, Brahma Reddy Battula brahmareddy.batt...@huawei.com wrote: Hello Sparkers I am newbie to spark and need help.. We are using spark 1.2, we are getting the following error and executor is getting killed..I seen SPARK-1930 and it should be in 1.2.. Any pointer to following error, like what might lead this error.. 2015-04-15 11:55:39,697 | WARN | Container Monitor | Container [pid=126843,containerID=container_1429065217137_0012_01_-411041790] is running beyond physical memory limits. Current usage: 26.0 GB of 26 GB physical memory used; 26.7 GB of 260 GB virtual memory used. Killing container. Dump of the process-tree for container_1429065217137_0012_01_-411041790 : |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE |- 126872 126843 126843 126843 (java) 2049457 22816 28673892352 6824864 /opt/huawei/Bigdata/jdk1.7.0_76//bin/java -server -XX:OnOutOfMemoryError=kill %p -Xms24576m -Xmx24576m -Dlog4j.configuration=file:/opt/huawei/Bigdata/DataSight_FM_BasePlatform_V100R001C00_Spark/spark/conf/log4j-executor.properties -Djava.library.path=/opt/huawei/Bigdata/DataSight_FM_BasePlatform_V100R001C00_Hadoop//hadoop/lib/native -Djava.io.tmpdir=/srv/BigData/hadoop/data4/nm/localdir/usercache/ossuser/appcache/application_1429065217137_0012/container_1429065217137_0012_01_-411041790/tmp -Dspark.driver.port=23204 -Dspark.random.port.max=23999 -Dspark.akka.threads=32 -Dspark.akka.frameSize=10 -Dspark.akka.timeout=100 -Dspark.ui.port=23000 -Dspark.random.port.min=23000 -Dspark.yarn.app.container.log.dir=/srv/BigData/hadoop/data5/nm/containerlogs/application_1429065217137_0012/container_1429065217137_0012_01_-411041790 org.apache.spark.executor.CoarseGrainedExecutorBackend akka.tcp://sparkDriver@172.57.1.61:23204/user/CoarseGrainedScheduler 3 hadoopc1h11 10 application_1429065217137_0012 |- 126843 76960 126843 126843 (bash) 0 0 11603968 331 /bin/bash -c /opt/huawei/Bigdata/jdk1.7.0_76//bin/java -server -XX:OnOutOfMemoryError='kill %p' -Xms24576m -Xmx24576m -Dlog4j.configuration=file:/opt/huawei/Bigdata/DataSight_FM_BasePlatform_V100R001C00_Spark/spark/conf/log4j-executor.properties -Djava.library.path=/opt/huawei/Bigdata/DataSight_FM_BasePlatform_V100R001C00_Hadoop//hadoop/lib/native -Djava.io.tmpdir=/srv/BigData/hadoop/data4/nm/localdir/usercache/ossuser/appcache/application_1429065217137_0012/container_1429065217137_0012_01_-411041790/tmp '-Dspark.driver.port=23204' '-Dspark.random.port.max=23999' '-Dspark.akka.threads=32' '-Dspark.akka.frameSize=10' '-Dspark.akka.timeout=100' '-Dspark.ui.port=23000' '-Dspark.random.port.min=23000' -Dspark.yarn.app.container.log.dir=/srv/BigData/hadoop/data5/nm/containerlogs/application_1429065217137_0012/container_1429065217137_0012_01_-411041790 org.apache.spark.executor.CoarseGrainedExecutorBackend akka.tcp://sparkDriver@172.57.1.61:23204/user/CoarseGrainedScheduler 3 hadoopc1h11 10 application_1429065217137_0012 1 /srv/BigData/hadoop/data5/nm/containerlogs/application_1429065217137_0012/container_1429065217137_0012_01_-411041790/stdout 2 /srv/BigData/hadoop/data5/nm/containerlogs/application_1429065217137_0012/container_1429065217137_0012_01_-411041790/stderr | org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl$MonitoringThread.run(ContainersMonitorImpl.java:447) And some doubts 1) why executor will not release memory, if there are tasks running..? 2) is there issue from hadoop which will lead this error..? Any help , will be appreciated... Thanks Regards Brahma Reddy Battula
Running beyond physical memory limits
Hello Sparkers I am newbie to spark and need help.. We are using spark 1.2, we are getting the following error and executor is getting killed..I seen SPARK-1930 and it should be in 1.2.. Any pointer to following error, like what might lead this error.. 2015-04-15 11:55:39,697 | WARN | Container Monitor | Container [pid=126843,containerID=container_1429065217137_0012_01_-411041790] is running beyond physical memory limits. Current usage: 26.0 GB of 26 GB physical memory used; 26.7 GB of 260 GB virtual memory used. Killing container. Dump of the process-tree for container_1429065217137_0012_01_-411041790 : |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE |- 126872 126843 126843 126843 (java) 2049457 22816 28673892352 6824864 /opt/huawei/Bigdata/jdk1.7.0_76//bin/java -server -XX:OnOutOfMemoryError=kill %p -Xms24576m -Xmx24576m -Dlog4j.configuration=file:/opt/huawei/Bigdata/DataSight_FM_BasePlatform_V100R001C00_Spark/spark/conf/log4j-executor.properties -Djava.library.path=/opt/huawei/Bigdata/DataSight_FM_BasePlatform_V100R001C00_Hadoop//hadoop/lib/native -Djava.io.tmpdir=/srv/BigData/hadoop/data4/nm/localdir/usercache/ossuser/appcache/application_1429065217137_0012/container_1429065217137_0012_01_-411041790/tmp -Dspark.driver.port=23204 -Dspark.random.port.max=23999 -Dspark.akka.threads=32 -Dspark.akka.frameSize=10 -Dspark.akka.timeout=100 -Dspark.ui.port=23000 -Dspark.random.port.min=23000 -Dspark.yarn.app.container.log.dir=/srv/BigData/hadoop/data5/nm/containerlogs/application_1429065217137_0012/container_1429065217137_0012_01_-411041790 org.apache.spark.executor.CoarseGrainedExecutorBackend akka.tcp://sparkDriver@172.57.1.61:23204/user/CoarseGrainedScheduler 3 hadoopc1h11 10 application_1429065217137_0012 |- 126843 76960 126843 126843 (bash) 0 0 11603968 331 /bin/bash -c /opt/huawei/Bigdata/jdk1.7.0_76//bin/java -server -XX:OnOutOfMemoryError='kill %p' -Xms24576m -Xmx24576m -Dlog4j.configuration=file:/opt/huawei/Bigdata/DataSight_FM_BasePlatform_V100R001C00_Spark/spark/conf/log4j-executor.properties -Djava.library.path=/opt/huawei/Bigdata/DataSight_FM_BasePlatform_V100R001C00_Hadoop//hadoop/lib/native -Djava.io.tmpdir=/srv/BigData/hadoop/data4/nm/localdir/usercache/ossuser/appcache/application_1429065217137_0012/container_1429065217137_0012_01_-411041790/tmp '-Dspark.driver.port=23204' '-Dspark.random.port.max=23999' '-Dspark.akka.threads=32' '-Dspark.akka.frameSize=10' '-Dspark.akka.timeout=100' '-Dspark.ui.port=23000' '-Dspark.random.port.min=23000' -Dspark.yarn.app.container.log.dir=/srv/BigData/hadoop/data5/nm/containerlogs/application_1429065217137_0012/container_1429065217137_0012_01_-411041790 org.apache.spark.executor.CoarseGrainedExecutorBackend akka.tcp://sparkDriver@172.57.1.61:23204/user/CoarseGrainedScheduler 3 hadoopc1h11 10 application_1429065217137_0012 1 /srv/BigData/hadoop/data5/nm/containerlogs/application_1429065217137_0012/container_1429065217137_0012_01_-411041790/stdout 2 /srv/BigData/hadoop/data5/nm/containerlogs/application_1429065217137_0012/container_1429065217137_0012_01_-411041790/stderr | org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl$MonitoringThread.run(ContainersMonitorImpl.java:447) And some doubts 1) why executor will not release memory, if there are tasks running..? 2) is there issue from hadoop which will lead this error..? Any help , will be appreciated... Thanks Regards Brahma Reddy Battula
Re: Running beyond physical memory limits
Did you try reducing your spark.executor.memory? Thanks Best Regards On Wed, Apr 15, 2015 at 2:29 PM, Brahma Reddy Battula brahmareddy.batt...@huawei.com wrote: Hello Sparkers I am newbie to spark and need help.. We are using spark 1.2, we are getting the following error and executor is getting killed..I seen SPARK-1930 and it should be in 1.2.. *Any pointer to following error, like what might lead this error.*. 2015-04-15 11:55:39,697 | WARN | Container Monitor | Container [pid=126843,containerID=container_1429065217137_0012_01_-411041790] is running beyond physical memory limits. Current usage: 26.0 GB of 26 GB physical memory used; 26.7 GB of 260 GB virtual memory used. Killing container. Dump of the process-tree for container_1429065217137_0012_01_-411041790 :|- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE |- 126872 126843 126843 126843 (java) 2049457 22816 28673892352 6824864 /opt/huawei/Bigdata/jdk1.7.0_76//bin/java -server -XX:OnOutOfMemoryError=kill %p -Xms24576m -Xmx24576m -Dlog4j.configuration=file:/opt/huawei/Bigdata/DataSight_FM_BasePlatform_V100R001C00_Spark/spark/conf/log4j-executor.properties -Djava.library.path=/opt/huawei/Bigdata/DataSight_FM_BasePlatform_V100R001C00_Hadoop//hadoop/lib/native -Djava.io.tmpdir=/srv/BigData/hadoop/data4/nm/localdir/usercache/ossuser/appcache/application_1429065217137_0012/container_1429065217137_0012_01_-411041790/tmp -Dspark.driver.port=23204 -Dspark.random.port.max=23999 -Dspark.akka.threads=32 -Dspark.akka.frameSize=10 -Dspark.akka.timeout=100 -Dspark.ui.port=23000 -Dspark.random.port.min=23000 -Dspark.yarn.app.container.log.dir=/srv/BigData/hadoop/data5/nm/containerlogs/application_1429065217137_0012/container_1429065217137_0012_01_-411041790 org.apache.spark.executor.CoarseGrainedExecutorBackend akka.tcp:// sparkDriver@172.57.1.61:23204/user/CoarseGrainedScheduler 3 hadoopc1h11 10 application_1429065217137_0012 |- 126843 76960 126843 126843 (bash) 0 0 11603968 331 /bin/bash -c /opt/huawei/Bigdata/jdk1.7.0_76//bin/java -server -XX:OnOutOfMemoryError='kill %p' -Xms24576m -Xmx24576m -Dlog4j.configuration=file:/opt/huawei/Bigdata/DataSight_FM_BasePlatform_V100R001C00_Spark/spark/conf/log4j-executor.properties -Djava.library.path=/opt/huawei/Bigdata/DataSight_FM_BasePlatform_V100R001C00_Hadoop//hadoop/lib/native -Djava.io.tmpdir=/srv/BigData/hadoop/data4/nm/localdir/usercache/ossuser/appcache/application_1429065217137_0012/container_1429065217137_0012_01_-411041790/tmp '-Dspark.driver.port=23204' '-Dspark.random.port.max=23999' '-Dspark.akka.threads=32' '-Dspark.akka.frameSize=10' '-Dspark.akka.timeout=100' '-Dspark.ui.port=23000' '-Dspark.random.port.min=23000' -Dspark.yarn.app.container.log.dir=/srv/BigData/hadoop/data5/nm/containerlogs/application_1429065217137_0012/container_1429065217137_0012_01_-411041790 org.apache.spark.executor.CoarseGrainedExecutorBackend akka.tcp:// sparkDriver@172.57.1.61:23204/user/CoarseGrainedScheduler 3 hadoopc1h11 10 application_1429065217137_0012 1 /srv/BigData/hadoop/data5/nm/containerlogs/application_1429065217137_0012/container_1429065217137_0012_01_-411041790/stdout 2 /srv/BigData/hadoop/data5/nm/containerlogs/application_1429065217137_0012/container_1429065217137_0012_01_-411041790/stderr | org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl$MonitoringThread.run(ContainersMonitorImpl.java:447) And some doubts 1) why executor will not release memory, if there are tasks running..? 2) is there issue from hadoop which will lead this error..? Any help , will be appreciated... Thanks Regards Brahma Reddy Battula
Re: Running beyond physical memory limits
All this means is that your JVM is using more memory than it requested from YARN. You need to increase the YARN memory overhead setting, perhaps. On Wed, Apr 15, 2015 at 9:59 AM, Brahma Reddy Battula brahmareddy.batt...@huawei.com wrote: Hello Sparkers I am newbie to spark and need help.. We are using spark 1.2, we are getting the following error and executor is getting killed..I seen SPARK-1930 and it should be in 1.2.. Any pointer to following error, like what might lead this error.. 2015-04-15 11:55:39,697 | WARN | Container Monitor | Container [pid=126843,containerID=container_1429065217137_0012_01_-411041790] is running beyond physical memory limits. Current usage: 26.0 GB of 26 GB physical memory used; 26.7 GB of 260 GB virtual memory used. Killing container. Dump of the process-tree for container_1429065217137_0012_01_-411041790 : |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE |- 126872 126843 126843 126843 (java) 2049457 22816 28673892352 6824864 /opt/huawei/Bigdata/jdk1.7.0_76//bin/java -server -XX:OnOutOfMemoryError=kill %p -Xms24576m -Xmx24576m -Dlog4j.configuration=file:/opt/huawei/Bigdata/DataSight_FM_BasePlatform_V100R001C00_Spark/spark/conf/log4j-executor.properties -Djava.library.path=/opt/huawei/Bigdata/DataSight_FM_BasePlatform_V100R001C00_Hadoop//hadoop/lib/native -Djava.io.tmpdir=/srv/BigData/hadoop/data4/nm/localdir/usercache/ossuser/appcache/application_1429065217137_0012/container_1429065217137_0012_01_-411041790/tmp -Dspark.driver.port=23204 -Dspark.random.port.max=23999 -Dspark.akka.threads=32 -Dspark.akka.frameSize=10 -Dspark.akka.timeout=100 -Dspark.ui.port=23000 -Dspark.random.port.min=23000 -Dspark.yarn.app.container.log.dir=/srv/BigData/hadoop/data5/nm/containerlogs/application_1429065217137_0012/container_1429065217137_0012_01_-411041790 org.apache.spark.executor.CoarseGrainedExecutorBackend akka.tcp://sparkDriver@172.57.1.61:23204/user/CoarseGrainedScheduler 3 hadoopc1h11 10 application_1429065217137_0012 |- 126843 76960 126843 126843 (bash) 0 0 11603968 331 /bin/bash -c /opt/huawei/Bigdata/jdk1.7.0_76//bin/java -server -XX:OnOutOfMemoryError='kill %p' -Xms24576m -Xmx24576m -Dlog4j.configuration=file:/opt/huawei/Bigdata/DataSight_FM_BasePlatform_V100R001C00_Spark/spark/conf/log4j-executor.properties -Djava.library.path=/opt/huawei/Bigdata/DataSight_FM_BasePlatform_V100R001C00_Hadoop//hadoop/lib/native -Djava.io.tmpdir=/srv/BigData/hadoop/data4/nm/localdir/usercache/ossuser/appcache/application_1429065217137_0012/container_1429065217137_0012_01_-411041790/tmp '-Dspark.driver.port=23204' '-Dspark.random.port.max=23999' '-Dspark.akka.threads=32' '-Dspark.akka.frameSize=10' '-Dspark.akka.timeout=100' '-Dspark.ui.port=23000' '-Dspark.random.port.min=23000' -Dspark.yarn.app.container.log.dir=/srv/BigData/hadoop/data5/nm/containerlogs/application_1429065217137_0012/container_1429065217137_0012_01_-411041790 org.apache.spark.executor.CoarseGrainedExecutorBackend akka.tcp://sparkDriver@172.57.1.61:23204/user/CoarseGrainedScheduler 3 hadoopc1h11 10 application_1429065217137_0012 1 /srv/BigData/hadoop/data5/nm/containerlogs/application_1429065217137_0012/container_1429065217137_0012_01_-411041790/stdout 2 /srv/BigData/hadoop/data5/nm/containerlogs/application_1429065217137_0012/container_1429065217137_0012_01_-411041790/stderr | org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl$MonitoringThread.run(ContainersMonitorImpl.java:447) And some doubts 1) why executor will not release memory, if there are tasks running..? 2) is there issue from hadoop which will lead this error..? Any help , will be appreciated... Thanks Regards Brahma Reddy Battula - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
