The setting to increase is spark.yarn.executor.memoryOverhead On Wed, Apr 15, 2015 at 6:35 AM, Brahma Reddy Battula < brahmareddy.batt...@huawei.com> wrote:
> Hello Sean Owen, > > Thanks for your reply..I"ll increase overhead memory and check it.. > > > Bytheway ,Any difference between 1.1 and 1.2 makes, this issue..? Since It > was passing spark 1.1 and throwing following error in 1.2...( this makes me > doubt full) > > > > Thanks & Regards > Brahma Reddy Battula > > > > > ________________________________________ > From: Sean Owen [so...@cloudera.com] > Sent: Wednesday, April 15, 2015 2:49 PM > To: Brahma Reddy Battula > Cc: Akhil Das; user@spark.apache.org > Subject: Re: Running beyond physical memory limits > > This is not related to executor memory, but the extra overhead > subtracted from the executor's size in order to avoid using more than > the physical memory that YARN allows. That is, if you declare a 32G > executor YARN lets you use 32G physical memory but your JVM heap must > be significantly less than 32G max. This is the overhead factor that > is subtracted for you, and it seems to need to be bigger in your case. > > On Wed, Apr 15, 2015 at 10:16 AM, Brahma Reddy Battula > <brahmareddy.batt...@huawei.com> wrote: > > Thanks lot for your reply.. > > > > There is no issue with spark1.1..Following issue came when I upgrade to > > spark2.0...Hence I did not decrease spark.executor.memory... > > I mean to say, used same config for spark1.1 and spark1.2.. > > > > Is there any issue with spark1.2..? > > Or Yarn will lead this..? > > And why executor will not release memory, if there are tasks running..? > > > > > > Thanks & Regards > > > > Brahma Reddy Battula > > > > > > ________________________________ > > From: Akhil Das [ak...@sigmoidanalytics.com] > > Sent: Wednesday, April 15, 2015 2:35 PM > > To: Brahma Reddy Battula > > Cc: user@spark.apache.org > > Subject: Re: Running beyond physical memory limits > > > > Did you try reducing your spark.executor.memory? > > > > Thanks > > Best Regards > > > > On Wed, Apr 15, 2015 at 2:29 PM, Brahma Reddy Battula > > <brahmareddy.batt...@huawei.com> wrote: > >> > >> Hello Sparkers > >> > >> > >> I am newbie to spark and need help.. We are using spark 1.2, we are > >> getting the following error and executor is getting killed..I seen > >> SPARK-1930 and it should be in 1.2.. > >> > >> Any pointer to following error, like what might lead this error.. > >> > >> > >> 2015-04-15 11:55:39,697 | WARN | Container Monitor | Container > >> [pid=126843,containerID=container_1429065217137_0012_01_-411041790] is > >> running beyond physical memory limits. Current usage: 26.0 GB of 26 GB > >> physical memory used; 26.7 GB of 260 GB virtual memory used. Killing > >> container. > >> Dump of the process-tree for container_1429065217137_0012_01_-411041790 > : > >> |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) > >> SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE > >> |- 126872 126843 126843 126843 (java) 2049457 22816 28673892352 > >> 6824864 /opt/huawei/Bigdata/jdk1.7.0_76//bin/java -server > >> -XX:OnOutOfMemoryError=kill %p -Xms24576m -Xmx24576m > >> > -Dlog4j.configuration=file:/opt/huawei/Bigdata/DataSight_FM_BasePlatform_V100R001C00_Spark/spark/conf/log4j-executor.properties > >> > -Djava.library.path=/opt/huawei/Bigdata/DataSight_FM_BasePlatform_V100R001C00_Hadoop//hadoop/lib/native > >> > -Djava.io.tmpdir=/srv/BigData/hadoop/data4/nm/localdir/usercache/ossuser/appcache/application_1429065217137_0012/container_1429065217137_0012_01_-411041790/tmp > >> -Dspark.driver.port=23204 -Dspark.random.port.max=23999 > >> -Dspark.akka.threads=32 -Dspark.akka.frameSize=10 > -Dspark.akka.timeout=100 > >> -Dspark.ui.port=23000 -Dspark.random.port.min=23000 > >> > -Dspark.yarn.app.container.log.dir=/srv/BigData/hadoop/data5/nm/containerlogs/application_1429065217137_0012/container_1429065217137_0012_01_-411041790 > >> org.apache.spark.executor.CoarseGrainedExecutorBackend > >> akka.tcp://sparkDriver@172.57.1.61:23204/user/CoarseGrainedScheduler 3 > >> hadoopc1h11 10 application_1429065217137_0012 |- 126843 76960 > 126843 > >> 126843 (bash) 0 0 11603968 331 /bin/bash -c > >> /opt/huawei/Bigdata/jdk1.7.0_76//bin/java -server > >> -XX:OnOutOfMemoryError='kill %p' -Xms24576m -Xmx24576m > >> > -Dlog4j.configuration=file:/opt/huawei/Bigdata/DataSight_FM_BasePlatform_V100R001C00_Spark/spark/conf/log4j-executor.properties > >> > -Djava.library.path=/opt/huawei/Bigdata/DataSight_FM_BasePlatform_V100R001C00_Hadoop//hadoop/lib/native > >> > -Djava.io.tmpdir=/srv/BigData/hadoop/data4/nm/localdir/usercache/ossuser/appcache/application_1429065217137_0012/container_1429065217137_0012_01_-411041790/tmp > >> '-Dspark.driver.port=23204' '-Dspark.random.port.max=23999' > >> '-Dspark.akka.threads=32' '-Dspark.akka.frameSize=10' > >> '-Dspark.akka.timeout=100' '-Dspark.ui.port=23000' > >> '-Dspark.random.port.min=23000' > >> > -Dspark.yarn.app.container.log.dir=/srv/BigData/hadoop/data5/nm/containerlogs/application_1429065217137_0012/container_1429065217137_0012_01_-411041790 > >> org.apache.spark.executor.CoarseGrainedExecutorBackend > >> akka.tcp://sparkDriver@172.57.1.61:23204/user/CoarseGrainedScheduler 3 > >> hadoopc1h11 10 application_1429065217137_0012 1> > >> > /srv/BigData/hadoop/data5/nm/containerlogs/application_1429065217137_0012/container_1429065217137_0012_01_-411041790/stdout > >> 2> > >> > /srv/BigData/hadoop/data5/nm/containerlogs/application_1429065217137_0012/container_1429065217137_0012_01_-411041790/stderr > >> | > >> > org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl$MonitoringThread.run(ContainersMonitorImpl.java:447) > >> > >> > >> > >> And some doubts > >> > >> > >> 1) why executor will not release memory, if there are tasks running..? > >> > >> > >> > >> 2) is there issue from hadoop which will lead this error..? > >> > >> > >> > >> Any help , will be appreciated... > >> > >> > >> > >> > >> Thanks & Regards > >> > >> Brahma Reddy Battula > >> > >> > >> > >> > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > For additional commands, e-mail: user-h...@spark.apache.org > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > For additional commands, e-mail: user-h...@spark.apache.org > >
