On 17 Feb 2016, at 01:29, Nirav Patel <npa...@xactlycorp.com<mailto:npa...@xactlycorp.com>> wrote:
I think you are not getting my question . I know how to tune executor memory settings and parallelism . That's not an issue. It's a specific question about what happens when physical memory limit of given executor is reached. Now yarn nodemanager has specific setting about provisioning virtual memory which can be utilized by a map reduce program. And I have seen it. But looks like spark application can not use virtual memory at all! It just kills an executor once it's physical memory limit is reached . Does anyone has explanation why such design choice was made? 1. The NM memory management is not specific to any app; it's for all containers. Where it is weak is that it assumes that the cost of a loss of a process is relatively low; its worse in long-lived (i.e. streaming) apps, as state gets lost. There's some improvements there in discussion on YARN-4692. 2. Why the killing? If some process starts to swap it hurts all the others —the performance of the entire cluster downgrades, without any obvious cause. That is, the person who underestimated their resource requirements doesn't get the direct feedback of "you need to ask for more memory" 3. There's some forgiveness at app launch, so that if your code does a fork() or two its vmem allocation can go up, but otherwise, if the vmem check is enabled, your process gets killed. The Spark AM gets told, and doesn't treat it as seriously as other problems (or view the specific host as unreliable in any way) 4. Otherwise, the fix is to ask for the memory you need. Looking at the stack trace, that's not VM/swap, thats JVM heap size. Separate problem. -Steve Ps - I have given 16gb per executor which is more than enough to handle biggest skewed in our data set. So Sent from my iPhone On Feb 15, 2016, at 8:40 AM, Sabarish Sasidharan <sabarish.sasidha...@manthan.com<mailto:sabarish.sasidha...@manthan.com>> wrote: Looks like your executors are running out of memory. YARN is not kicking them out. Just increase the executor memory. Also considering increasing the parallelism ie the number of partitions. Regards Sab On 11-Feb-2016 5:46 am, "Nirav Patel" <npa...@xactlycorp.com<mailto:npa...@xactlycorp.com>> wrote: In Yarn we have following settings enabled so that job can use virtual memory to have a capacity beyond physical memory off course. <property> <name>yarn.nodemanager.vmem-check-enabled</name> <value>false</value> </property> <property> <name>yarn.nodemanager.pmem-check-enabled</name> <value>false</value> </property> vmem to pmem ration is 2:1. However spark doesn't seem to be able to utilize this vmem limits we are getting following heap space error which seemed to be contained within spark executor. 16/02/09 23:08:06 ERROR executor.CoarseGrainedExecutorBackend: RECEIVED SIGNAL 15: SIGTERM 16/02/09 23:08:06 ERROR executor.Executor: Exception in task 4.0 in stage 7.6 (TID 22363) java.lang.OutOfMemoryError: Java heap space at java.util.IdentityHashMap.resize(IdentityHashMap.java:469) at java.util.IdentityHashMap.put(IdentityHashMap.java:445) at org.apache.spark.util.SizeEstimator$SearchState.enqueue(SizeEstimator.scala:159) at org.apache.spark.util.SizeEstimator$$anonfun$visitSingleObject$1.apply(SizeEstimator.scala:203) at org.apache.spark.util.SizeEstimator$$anonfun$visitSingleObject$1.apply(SizeEstimator.scala:202) at scala.collection.immutable.List.foreach(List.scala:318) at org.apache.spark.util.SizeEstimator$.visitSingleObject(SizeEstimator.scala:202) at org.apache.spark.util.SizeEstimator$.org$apache$spark$util$SizeEstimator$$estimate(SizeEstimator.scala:186) at org.apache.spark.util.SizeEstimator$.estimate(SizeEstimator.scala:54) at org.apache.spark.util.collection.SizeTracker$class.takeSample(SizeTracker.scala:78) at org.apache.spark.util.collection.SizeTracker$class.afterUpdate(SizeTracker.scala:70) at org.apache.spark.util.collection.SizeTrackingVector.$plus$eq(SizeTrackingVector.scala:31) at org.apache.spark.storage.MemoryStore.unrollSafely(MemoryStore.scala:278) at org.apache.spark.CacheManager.putInBlockManager(CacheManager.scala:171) at org.apache.spark.CacheManager.getOrCompute(CacheManager.scala:78) at org.apache.spark.rdd.RDD.iterator(RDD.scala:262) at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:300) at org.apache.spark.rdd.RDD.iterator(RDD.scala:264) at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66) at org.apache.spark.scheduler.Task.run(Task.scala:88) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:744) Yarn resource manager doesn't give any indication that whether container ran out of phycial or virtual memory limits. Also how to profile this container memory usage? We know our data is skewed so some of the executor will have large data (~2M RDD objects) to process. I used following as executorJavaOpts but it doesn't seem to work. -XX:-HeapDumpOnOutOfMemoryError -XX:OnOutOfMemoryError='kill -3 %p' -XX:HeapDumpPath=/opt/cores/spark [What's New with Xactly]<http://www.xactlycorp.com/email-click/> [https://www.xactlycorp.com/wp-content/uploads/2015/07/nyse_xtly_alt_24.png]<https://www.nyse.com/quote/XNYS:XTLY> [LinkedIn] <https://www.linkedin.com/company/xactly-corporation> [Twitter] <https://twitter.com/Xactly> [Facebook] <https://www.facebook.com/XactlyCorp> [YouTube] <http://www.youtube.com/xactlycorporation> [What's New with Xactly]<http://www.xactlycorp.com/email-click/> [https://www.xactlycorp.com/wp-content/uploads/2015/07/nyse_xtly_alt_24.png]<https://www.nyse.com/quote/XNYS:XTLY> [LinkedIn] <https://www.linkedin.com/company/xactly-corporation> [Twitter] <https://twitter.com/Xactly> [Facebook] <https://www.facebook.com/XactlyCorp> [YouTube] <http://www.youtube.com/xactlycorporation>
