[ https://issues.apache.org/jira/browse/SPARK-2468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14202490#comment-14202490 ]
Aaron Davidson commented on SPARK-2468: --------------------------------------- [~lianhuiwang] Can you try again with preferDirectBufs set to true, and just setting maxUsableCores down to the number of cores each container actually has? It's possible the performance discrepancy you're seeing is simply due to heap byte buffers not being as fast as direct ones. You might also decrease the Java heap size a bit while keeping the container size the same, if _any_ direct memory allocation is causing the container to be killed. [~zzcclp] Same suggestion for you about setting preferDirectBufs to true and setting maxUsableCores down, but I will also perform another round of benchmarking -- it's possible we accidentally introduced a performance regression in the last few patches. Comparing Hadoop vs Spark performance is a different matter. A few suggestions on your setup: You should set executor-cores to 5, so that each executor is actually using 5 cores instead of just 1. You're losing significant parallelism because of this setting, as Spark will only launch 1 task per core on an executor at any given time. Second, groupBy() is inefficient (it's doc was changed recently to reflect this), and should be avoided. I would recommend changing your job to sort the whole RDD using something similar to {code}mapR.map { x => ((x._1._1, x._2._1), x) }.sortByKey(){code}, which would not require that all values for a single group fit in memory. This would still effectively group by x._1._1, but would sort within each group by x._2._1, and would utilize Spark's efficient sorting machinery. > Netty-based block server / client module > ---------------------------------------- > > Key: SPARK-2468 > URL: https://issues.apache.org/jira/browse/SPARK-2468 > Project: Spark > Issue Type: Improvement > Components: Shuffle, Spark Core > Reporter: Reynold Xin > Assignee: Reynold Xin > Priority: Critical > Fix For: 1.2.0 > > > Right now shuffle send goes through the block manager. This is inefficient > because it requires loading a block from disk into a kernel buffer, then into > a user space buffer, and then back to a kernel send buffer before it reaches > the NIC. It does multiple copies of the data and context switching between > kernel/user. It also creates unnecessary buffer in the JVM that increases GC > Instead, we should use FileChannel.transferTo, which handles this in the > kernel space with zero-copy. See > http://www.ibm.com/developerworks/library/j-zerocopy/ > One potential solution is to use Netty. Spark already has a Netty based > network module implemented (org.apache.spark.network.netty). However, it > lacks some functionality and is turned off by default. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org