[
https://issues.apache.org/jira/browse/SPARK-2468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14210074#comment-14210074
]
Aaron Davidson commented on SPARK-2468:
---------------------------------------
Here is my spark configuration (note 32 cores total):
spark.shuffle.io.clientThreads =16,
spark.shuffle.io.serverThreads =16,
spark.serializer = "org.apache.spark.serializer.KryoSerializer",
spark.shuffle.blockTransferService = "netty",
spark.shuffle.compress = false,
spark.shuffle.io.maxRetries = 0,
spark.reducer.maxMbInFlight = 512
Forgot to mention, but #3155 now automatically sets
spark.shuffle.io.clientThreads and spark.shuffle.io.serverThreads based on the
number of cores the Executor has allotted to it. You can override it by
setting those properties by hand, but ideally the default behavior is
sufficient.
> Netty-based block server / client module
> ----------------------------------------
>
> Key: SPARK-2468
> URL: https://issues.apache.org/jira/browse/SPARK-2468
> Project: Spark
> Issue Type: Improvement
> Components: Shuffle, Spark Core
> Reporter: Reynold Xin
> Assignee: Reynold Xin
> Priority: Critical
> Fix For: 1.2.0
>
>
> Right now shuffle send goes through the block manager. This is inefficient
> because it requires loading a block from disk into a kernel buffer, then into
> a user space buffer, and then back to a kernel send buffer before it reaches
> the NIC. It does multiple copies of the data and context switching between
> kernel/user. It also creates unnecessary buffer in the JVM that increases GC
> Instead, we should use FileChannel.transferTo, which handles this in the
> kernel space with zero-copy. See
> http://www.ibm.com/developerworks/library/j-zerocopy/
> One potential solution is to use Netty. Spark already has a Netty based
> network module implemented (org.apache.spark.network.netty). However, it
> lacks some functionality and is turned off by default.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
