Repository: spark Updated Branches: refs/heads/master 7c0ed13d2 -> fbca6b6ce
[SPARK-4864] Add documentation to Netty-based configs Author: Aaron Davidson <[email protected]> Closes #3713 from aarondav/netty-configs and squashes the following commits: 8a8b373 [Aaron Davidson] Address Patrick's comments 3b1f84e [Aaron Davidson] [SPARK-4864] Add documentation to Netty-based configs Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/fbca6b6c Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/fbca6b6c Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/fbca6b6c Branch: refs/heads/master Commit: fbca6b6ce293b1997b40abeb9ab77b8a969a5fc9 Parents: 7c0ed13 Author: Aaron Davidson <[email protected]> Authored: Mon Dec 22 13:09:22 2014 -0800 Committer: Patrick Wendell <[email protected]> Committed: Mon Dec 22 13:09:22 2014 -0800 ---------------------------------------------------------------------- docs/configuration.md | 35 ++++++++++++++++++++ .../spark/network/util/TransportConf.java | 2 +- 2 files changed, 36 insertions(+), 1 deletion(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/spark/blob/fbca6b6c/docs/configuration.md ---------------------------------------------------------------------- diff --git a/docs/configuration.md b/docs/configuration.md index 2c8dea8..2cc013c 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -852,6 +852,41 @@ Apart from these, the following properties are also available, and may be useful between nodes leading to flooding the network with those. </td> </tr> +<tr> + <td><code>spark.shuffle.io.preferDirectBufs</code></td> + <td>true</td> + <td> + (Netty only) Off-heap buffers are used to reduce garbage collection during shuffle and cache + block transfer. For environments where off-heap memory is tightly limited, users may wish to + turn this off to force all allocations from Netty to be on-heap. + </td> +</tr> +<tr> + <td><code>spark.shuffle.io.numConnectionsPerPeer</code></td> + <td>1</td> + <td> + (Netty only) Connections between hosts are reused in order to reduce connection buildup for + large clusters. For clusters with many hard disks and few hosts, this may result in insufficient + concurrency to saturate all disks, and so users may consider increasing this value. + </td> +</tr> +<tr> + <td><code>spark.shuffle.io.maxRetries</code></td> + <td>3</td> + <td> + (Netty only) Fetches that fail due to IO-related exceptions are automatically retried if this is + set to a non-zero value. This retry logic helps stabilize large shuffles in the face of long GC + pauses or transient network connectivity issues. + </td> +</tr> +<tr> + <td><code>spark.shuffle.io.retryWait</code></td> + <td>5</td> + <td> + (Netty only) Seconds to wait between retries of fetches. The maximum delay caused by retrying + is simply <code>maxRetries * retryWait</code>, by default 15 seconds. + </td> +</tr> </table> #### Scheduling http://git-wip-us.apache.org/repos/asf/spark/blob/fbca6b6c/network/common/src/main/java/org/apache/spark/network/util/TransportConf.java ---------------------------------------------------------------------- diff --git a/network/common/src/main/java/org/apache/spark/network/util/TransportConf.java b/network/common/src/main/java/org/apache/spark/network/util/TransportConf.java index 13b37f9..7c9adf5 100644 --- a/network/common/src/main/java/org/apache/spark/network/util/TransportConf.java +++ b/network/common/src/main/java/org/apache/spark/network/util/TransportConf.java @@ -40,7 +40,7 @@ public class TransportConf { return conf.getInt("spark.shuffle.io.connectionTimeout", 120) * 1000; } - /** Number of concurrent connections between two nodes for fetching data. **/ + /** Number of concurrent connections between two nodes for fetching data. */ public int numConnectionsPerPeer() { return conf.getInt("spark.shuffle.io.numConnectionsPerPeer", 1); } --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
