Repository: spark Updated Branches: refs/heads/master 60472dbfd -> cfb25b27c
[SPARK-21530] Update description of spark.shuffle.maxChunksBeingTransferred. ## What changes were proposed in this pull request? Update the description of `spark.shuffle.maxChunksBeingTransferred` to include that the new coming connections will be closed when the max is hit and client should have retry mechanism. Author: jinxing <jinxing6...@126.com> Closes #18735 from jinxing64/SPARK-21530. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/cfb25b27 Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/cfb25b27 Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/cfb25b27 Branch: refs/heads/master Commit: cfb25b27c0b32a8a70a518955fb269314b1fd716 Parents: 60472db Author: jinxing <jinxing6...@126.com> Authored: Thu Jul 27 11:55:48 2017 +0800 Committer: Wenchen Fan <wenc...@databricks.com> Committed: Thu Jul 27 11:55:48 2017 +0800 ---------------------------------------------------------------------- .../main/java/org/apache/spark/network/util/TransportConf.java | 6 +++++- docs/configuration.md | 6 +++++- 2 files changed, 10 insertions(+), 2 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/spark/blob/cfb25b27/common/network-common/src/main/java/org/apache/spark/network/util/TransportConf.java ---------------------------------------------------------------------- diff --git a/common/network-common/src/main/java/org/apache/spark/network/util/TransportConf.java b/common/network-common/src/main/java/org/apache/spark/network/util/TransportConf.java index ea52e9f..88256b8 100644 --- a/common/network-common/src/main/java/org/apache/spark/network/util/TransportConf.java +++ b/common/network-common/src/main/java/org/apache/spark/network/util/TransportConf.java @@ -258,7 +258,11 @@ public class TransportConf { } /** - * The max number of chunks allowed to being transferred at the same time on shuffle service. + * The max number of chunks allowed to be transferred at the same time on shuffle service. + * Note that new incoming connections will be closed when the max number is hit. The client will + * retry according to the shuffle retry configs (see `spark.shuffle.io.maxRetries` and + * `spark.shuffle.io.retryWait`), if those limits are reached the task will fail with fetch + * failure. */ public long maxChunksBeingTransferred() { return conf.getLong("spark.shuffle.maxChunksBeingTransferred", Long.MAX_VALUE); http://git-wip-us.apache.org/repos/asf/spark/blob/cfb25b27/docs/configuration.md ---------------------------------------------------------------------- diff --git a/docs/configuration.md b/docs/configuration.md index f4b6f46..500f980 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -635,7 +635,11 @@ Apart from these, the following properties are also available, and may be useful <td><code>spark.shuffle.maxChunksBeingTransferred</code></td> <td>Long.MAX_VALUE</td> <td> - The max number of chunks allowed to being transferred at the same time on shuffle service. + The max number of chunks allowed to be transferred at the same time on shuffle service. + Note that new incoming connections will be closed when the max number is hit. The client will + retry according to the shuffle retry configs (see <code>spark.shuffle.io.maxRetries</code> and + <code>spark.shuffle.io.retryWait</code>), if those limits are reached the task will fail with + fetch failure. </td> </tr> <tr> --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org