Repository: spark Updated Branches: refs/heads/branch-1.0 9e908ab2e -> e4f5577e2
update spark.default.parallelism actually, the value 8 is only valid in mesos fine-grained mode : <code> override def defaultParallelism() = sc.conf.getInt("spark.default.parallelism", 8) </code> while in coarse-grained model including mesos coares-grained, the value of the property depending on core numbers! <code> override def defaultParallelism(): Int = { conf.getInt("spark.default.parallelism", math.max(totalCoreCount.get(), 2)) } </code> Author: Chen Chao <crazy...@gmail.com> Closes #389 from CrazyJvm/patch-2 and squashes the following commits: 84a7fe4 [Chen Chao] miss </li> at the end of every single line 04a9796 [Chen Chao] change format ee0fae0 [Chen Chao] update spark.default.parallelism (cherry picked from commit 9edd88782e0268439c5ab57400d6a7ab432fc269) Signed-off-by: Patrick Wendell <pwend...@gmail.com> Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/e4f5577e Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/e4f5577e Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/e4f5577e Branch: refs/heads/branch-1.0 Commit: e4f5577e28ac53fc16aa966ed1f1bf9463de9337 Parents: 9e908ab Author: Chen Chao <crazy...@gmail.com> Authored: Wed Apr 16 09:14:18 2014 -0700 Committer: Patrick Wendell <pwend...@gmail.com> Committed: Wed Apr 16 09:14:30 2014 -0700 ---------------------------------------------------------------------- docs/configuration.md | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/spark/blob/e4f5577e/docs/configuration.md ---------------------------------------------------------------------- diff --git a/docs/configuration.md b/docs/configuration.md index f3bfd03..a302983 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -96,7 +96,13 @@ Apart from these, the following properties are also available, and may be useful <tr><th>Property Name</th><th>Default</th><th>Meaning</th></tr> <tr> <td>spark.default.parallelism</td> - <td>8</td> + <td> + <ul> + <li>Mesos fine grained mode: 8</li> + <li>Local mode: core number of the local machine</li> + <li>Others: total core number of all executor nodes or 2, whichever is larger</li> + </ul> + </td> <td> Default number of tasks to use across the cluster for distributed shuffle operations (<code>groupByKey</code>, <code>reduceByKey</code>, etc) when not set by user.