Repository: spark
Updated Branches:
refs/heads/master fec462c15 -> 9edd88782
update spark.default.parallelism
actually, the value 8 is only valid in mesos fine-grained mode :
<code>
override def defaultParallelism() =
sc.conf.getInt("spark.default.parallelism", 8)
</code>
while in coarse-grained model including mesos coares-grained, the value of the
property depending on core numbers!
<code>
override def defaultParallelism(): Int = {
conf.getInt("spark.default.parallelism", math.max(totalCoreCount.get(), 2))
}
</code>
Author: Chen Chao <[email protected]>
Closes #389 from CrazyJvm/patch-2 and squashes the following commits:
84a7fe4 [Chen Chao] miss </li> at the end of every single line
04a9796 [Chen Chao] change format
ee0fae0 [Chen Chao] update spark.default.parallelism
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/9edd8878
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/9edd8878
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/9edd8878
Branch: refs/heads/master
Commit: 9edd88782e0268439c5ab57400d6a7ab432fc269
Parents: fec462c
Author: Chen Chao <[email protected]>
Authored: Wed Apr 16 09:14:18 2014 -0700
Committer: Patrick Wendell <[email protected]>
Committed: Wed Apr 16 09:14:18 2014 -0700
----------------------------------------------------------------------
docs/configuration.md | 8 +++++++-
1 file changed, 7 insertions(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/spark/blob/9edd8878/docs/configuration.md
----------------------------------------------------------------------
diff --git a/docs/configuration.md b/docs/configuration.md
index f3bfd03..a302983 100644
--- a/docs/configuration.md
+++ b/docs/configuration.md
@@ -96,7 +96,13 @@ Apart from these, the following properties are also
available, and may be useful
<tr><th>Property Name</th><th>Default</th><th>Meaning</th></tr>
<tr>
<td>spark.default.parallelism</td>
- <td>8</td>
+ <td>
+ <ul>
+ <li>Mesos fine grained mode: 8</li>
+ <li>Local mode: core number of the local machine</li>
+ <li>Others: total core number of all executor nodes or 2, whichever is
larger</li>
+ </ul>
+ </td>
<td>
Default number of tasks to use across the cluster for distributed shuffle
operations (<code>groupByKey</code>,
<code>reduceByKey</code>, etc) when not set by user.
