[
https://issues.apache.org/jira/browse/SPARK-822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14291235#comment-14291235
]
Josh Rosen commented on SPARK-822:
----------------------------------
I merged a pull request that adds a link to the earlier discussion to the
defaultMinPartitions documentation so that folks don't re-attempt to change
this: https://github.com/apache/spark/pull/4102
> defaultMinSplits can't be set higher than 2
> -------------------------------------------
>
> Key: SPARK-822
> URL: https://issues.apache.org/jira/browse/SPARK-822
> Project: Spark
> Issue Type: Bug
> Components: PySpark, Spark Core
> Affects Versions: 0.8.0, 0.7.3
> Reporter: Josh Rosen
>
> {{SparkContext.defaultMinSplits}} is used to control the default level of
> parallelism when reading input from files. Unfortunately, the current
> implementation doesn't allow defaultMinSplits to be set higher than 2.
> [From
> 0.7.3|https://github.com/mesos/spark/blob/v0.7.3/core/src/main/scala/spark/SparkContext.scala#L707]:
> {code}
> /** Default level of parallelism to use when not given by user (e.g.
> parallelize and makeRDD). */
> def defaultParallelism: Int = taskScheduler.defaultParallelism
> /** Default min number of partitions for Hadoop RDDs when not given by user
> */
> def defaultMinSplits: Int = math.min(defaultParallelism, 2)
> {code}
> I think that the {{math.min}} should be {{math.max}} instead. PySpark has
> the same problem.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
