GitHub user CrazyJvm opened a pull request:
https://github.com/apache/spark/pull/766
default task number misleading in several places
private[streaming] def defaultPartitioner(numPartitions: Int =
self.ssc.sc.defaultParallelism){
new HashPartitioner(numPartitions)
}
it represents that the default task number in Spark Streaming relies on the
variable defaultParallelism in SparkContext, which is decided by the config
property spark.default.parallelism
the property "spark.default.parallelism" refers to
https://github.com/apache/spark/pull/389
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/CrazyJvm/spark patch-7
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/766.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #766
----
commit cc5b66c1883eca8862b8f37ef50d64cc0408c54c
Author: Chen Chao <[email protected]>
Date: 2014-05-14T07:45:10Z
default task number misleading in several places
<code>
private[streaming] def defaultPartitioner(numPartitions: Int =
self.ssc.sc.defaultParallelism){
new HashPartitioner(numPartitions)
}
</code>
it represents that the default task number in Spark Streaming relies on the
variable defaultParallelism in SparkContext, which is decided by the config
property spark.default.parallelism
the property "spark.default.parallelism" refers to
https://github.com/apache/spark/pull/389
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
