GitHub user yhuai opened a pull request:
https://github.com/apache/spark/pull/8296
[SPARK-10087] [CORE] [BRANCH-1.5] Disable
spark.shuffle.reduceLocality.enabled by default.
https://issues.apache.org/jira/browse/SPARK-10087
In some cases, when spark.shuffle.reduceLocality.enabled is enabled, we are
scheduling all reducers to the same executor (the cluster has plenty of
resources). Changing spark.shuffle.reduceLocality.enabled to false resolve the
problem.
Comments of https://github.com/apache/spark/pull/8280 provide more details
of the symptom of this issue.
This PR changes the default setting of
`spark.shuffle.reduceLocality.enabled` to `false` for branch 1.5.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/yhuai/spark
setNumPartitionsCorrectly-branch1.5
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/8296.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #8296
----
commit 90b25a3585cdce1f0fac9c8e43152e5efd8fed13
Author: Yin Huai <[email protected]>
Date: 2015-08-19T01:48:23Z
Disable spark.shuffle.reduceLocality.enabled by default.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
