GitHub user mateiz opened a pull request:
https://github.com/apache/spark/pull/8220
[SPARK-10008] Ensure shuffle locality doesn't take precedence over narrow
deps
The shuffle locality patch made the DAGScheduler aware of shuffle data,
but for RDDs that have both narrow and shuffle dependencies, it can
cause them to place tasks based on the shuffle dependency instead of the
narrow one. This case is common in iterative join-based algorithms like
PageRank and ALS, where one RDD is hash-partitioned and one isn't.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/mateiz/spark shuffle-loc-fix
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/8220.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #8220
----
commit 99dd008680add9b4d779f6831834165798a74bcc
Author: Matei Zaharia <[email protected]>
Date: 2015-08-15T02:27:59Z
Ensure shuffle locality doesn't take precedence over narrow deps
The shuffle locality patch made the DAGScheduler aware of shuffle data,
but for RDDs that have both narrow and shuffle dependencies, it can
cause them to place tasks based on the shuffle dependency instead of the
narrow one. This case is common in iterative join-based algorithms like
PageRank and ALS, where one RDD is hash-partitioned and one isn't.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
