GitHub user staple opened a pull request:
https://github.com/apache/spark/pull/1362
[SPARK-695] In DAGScheduler's getPreferredLocs, track set of visited
partitions.
What do you think of a method like this for avoiding exponential path
exploration in DAGScheduler's getPreferredLocs, per SPARK-695?
Some minor cleanups are also included. The comment I removed in
Dependency.scala seemed incorrect to me, per my commit comment:
Remove apparently incorrect comment describing NarrowDependency.
Each of a CartesianRDD's dependencies, for example, is a
NarrowDependency, but child partitions of
these dependencies may depend on shared parent partitions. For
example, the cartesian product of
RDDs a and b containing partitions [ a_0 ] and [ b_0, b_1 ]
respectively will be partitioned as
[ ( a_0, b_0 ), ( a_0, b_1 ) ]. Each child partition depends on parent
partition a_0.
I'm new to spark, scala (and even java) so a careful review may be in order.
Thanks,
Aaron
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/staple/spark SPARK-695
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/1362.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1362
----
commit 16d08271b30be6d33866f4b0411623e53e1149aa
Author: Aaron Staple <[email protected]>
Date: 2014-07-10T06:07:37Z
[SPARK-695] In DAGScheduler's getPreferredLocs, track set of visited
partitions.
commit 2f30ba15994b02713a1d52ada954dd596cf6732a
Author: Aaron Staple <[email protected]>
Date: 2014-07-10T06:07:37Z
Remove apparently incorrect comment describing NarrowDependency.
Each of a CartesianRDD's dependencies, for example, is a NarrowDependency,
but child partitions of
these dependencies may depend on shared parent partitions. For example,
the cartesian product of
RDDs a and b containing partitions [ a_0 ] and [ b_0, b_1 ] respectively
will be partitioned as
[ ( a_0, b_0 ), ( a_0, b_1 ) ]. Each child partition depends on parent
partition a_0.
commit 60f348fe51344dbff6f7d26d8b8bb6db8bf29596
Author: Aaron Staple <[email protected]>
Date: 2014-07-10T06:07:37Z
Clarify comment.
commit 6da1786e29a250ee2e98e2d34e0306693a75b8bc
Author: Aaron Staple <[email protected]>
Date: 2014-07-10T06:07:37Z
Remove unused variable.
commit ada3026b100824218f7c3ff332738411ef6387ff
Author: Aaron Staple <[email protected]>
Date: 2014-07-10T06:07:37Z
Fix indentation.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
