GitHub user viirya opened a pull request:
https://github.com/apache/spark/pull/6454
[SPARK-6307][Core] Coalesce RDDs before RDD.cartesian
JIRA: https://issues.apache.org/jira/browse/SPARK-6307 and
https://issues.apache.org/jira/browse/SPARK-6922
Alternative approach to this issue. Ref. #5572.
This PR tries to coalesce RDDs before `RDD.cartesian`. It also implements
the idea 1 discussed in #5572. So in `cartesian` we will fetch each partition
in RDD2 once for each partition in RDD1.
For performance test, using the same example codes in #5572, it costs about
4s.
cc @tbertelsen, @squito, @cloud-fan.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/viirya/spark-1 speed_up_cartesian
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/6454.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #6454
----
commit 72e199e348f719b89d91b3a92de028a59da252e5
Author: Liang-Chi Hsieh <[email protected]>
Date: 2015-05-28T10:23:39Z
Coalesce RDDs before RDD.cartesian.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
