GitHub user rxin opened a pull request:
https://github.com/apache/spark/pull/1452
[SPARK-2521] Broadcast RDD object once per TaskSet (instead of sending it
for every task).
This should substantially reduces task size for large jobs (with large
closures). It should improve task dispatching and scheduler throughput.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/rxin/spark broadcast-task
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/1452.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1452
----
commit 73b2783fef785941fc966ad32f2fd987b12447ae
Author: Reynold Xin <[email protected]>
Date: 2014-07-16T23:34:34Z
[SPARK-2534] Avoid pulling in the entire RDD in groupByKey.
commit 2f7fc5f00b6bbb3970224119d4380f0117fcb5be
Author: Reynold Xin <[email protected]>
Date: 2014-07-16T23:53:35Z
[SPARK-2521] Broadcast RDD object once per TaskSet (instead of sending it
for every task).
commit 9a4a770f0350a863f9096d4486aa124af9a07d6a
Author: Reynold Xin <[email protected]>
Date: 2014-07-16T23:56:55Z
Explain why broadcasting serialized copy of the task.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
