GitHub user mateiz opened a pull request:
https://github.com/apache/spark/pull/8180
[SPARK-9851] Support submitting map stages individually in DAGScheduler
This patch adds support for submitting map stages in a DAG individually so
that we can make downstream decisions after seeing statistics about their
output, as part of SPARK-9850. I also added more comments to many of the key
classes in DAGScheduler. By itself, the patch is not super useful except maybe
to switch between a shuffle and broadcast join, but with the other subtasks of
SPARK-9850 we'll be able to do more interesting decisions.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/mateiz/spark spark-9851
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/8180.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #8180
----
commit 320709f4958b5927572caf3aedc13e755b920e92
Author: Matei Zaharia <[email protected]>
Date: 2015-07-24T01:42:36Z
Initial work on adaptive execution in DAGScheduler
commit 3f167512088997f91856322768ee7382138c5f67
Author: Matei Zaharia <[email protected]>
Date: 2015-08-12T23:44:36Z
Remove reduce task stuff, clean up, add docs and tests
commit f4da6191022f104e56121f0da89a5a1ff9325cb2
Author: Matei Zaharia <[email protected]>
Date: 2015-08-13T21:16:30Z
Add a test to FailureSuite too
commit 8b5f85edb111cfd17dc75798d7146d3ff43e2306
Author: Matei Zaharia <[email protected]>
Date: 2015-08-13T21:57:52Z
Another test suite
commit 78fd69155f6cac5f5b81fe01b60598e657ba5db3
Author: Matei Zaharia <[email protected]>
Date: 2015-08-13T22:02:28Z
tweak
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
