GitHub user lianhuiwang opened a pull request:
https://github.com/apache/spark/pull/13028
[SPARK-9860][SQL][WIP] Join: Determine the join strategy (broadcast join or
shuffle join) at runtime
## What changes were proposed in this pull request?
1. add query fragment for execute stage according to exchange.
2. before running no leaf query fragment, optimize plan using
mapOutputStatistics of its child fragments.
if size of one exchange is less than autoBroadcastJoinThreshold, it
transforms sort merge join to broadcast join.
## How was this patch tested?
unit tests
later i will add some unit tests.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/lianhuiwang/spark adaptive-2
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/13028.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #13028
----
commit 7ff44e18f575f907cc6cdf7d667bc985b8834d1c
Author: Lianhui Wang <[email protected]>
Date: 2016-05-10T08:06:46Z
init commit
commit 25b00d73325f97e1a4ff5a3795b068387a80a53f
Author: Lianhui Wang <[email protected]>
Date: 2016-05-10T09:09:59Z
update minor code
commit 73de7b2d4a185edb22b896464daa4b18ee32914d
Author: Lianhui Wang <[email protected]>
Date: 2016-05-10T14:18:07Z
code refactor
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
