GitHub user gatorsmile reopened a pull request:
https://github.com/apache/spark/pull/10566
[SPARK-12613] [SQL] Outer Join Elimination by Parent Join Condition
This PR is another enhancement to Optimizer. It does not conflict with the
other PRs (https://github.com/apache/spark/pull/10567 and
https://github.com/apache/spark/pull/10551).
Given an outer join (**OJ**) is involved in another join (called parent
join **PJ**), when the join type of **PJ** is `inner`, `left-semi`,
`left-outer` and `right-outer`, checking if the join condition of the **PJ**
satisfies the following two conditions:
1) there exist null filtering predicates against the columns in the
null-supplying side of **PJ**.
2) these columns are from **OJ**.
If having such join predicates, execute the elimination rules:
- `full outer` -> `inner` if both sides of **OJ** have such predicates
- `left outer` -> `inner` if the right side of **OJ** has such predicates
- `right outer` -> `inner` if the left side of **OJ** has such predicates
- `full outer` -> `left outer` if only the left side of **OJ** has such
predicates
- `full outer` -> `right outer` if only the right side of **OJ** has such
predicates
If applicable, this can greatly improve the performance, since `outer join`
is much slower than `inner join`, `full outer` join is much slower than
`left`/`right outer` join.
BTW, since the rule is different from the rule in
https://github.com/apache/spark/pull/10567, I did not merge them in the same
one for simplifying the code review.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/gatorsmile/spark
OuterJoinEliminationByParentJoinPredicate
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/10566.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #10566
----
commit 01e4cdfcfc4ac37644165923c6e8eb65fcfdf3ac
Author: gatorsmile <[email protected]>
Date: 2015-11-13T22:50:39Z
Merge remote-tracking branch 'upstream/master'
commit 6835704c273abc13e8eda37f5a10715027e4d17b
Author: gatorsmile <[email protected]>
Date: 2015-11-14T02:50:51Z
Merge remote-tracking branch 'upstream/master'
commit 9180687775649f97763bdbd7c004fe6fc392989c
Author: gatorsmile <[email protected]>
Date: 2015-11-14T17:01:59Z
Merge remote-tracking branch 'upstream/master'
commit b38a21ef6146784e4b93ef4ce8c899f1eee14572
Author: gatorsmile <[email protected]>
Date: 2015-11-17T02:30:26Z
SPARK-11633
commit d2b84af8cce7fc2c03c748a2d443c07bad3f0ed1
Author: gatorsmile <[email protected]>
Date: 2015-11-17T02:32:12Z
Merge remote-tracking branch 'upstream/master' into joinMakeCopy
commit fda8025195a0f872c9b357e8d86a1ce37d4c8379
Author: xiaoli <[email protected]>
Date: 2015-11-17T04:16:43Z
Merge remote-tracking branch 'upstream/master'
commit ac0dccd0bc7149ec1e6a2e89d2c462388ad4a80c
Author: xiaoli <[email protected]>
Date: 2015-11-17T04:18:59Z
Merge branch 'master' of https://github.com/gatorsmile/spark
commit 6e0018b67faf769f11032a217417bd09bace7ed7
Author: Xiao Li <[email protected]>
Date: 2015-11-20T18:51:19Z
Merge remote-tracking branch 'upstream/master'
commit 0546772f151f83d6d3cf4d000cbe341f52545007
Author: gatorsmile <[email protected]>
Date: 2015-11-20T18:56:45Z
converge
commit b37a64f13956b6ddd0e38ddfd9fe1caee611f1a8
Author: gatorsmile <[email protected]>
Date: 2015-11-20T18:58:37Z
converge
commit bde74f83e24c2dc9bd9fd9e5541362049594c972
Author: gatorsmile <[email protected]>
Date: 2016-01-03T17:45:38Z
Merge remote-tracking branch 'upstream/master' into
OuterJoinEliminationByParentJoinPredicate
commit e18ba758aa94cc75115cc689f49b75ccd5d0ce51
Author: gatorsmile <[email protected]>
Date: 2016-01-04T02:21:59Z
Merge remote-tracking branch 'upstream/master' into
OuterJoinEliminationByParentJoinPredicate
commit d6a6e9cc31b0f7547b35cf25884135ea65b03676
Author: gatorsmile <[email protected]>
Date: 2016-01-04T02:40:26Z
outer join elimination by parent join.
commit c2a872c226dd3ca59c12ee74ff7b12372be639fe
Author: gatorsmile <[email protected]>
Date: 2016-01-06T04:09:27Z
Merge remote-tracking branch 'upstream/master'
commit ab6dbd74421fde3690c53c80c9873510d70174e5
Author: gatorsmile <[email protected]>
Date: 2016-01-06T19:59:17Z
Merge remote-tracking branch 'upstream/master'
commit 4276356305dc70585e0d76bc271da59cfd446867
Author: gatorsmile <[email protected]>
Date: 2016-01-06T22:06:13Z
Merge remote-tracking branch 'upstream/master'
commit 2dab7087855aa7e3411a1399fb1535ab30f2aa14
Author: gatorsmile <[email protected]>
Date: 2016-01-07T15:28:57Z
Merge remote-tracking branch 'upstream/master'
commit 0458770af543520f3bc7d45289d8aef07b0a36e6
Author: gatorsmile <[email protected]>
Date: 2016-01-08T22:20:32Z
Merge remote-tracking branch 'upstream/master'
commit 1debdfa8d14a6330ab46ed4b62c71e4fb35ca286
Author: gatorsmile <[email protected]>
Date: 2016-01-09T05:31:06Z
Merge remote-tracking branch 'upstream/master'
commit 763706d8c13d21faba182a30c885af1e1b7c9ee0
Author: gatorsmile <[email protected]>
Date: 2016-01-14T07:01:13Z
Merge remote-tracking branch 'upstream/master'
commit 4de6ec1c78a726ea3bd85ec32c1038d4c1b2e713
Author: gatorsmile <[email protected]>
Date: 2016-01-18T22:13:58Z
Merge remote-tracking branch 'upstream/master'
commit 9422a4fc32fec9b956ae1db5e5c6f3599a230fbc
Author: gatorsmile <[email protected]>
Date: 2016-01-19T21:13:35Z
Merge remote-tracking branch 'upstream/master'
commit 52bdf48b2cae6322cdb6bcb8b543e20acbdb15fc
Author: gatorsmile <[email protected]>
Date: 2016-01-20T17:10:26Z
Merge remote-tracking branch 'upstream/master'
commit 1e95df34bd7d0ca8a0fa389db53c4bd030e7fa10
Author: gatorsmile <[email protected]>
Date: 2016-01-23T05:51:41Z
Merge remote-tracking branch 'upstream/master'
commit fab24cfa2e1a3ad8b1bde7aaa2dcba144df31e53
Author: gatorsmile <[email protected]>
Date: 2016-02-01T07:39:27Z
Merge remote-tracking branch 'upstream/master'
commit 8b2e33be439722313c8129197f72ffbb5f5a0b48
Author: gatorsmile <[email protected]>
Date: 2016-02-05T07:01:36Z
Merge remote-tracking branch 'upstream/master'
commit 2ee1876efe378de25011990fcaaa1bffc9aac5bc
Author: gatorsmile <[email protected]>
Date: 2016-02-11T01:53:14Z
Merge remote-tracking branch 'upstream/master'
commit b9f0090fdba30fa0c0bd41f720af05c71a6ddfc0
Author: gatorsmile <[email protected]>
Date: 2016-02-12T21:19:32Z
Merge remote-tracking branch 'upstream/master'
commit ade6f7e36eb8f2328af74f6a98a46a89b7d804d9
Author: gatorsmile <[email protected]>
Date: 2016-02-15T23:57:30Z
Merge remote-tracking branch 'upstream/master'
commit 9fd63d20c8636ed8145526c784083612b993ed6f
Author: gatorsmile <[email protected]>
Date: 2016-02-19T03:45:37Z
Merge remote-tracking branch 'upstream/master'
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
