GitHub user dilipbiswal opened a pull request:
https://github.com/apache/spark/pull/16954
[SPARK-18874][SQL] First phase: Deferring the correlated predicate pull up
to Optimizer phase
## What changes were proposed in this pull request?
Currently Analyzer as part of ResolveSubquery, pulls up the correlated
predicates to its
originating SubqueryExpression. The subquery plan is then transformed to
remove the correlated
predicates after they are moved up to the outer plan. In this PR, the task
of pulling up
correlated predicates is deferred to Optimizer. This is the initial work
that will allow us to
support the form of correlated subqueries that we don't support today. The
design document
from @nsyca can be found in the following link :
[DesignDoc](https://docs.google.com/document/d/1QDZ8JwU63RwGFS6KVF54Rjj9ZJyK33d49ZWbjFBaIgU/edit#)
The brief description of code changes (hopefully to aid with code review)
can be be found in the
following link:
[CodeChanges](https://docs.google.com/document/d/18mqjhL9V1An-tNta7aVE13HkALRZ5GZ24AATA-Vqqf0/edit#)
## How was this patch tested?
The test case PRs were submitted earlier using.
[16337](https://github.com/apache/spark/pull/16337)
[16759](https://github.com/apache/spark/pull/16759)
[16841](https://github.com/apache/spark/pull/16841)
[16915](https://github.com/apache/spark/pull/16915)
[16798](https://github.com/apache/spark/pull/16798)
[16712](https://github.com/apache/spark/pull/16712)
[16710](https://github.com/apache/spark/pull/16710)
[16760](https://github.com/apache/spark/pull/16760)
[16802](https://github.com/apache/spark/pull/16802)
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/dilipbiswal/spark SPARK-18874
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/16954.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #16954
----
commit 33844bb8c6e72c35e12a7ea9f124ad3c6b8bd43c
Author: Dilip Biswal <[email protected]>
Date: 2017-02-13T23:25:39Z
[SPARK-18874] First phase: Deferring the correlated predicate pull up to
Optimizer phase
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
