GitHub user cloud-fan opened a pull request:
https://github.com/apache/spark/pull/16645
[SPARK-19290][SQL] add post-hoc resolution
## What changes were proposed in this pull request?
To implement DDL commands, we added several analyzer rules in sql/hive
module to analyze DDL related plans. However, our `Analyzer` currently only
have one extending interface: `extendedResolutionRules`, which defines extra
rules that will be run together with other rules in the resolution batch, and
doesn't fit DDL rules well, because:
1. DDL rules may do some checking and normalization, but we may do it many
times as the resolution batch will run rules again and again, until fixed
point, and it's hard to tell if a DDL rule has already done its checking and
normalization. It's fine because DDL rules are idempotent, but it's bad for
analysis performance
2. some DDL rules may depend on others, and it's pretty hard to write `if`
conditions to guarantee the dependencies. It will be good if we have a batch
which run rules in one pass, so that we can guarantee the dependencies by rules
order.
This PR adds a new extending interface in `Analyzer`:
`postHocResolutionRules`, which defines rules that will be run only once in a
batch runs right after the resolution batch.
## How was this patch tested?
existing tests
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/cloud-fan/spark analyzer
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/16645.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #16645
----
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
