GitHub user cloud-fan opened a pull request:
https://github.com/apache/spark/pull/16269
[WIP] simplify data source analysis
## What changes were proposed in this pull request?
The current way of resolving `InsertIntoTable` and `CreateTable` is
convoluted: sometimes we replace them with concrete implementation commands
during analysis, sometimes during planning phase.
And the error checking logic is also a mess: we may put it in extended
analyzer rules, or extended checking rules, or `CheckAnalysis`.
This PR simplifies the data source analysis:
1. `InsertIntoTable` and `CreateTable` are always unresolved and need to
be replaced by concrete implementation commands during analysis.
2. The error checking logic is mainly in 2 rules: `PreprocessTableCreation`
and `PreprocessTableInsertion`.
This PR is based on https://github.com/apache/spark/pull/15996, please
review the last commit.
## How was this patch tested?
existing test.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/cloud-fan/spark ddl
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/16269.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #16269
----
commit 30e72e384f0134780ea256c453ff1b2c4bf662d6
Author: Wenchen Fan <[email protected]>
Date: 2016-11-23T16:37:04Z
remove OverwriteOptions
commit b5f43946fd72932f7e23ac1f1b3866b150fe745b
Author: Wenchen Fan <[email protected]>
Date: 2016-11-28T05:45:13Z
fix test
commit f67193912ccf40f8c61da2410671b2d388e764bb
Author: Wenchen Fan <[email protected]>
Date: 2016-12-11T12:20:51Z
Merge remote-tracking branch 'origin/master' into overwrite
commit 323a97c39541f2a6093d7b687b7f79993fc7391a
Author: Wenchen Fan <[email protected]>
Date: 2016-12-11T12:36:00Z
simplify
commit ed548e6437d64e67545f6bcb60384eb8badb4cec
Author: Wenchen Fan <[email protected]>
Date: 2016-12-13T05:20:52Z
add comment
commit 172f6eb5eeb36819aaf731c547540c5af90c49cc
Author: Wenchen Fan <[email protected]>
Date: 2016-12-12T14:27:39Z
Simplify CreateDataSourceTableAsSelectCommand
commit c075191c842205eaab9567b4143c8f41008d601d
Author: Wenchen Fan <[email protected]>
Date: 2016-12-13T18:35:05Z
simplify data source analysis
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
