GitHub user andrewor14 opened a pull request:
https://github.com/apache/spark/pull/11750
[SPARK-13923] Implement SessionCatalog
## What changes were proposed in this pull request?
As part of the effort to merge `SQLContext` and `HiveContext`, this patch
implements an internal catalog called `SessionCatalog` that handles temporary
functions and tables and delegates metastore operations to `ExternalCatalog`.
Currently, this is still dead code, but in the future it will be part of
`SessionState` and will replace `o.a.s.sql.catalyst.analysis.Catalog`.
A recent patch #11573 parses Hive commands ourselves in Spark, but still
passes the entire query text to Hive. In a future patch, we will use
`SessionCatalog` to implement the parsed commands.
## How was this patch tested?
700+ lines of tests in `SessionCatalogSuite`.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/andrewor14/spark temp-catalog
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/11750.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #11750
----
commit 4da28a57e65700b24887acb36330ca5d4ba5703b
Author: Andrew Or <[email protected]>
Date: 2016-03-11T00:35:35Z
Merge in @yhuai's changes
commit d09178cc5407f21be595151b258416f9b8fb0b77
Author: Andrew Or <[email protected]>
Date: 2016-03-14T23:27:11Z
Clean up table method signatures + add comments
commit 210b476f4c211652841fa791191db173c1f5a60f
Author: Andrew Or <[email protected]>
Date: 2016-03-15T00:14:59Z
Do the same for functions and partitions
commit 0a25e34773a4b4ba077c3546923a3b12e61064d0
Author: Andrew Or <[email protected]>
Date: 2016-03-15T20:53:42Z
Take into account current database in table methods
commit 19047521e6a95471641b13c8f3d64429b82e9c80
Author: Andrew Or <[email protected]>
Date: 2016-03-15T21:22:37Z
Refactor CatalogTable to use TableIdentifier
This is a standalone commit such that in the future we can split
it out into a separate patch if preferrable.
commit db2755d150b737eae96f698fd3481f6f5f607c61
Author: Andrew Or <[email protected]>
Date: 2016-03-15T21:33:20Z
Refactor CatalogFunction to use FunctionIdentifier
commit 132e7fc51750a18e1aaf7dc780df7199088a1704
Author: Andrew Or <[email protected]>
Date: 2016-03-15T21:38:50Z
Fix tests
commit 3926738b393655188bc679030b55c14ed135014b
Author: Andrew Or <[email protected]>
Date: 2016-03-15T21:39:22Z
Document and clean up function methods
commit 76d24ad8806fd409c8f099c38b0781cea6ba3eb2
Author: Andrew Or <[email protected]>
Date: 2016-03-15T22:32:38Z
Implement SessionCatalog using ExternalCatalog
commit 8f6b67abd92c4866aaae3f742160b261239c50d7
Author: Andrew Or <[email protected]>
Date: 2016-03-15T22:33:30Z
Fix style
commit cd606e2cff446aaecff6b6cb74382ff3af35b474
Author: Andrew Or <[email protected]>
Date: 2016-03-15T22:44:23Z
Clean up duplicate code in Table/FunctionIdentifier
commit a5a4b0cd9b253f5297fc9c19568820c9626ea65e
Author: Andrew Or <[email protected]>
Date: 2016-03-15T23:12:55Z
Refactor CatalogTestCases to make methods accessible
commit 1b905bb2bf3bb70bd438b467352a890c1568d712
Author: Andrew Or <[email protected]>
Date: 2016-03-15T23:59:38Z
Fix infinite loop (woops)
commit a7951a9ab1ca9d008572b82431afab96232d5a58
Author: Andrew Or <[email protected]>
Date: 2016-03-16T01:52:30Z
Implement tests for databases and tables
commit f33c8ef44fdd38ca9e5c35b421e1ddb8d93f1de6
Author: Andrew Or <[email protected]>
Date: 2016-03-16T02:04:03Z
Implement tests for table partitions
commit 594390bcbfb6d806417dceedb2c24091950dd7c9
Author: Andrew Or <[email protected]>
Date: 2016-03-16T02:41:30Z
Implement tests for functions
commit 3b2e48a4efa81bb886331ce5818e2d44c7c2f7da
Author: Andrew Or <[email protected]>
Date: 2016-03-16T02:42:22Z
Add TODO
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
