GitHub user andrewor14 opened a pull request:
https://github.com/apache/spark/pull/11069
[SPARK-13079] Extend Catalog API + implement InMemoryCatalog
This patch extends the existing Catalog API added in #10982 to include
methods for handling table partitions. In particular, a partition is identified
by `PartitionSpec`, which is just a `Map[String, String]`. The Catalog is still
not used by anything yet, but its API is now more or less complete and an
implementation is fully tested.
About 200 lines are test code.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/andrewor14/spark catalog
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/11069.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #11069
----
commit 38166af20845e0356e366f697a2e1dac36acc31c
Author: Andrew Or <[email protected]>
Date: 2016-02-03T22:28:31Z
Add partitions related methods to Catalog
These are a subset of the public interfaces exposed by Hive.
This commit just adds the skeleton without implementing any of
them.
commit 461960b4a6b656f3d1a54d4d1d72b3a53e892718
Author: Andrew Or <[email protected]>
Date: 2016-02-03T23:03:04Z
Modify partition method based on feedback
commit 1000646adf36ed36cc8dc6f0e6c7840938a0d4fe
Author: Andrew Or <[email protected]>
Date: 2016-02-04T00:36:02Z
Implement new partition methods in InMemoryCatalog
commit 1b40002c215445460d9c42d992736dd29b5ef700
Author: Andrew Or <[email protected]>
Date: 2016-02-04T01:35:00Z
Add tests for functions and partitions methods
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
