GitHub user tejasapatil opened a pull request:
https://github.com/apache/spark/pull/13059
[SPARK-15275] [SQL] CatalogTable should store sort ordering for sorted
columns
## What changes were proposed in this pull request?
Jira link : https://issues.apache.org/jira/browse/SPARK-15275
For bucketed tables in Hive, one can also add constraint about column
sortedness along with ordering.
As per the spec in [0], CREATE TABLE statement can allow SORT ordering as
well:
[CLUSTERED BY (col_name, col_name, ...) [SORTED BY (col_name [ASC|DESC],
...)] INTO num_buckets BUCKETS]
[0] :
https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-CreateTable
Currently CatalogTable does not store any information about the sort
ordering and just has the names of the sorted columns. This PR adds
`CatalogSortOrder` to hold the sorted column name and the sorted order.
Currently this information is *not* used in query execution but can be used as
more support for bucketing is added. Possible advantage is ability to skip rows
while performing predicate matching.
## How was this patch tested?
Currently trunk does support creating bucketed hive tables. I am relying on
existing tests.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/tejasapatil/spark sort_order
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/13059.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #13059
----
commit 902dd8e23ab56cb202c1d9193f36a2f48c7a13c1
Author: Tejas Patil <[email protected]>
Date: 2016-05-09T06:08:53Z
Metastore relation to save table bucketing information
commit c2f81ca5447b4e928c666a8aae1667d54dc35378
Author: Tejas Patil <[email protected]>
Date: 2016-05-10T18:57:13Z
CatalogTable should include sorted columns ordering information
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
