GitHub user xwu0226 opened a pull request:
https://github.com/apache/spark/pull/16626
[SPARK-19261][SQL] Alter add columns for Hive tables
## What changes were proposed in this pull request?
Support ALTER TABLE ADD COLUMNS (...) syntax for Hive tables.
In this PR, we consider a few aspects:
1. View is not supported for alter add
2. Since tables created in SparkSQL with Hive DDL syntax will populate
table properties with schema information, we need make sure the consistency of
the schema before and after ALTER operation in order for future use. Because of
this issue, there will be a new PR need to support ALTER TABLE ADD COLUMNS for
datasource tables, carefully.
3. For embedded-schema type of format, such as `parquet`, we need to make
sure that the predicate on the newly-added columns can be evaluated properly,
or pushed down properly. In case of the data file does not have the columns for
the newly-added columns, such predicates should return as if the column values
are NULLs.
## How was this patch tested?
Add new test cases
Please review http://spark.apache.org/contributing.html before opening a
pull request.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/xwu0226/spark alter_add_columns
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/16626.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #16626
----
commit dba7dcdda36568b910b57de9bea368cec6d793dd
Author: Xin Wu <[email protected]>
Date: 2016-11-21T18:41:15Z
alter_add_col: initial changes
commit 3750541f7ccd73c7f92fb916e16c1ab4575ec140
Author: Xin Wu <[email protected]>
Date: 2016-12-01T22:03:28Z
add testcases
commit f7d9f031303d273daa8973517c6a09684e3ba52f
Author: Xin Wu <[email protected]>
Date: 2016-12-01T23:00:38Z
negative testcases
commit 4cd7848c291ff019e17a3841c755d5befc58178a
Author: Xin Wu <[email protected]>
Date: 2016-12-05T18:35:14Z
remove non-support testcase
commit 29347f6f52409e8c330101af8dba9bc335786c34
Author: Xin Wu <[email protected]>
Date: 2016-12-05T20:31:55Z
fix testcase
commit 7cc03483f3546a1d8d38bf67a347bea261e0a1b6
Author: Xin Wu <[email protected]>
Date: 2016-12-07T21:56:40Z
update testcases
commit 0272fbad90815c3d92f5d06ff3a11299305630e3
Author: Xin Wu <[email protected]>
Date: 2016-12-07T23:10:54Z
update testcases
commit 96fb6777e74435bc96baac5fa2d0a17183e3f363
Author: Xin Wu <[email protected]>
Date: 2017-01-13T20:53:44Z
update testcases
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
