Repository: spark Updated Branches: refs/heads/master 9cf9a83af -> 49bea5a7e
[SPARK-25833][SQL][DOCS] Update migration guide for Hive view compatibility ## What changes were proposed in this pull request? Both Spark and Hive support views. However in some cases views created by Hive are not readable by Spark. For example, if column aliases are not specified in view definition queries, both Spark and Hive will generate alias names, but in different ways. In order for Spark to be able to read views created by Hive, users should explicitly specify column aliases in view definition queries. Given that it's not uncommon that Hive and Spark are used together in enterprise data warehouse, this PR aims to explicitly describe this compatibility issue to help users troubleshoot this issue easily. ## How was this patch tested? Docs are manually generated and checked locally. ``` SKIP_API=1 jekyll serve ``` Closes #22868 from seancxmao/SPARK-25833. Authored-by: seancxmao <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]> Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/49bea5a7 Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/49bea5a7 Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/49bea5a7 Branch: refs/heads/master Commit: 49bea5a7e87ec3ce9cd9466725d81096a54a591b Parents: 9cf9a83 Author: seancxmao <[email protected]> Authored: Tue Oct 30 23:05:31 2018 -0700 Committer: Dongjoon Hyun <[email protected]> Committed: Tue Oct 30 23:05:31 2018 -0700 ---------------------------------------------------------------------- docs/sql-migration-guide-hive-compatibility.md | 15 +++++++++++++++ 1 file changed, 15 insertions(+) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/spark/blob/49bea5a7/docs/sql-migration-guide-hive-compatibility.md ---------------------------------------------------------------------- diff --git a/docs/sql-migration-guide-hive-compatibility.md b/docs/sql-migration-guide-hive-compatibility.md index 0234ea2..9484941 100644 --- a/docs/sql-migration-guide-hive-compatibility.md +++ b/docs/sql-migration-guide-hive-compatibility.md @@ -51,6 +51,21 @@ Spark SQL supports the vast majority of Hive features, such as: * Explain * Partitioned tables including dynamic partition insertion * View + * If column aliases are not specified in view definition queries, both Spark and Hive will + generate alias names, but in different ways. In order for Spark to be able to read views created + by Hive, users should explicitly specify column aliases in view definition queries. As an + example, Spark cannot read `v1` created as below by Hive. + + ``` + CREATE VIEW v1 AS SELECT * FROM (SELECT c + 1 FROM (SELECT 1 c) t1) t2; + ``` + + Instead, you should create `v1` as below with column aliases explicitly specified. + + ``` + CREATE VIEW v1 AS SELECT * FROM (SELECT c + 1 AS inc_c FROM (SELECT 1 c) t1) t2; + ``` + * All Hive DDL Functions, including: * `CREATE TABLE` * `CREATE TABLE AS SELECT` --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
