[
https://issues.apache.org/jira/browse/SPARK-11012?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yin Huai resolved SPARK-11012.
------------------------------
Resolution: Fixed
Fix Version/s: 2.0.0
> Canonicalize view definitions
> -----------------------------
>
> Key: SPARK-11012
> URL: https://issues.apache.org/jira/browse/SPARK-11012
> Project: Spark
> Issue Type: New Feature
> Components: SQL
> Affects Versions: 2.0.0
> Reporter: Yin Huai
> Fix For: 2.0.0
>
>
> In SPARK-10337, we added the first step of supporting view natively, which is
> basically wrapping the original view definition SQL text with an extra
> {{SELECT}} and then store the wrapped SQL text into metastore. This approach
> suffers at least two issues:
> # Switching current database may break view queries
> # HiveQL doesn't allow CTE as subquery, thus CTE can't be used in view
> definition
> To fix these issues, we need to canonicalize the view definition. For
> example, for a SQL string
> {code:sql}
> SELECT a, b FROM table
> {code}
> we will save this text to Hive metastore as
> {code:sql}
> SELECT `table`.`a`, `table`.`b` FROM `currentDB`.`table`
> {code}
> The core infrastructure of this work is SQL query string generation
> (SPARK-12593). Namely, converting resolved logical query plans back to
> canonicalized SQL query strings. [PR
> #10541|https://github.com/apache/spark/pull/10541] set up basic
> infrastructure of SQL generation, but more language structures need to be
> supported.
> [PR #10541|https://github.com/apache/spark/pull/10541] added round-trip
> testing infrastructure for SQL generation. All queries tested by test suites
> extending {{HiveComparisonTest}} are executed in the following order:
> # Parsing query string to logical plan
> # Converting resolved logical plan back to canonicalized SQL query string
> # Executing generated SQL query string
> # Comparing query results with golden answers
> Note that not all resolved logical query plan can be converted back to SQL
> query string. Either because it consists of some language structure that has
> not been supported yet, or it doesn't have a SQL representation inherently
> (e.g. query plans built on top of local Scala collections).
> If a logical plan is inconvertible, {{HiveComparisonTest}} falls back to its
> original behavior, namely executing the original SQL query string and compare
> the results with golden answers.
> SQL generation details are logged and can be found in
> {{sql/hive/target/unit-tests.log}} (log level should be at least DEBUG).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
