[
https://issues.apache.org/jira/browse/SPARK-29059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun updated SPARK-29059:
----------------------------------
Affects Version/s: (was: 3.0.0)
3.1.0
> [SPIP] Support for Hive Materialized Views in Spark SQL.
> --------------------------------------------------------
>
> Key: SPARK-29059
> URL: https://issues.apache.org/jira/browse/SPARK-29059
> Project: Spark
> Issue Type: New Feature
> Components: Spark Core
> Affects Versions: 3.1.0
> Reporter: Amogh Margoor
> Priority: Minor
>
> Materialized view was introduced in Apache Hive 3.0.0. Currently, Spark
> Catalyst does not optimize queries against Hive tables using Materialized
> View the way Apache Calcite does it for Hive. This Jira is to add support for
> the same.
> We have developed it in our internal trunk and would like to open source it.
> It would consist of 3 major parts:
> # Reading MV related Hive Metadata
> # Implication Engine which would figure out if an expression exp1 implies
> another expression exp2 i.e., if exp1 => exp2 is a tautology. This is similar
> to RexImplication checker in Apache Calcite.
> # Catalyst rule to replace tables by it's Materialized view using
> Implication Engine. For e.g., if MV 'mv' has been created in Hive using query
> 'select * from foo where x > 10 && x <110' then query 'select * from foo
> where x > 70 and x < 100' will be transformed into 'select * from mv where x
> >70 and x < 100'
> Note that Implication Engine and Catalyst Rule is generic can be used even
> when Spark decides to have it's own Materialized View.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
