[jira] [Updated] (SPARK-29059) [SPIP] Support for Hive Materialized Views in Spark SQL.
[ https://issues.apache.org/jira/browse/SPARK-29059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-29059: -- Affects Version/s: (was: 3.0.0) 3.1.0 > [SPIP] Support for Hive Materialized Views in Spark SQL. > > > Key: SPARK-29059 > URL: https://issues.apache.org/jira/browse/SPARK-29059 > Project: Spark > Issue Type: New Feature > Components: Spark Core >Affects Versions: 3.1.0 >Reporter: Amogh Margoor >Priority: Minor > > Materialized view was introduced in Apache Hive 3.0.0. Currently, Spark > Catalyst does not optimize queries against Hive tables using Materialized > View the way Apache Calcite does it for Hive. This Jira is to add support for > the same. > We have developed it in our internal trunk and would like to open source it. > It would consist of 3 major parts: > # Reading MV related Hive Metadata > # Implication Engine which would figure out if an expression exp1 implies > another expression exp2 i.e., if exp1 => exp2 is a tautology. This is similar > to RexImplication checker in Apache Calcite. > # Catalyst rule to replace tables by it's Materialized view using > Implication Engine. For e.g., if MV 'mv' has been created in Hive using query > 'select * from foo where x > 10 && x <110' then query 'select * from foo > where x > 70 and x < 100' will be transformed into 'select * from mv where x > >70 and x < 100' > Note that Implication Engine and Catalyst Rule is generic can be used even > when Spark decides to have it's own Materialized View. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-29059) [SPIP] Support for Hive Materialized Views in Spark SQL.
[ https://issues.apache.org/jira/browse/SPARK-29059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amogh Margoor updated SPARK-29059: -- Issue Type: New Feature (was: Task) > [SPIP] Support for Hive Materialized Views in Spark SQL. > > > Key: SPARK-29059 > URL: https://issues.apache.org/jira/browse/SPARK-29059 > Project: Spark > Issue Type: New Feature > Components: Spark Core >Affects Versions: 3.0.0 >Reporter: Amogh Margoor >Priority: Minor > > Materialized view was introduced in Apache Hive 3.0.0. Currently, Spark > Catalyst does not optimize queries against Hive tables using Materialized > View the way Apache Calcite does it for Hive. This Jira is to add support for > the same. > We have developed it in our internal trunk and would like to open source it. > It would consist of 3 major parts: > # Reading MV related Hive Metadata > # Implication Engine which would figure out if an expression exp1 implies > another expression exp2 i.e., if exp1 => exp2 is a tautology. This is similar > to RexImplication checker in Apache Calcite. > # Catalyst rule to replace tables by it's Materialized view using > Implication Engine. For e.g., if MV 'mv' has been created in Hive using query > 'select * from foo where x > 10 && x <110' then query 'select * from foo > where x > 70 and x < 100' will be transformed into 'select * from mv where x > >70 and x < 100' > Note that Implication Engine and Catalyst Rule is generic can be used even > when Spark decides to have it's own Materialized View. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-29059) [SPIP] Support for Hive Materialized Views in Spark SQL.
[ https://issues.apache.org/jira/browse/SPARK-29059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amogh Margoor updated SPARK-29059: -- Summary: [SPIP] Support for Hive Materialized Views in Spark SQL. (was: Support for Hive Materialized Views in Spark SQL.) > [SPIP] Support for Hive Materialized Views in Spark SQL. > > > Key: SPARK-29059 > URL: https://issues.apache.org/jira/browse/SPARK-29059 > Project: Spark > Issue Type: Task > Components: Spark Core >Affects Versions: 3.0.0 >Reporter: Amogh Margoor >Priority: Minor > > Materialized view was introduced in Apache Hive 3.0.0. Currently, Spark > Catalyst does not optimize queries against Hive tables using Materialized > View the way Apache Calcite does it for Hive. This Jira is to add support for > the same. > We have developed it in our internal trunk and would like to open source it. > It would consist of 3 major parts: > # Reading MV related Hive Metadata > # Implication Engine which would figure out if an expression exp1 implies > another expression exp2 i.e., if exp1 => exp2 is a tautology. This is similar > to RexImplication checker in Apache Calcite. > # Catalyst rule to replace tables by it's Materialized view using > Implication Engine. For e.g., if MV 'mv' has been created in Hive using query > 'select * from foo where x > 10 && x <110' then query 'select * from foo > where x > 70 and x < 100' will be transformed into 'select * from mv where x > >70 and x < 100' > Note that Implication Engine and Catalyst Rule is generic can be used even > when Spark decides to have it's own Materialized View. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
