[
https://issues.apache.org/jira/browse/SPARK-29038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16938374#comment-16938374
]
Lantao Jin commented on SPARK-29038:
------------------------------------
[~smilegator], yes, parquet does not good for incremental refresh. Current our
implementation is based on parquet with entire refresh. But it is not strongly
coupled to parquet. Delta would be an option if enable incremental refresh. I
will investigate it.
[~amargoor] I didn't see any SPIP ticket for it. But I personally +1 for
letting Catalyst to recognizing Hive materialized views. I can help to review
when it's done. In the meantime, I think it doesn't conflict with building a
Spark native materialized view. We have enable this feature on production. Hope
could get more inputs from community about this.
> SPIP: Support Spark Materialized View
> -------------------------------------
>
> Key: SPARK-29038
> URL: https://issues.apache.org/jira/browse/SPARK-29038
> Project: Spark
> Issue Type: New Feature
> Components: SQL
> Affects Versions: 3.0.0
> Reporter: Lantao Jin
> Priority: Major
>
> Materialized view is an important approach in DBMS to cache data to
> accelerate queries. By creating a materialized view through SQL, the data
> that can be cached is very flexible, and needs to be configured arbitrarily
> according to specific usage scenarios. The Materialization Manager
> automatically updates the cache data according to changes in detail source
> tables, simplifying user work. When user submit query, Spark optimizer
> rewrites the execution plan based on the available materialized view to
> determine the optimal execution plan.
> Details in [design
> doc|https://docs.google.com/document/d/1q5pjSWoTNVc9zsAfbNzJ-guHyVwPsEroIEP8Cca179A/edit?usp=sharing]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
