[
https://issues.apache.org/jira/browse/HIVE-30227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Denys Kuzmenko updated HIVE-30227:
----------------------------------
Description:
An incremental rebuild of a materialized view in insert-delete mode
(AGGREGATE_INSERT_DELETE_REBUILD: a V3 Iceberg materialized view with count(*)
over full-ACID sources) has two DELETE branches into the view: one for the
groups it updates and one for the groups whose count drops to 0. Each branch
gets its own DISTRIBUTE BY FILE__PATH, reducer and DV writer, so a data file
holding rows of both gets two DVs in one commit, which Iceberg then merges at
commit time (merged-dvs). HIVE-30112 made every other merge-on-read DML write
one DV per data file.
*Fix*
Combine the two delete branches of the rebuild into one, as SplitMergeRewriter
does for a MERGE with UPDATE and DELETE clauses
(AlterMaterializedViewRebuildAnalyzer: the two DestClausePrefix.DELETE
branches).
Found in the review of HIVE-30112.
> Iceberg: a V3 materialized view rebuild in insert-delete mode writes two DVs
> per data file
> ------------------------------------------------------------------------------------------
>
> Key: HIVE-30227
> URL: https://issues.apache.org/jira/browse/HIVE-30227
> Project: Hive
> Issue Type: Improvement
> Reporter: Denys Kuzmenko
> Priority: Major
>
> An incremental rebuild of a materialized view in insert-delete mode
> (AGGREGATE_INSERT_DELETE_REBUILD: a V3 Iceberg materialized view with
> count(*) over full-ACID sources) has two DELETE branches into the view: one
> for the groups it updates and one for the groups whose count drops to 0. Each
> branch gets its own DISTRIBUTE BY FILE__PATH, reducer and DV writer, so a
> data file holding rows of both gets two DVs in one commit, which Iceberg then
> merges at commit time (merged-dvs). HIVE-30112 made every other merge-on-read
> DML write one DV per data file.
> *Fix*
> Combine the two delete branches of the rebuild into one, as
> SplitMergeRewriter does for a MERGE with UPDATE and DELETE clauses
> (AlterMaterializedViewRebuildAnalyzer: the two DestClausePrefix.DELETE
> branches).
> Found in the review of HIVE-30112.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
