[
https://issues.apache.org/jira/browse/HIVE-30227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Denys Kuzmenko updated HIVE-30227:
----------------------------------
Description:
An incremental rebuild of a materialized view in insert-delete mode
(AGGREGATE_INSERT_DELETE_REBUILD: a V3 Iceberg materialized view with count(*)
over full-ACID sources) deletes through two DELETE branches: one for the groups
it updates and one for the groups whose count drops to 0. Neither branch is
distributed by FILE__PATH, and a data file can hold rows of both, so a data
file can get several DVs in one commit, which Iceberg then merges at commit
time (merged-dvs). HIVE-30112 made DELETE, UPDATE and MERGE write one DV per
data file.
*Fix*
- Distribute the delete branches of the rebuild by
HiveStorageHandler#acidDistributeColumns, as the DML rewriters do
(AlterMaterializedViewRebuildAnalyzer, MaterializedViewASTBuilder,
NonNativeAcidMaterializedViewASTBuilder); the first commit of the HIVE-30112 PR
(#6924) has this part with iceberg_mv_rebuild_dv_per_file.q.
- Combine the two delete branches into one, as SplitMergeRewriter does for a
MERGE with UPDATE and DELETE clauses.
Found in the review of HIVE-30112.
was:
An incremental rebuild of a materialized view in insert-delete mode
(AGGREGATE_INSERT_DELETE_REBUILD: a V3 Iceberg materialized view with count(*)
over full-ACID sources) has two DELETE branches into the view: one for the
groups it updates and one for the groups whose count drops to 0. Each branch
gets its own DISTRIBUTE BY FILE__PATH, reducer and DV writer, so a data file
holding rows of both gets two DVs in one commit, which Iceberg then merges at
commit time (merged-dvs). HIVE-30112 made every other merge-on-read DML write
one DV per data file.
*Fix*
Combine the two delete branches of the rebuild into one, as SplitMergeRewriter
does for a MERGE with UPDATE and DELETE clauses
(AlterMaterializedViewRebuildAnalyzer: the two DestClausePrefix.DELETE
branches).
Found in the review of HIVE-30112.
> Iceberg: a V3 materialized view rebuild in insert-delete mode writes two DVs
> per data file
> ------------------------------------------------------------------------------------------
>
> Key: HIVE-30227
> URL: https://issues.apache.org/jira/browse/HIVE-30227
> Project: Hive
> Issue Type: Bug
> Reporter: Denys Kuzmenko
> Priority: Major
>
> An incremental rebuild of a materialized view in insert-delete mode
> (AGGREGATE_INSERT_DELETE_REBUILD: a V3 Iceberg materialized view with
> count(*) over full-ACID sources) deletes through two DELETE branches: one for
> the groups it updates and one for the groups whose count drops to 0. Neither
> branch is distributed by FILE__PATH, and a data file can hold rows of both,
> so a data file can get several DVs in one commit, which Iceberg then merges
> at commit time (merged-dvs). HIVE-30112 made DELETE, UPDATE and MERGE write
> one DV per data file.
> *Fix*
> - Distribute the delete branches of the rebuild by
> HiveStorageHandler#acidDistributeColumns, as the DML rewriters do
> (AlterMaterializedViewRebuildAnalyzer, MaterializedViewASTBuilder,
> NonNativeAcidMaterializedViewASTBuilder); the first commit of the HIVE-30112
> PR (#6924) has this part with iceberg_mv_rebuild_dv_per_file.q.
> - Combine the two delete branches into one, as SplitMergeRewriter does for a
> MERGE with UPDATE and DELETE clauses.
> Found in the review of HIVE-30112.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
