[
https://issues.apache.org/jira/browse/HIVE-30251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Denys Kuzmenko updated HIVE-30251:
----------------------------------
Description:
*Problem*
An incremental rebuild of an Iceberg materialized view with
{{write.merge.mode=copy-on-write}} keeps the rows it replaces: every updated
group keeps its old row, twice, next to its new one. There is no
error.
{{AlterMaterializedViewRebuildAnalyzer}} rewrites the incremental rebuild
into a multi-insert with a DELETE branch, under {{Context.Operation.MERGE}},
whatever the write mode of the view. With
{{write.merge.mode=copy-on-write}} the file sink of that branch is
copy-on-write, so {{HiveIcebergCopyOnWriteRecordWriter}} writes the rows to
delete back as data and marks no data file as replaced: the old
file stays, and its rows are written again.
A partitioned view with {{write.merge.mode=copy-on-write}} and
{{write.delete.mode=merge-on-read}} fails to compile since HIVE-30112: the
partition expressions of the DELETE branch are positioned for a
merge-on-read delete row, but the row has the copy-on-write layout. Before
HIVE-30112 it compiled and kept the stale rows as above.
*Reproducer*
{code:sql}
set hive.support.concurrency=true;
set hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DbTxnManager;
set hive.materializedview.rebuild.incremental.factor=0.01;
create external table src_ice(a int, b int) stored by iceberg stored as orc
tblproperties ('format-version'='2');
insert into src_ice values (1, 1), (2, 2), (3, 2);
create materialized view mv_cow stored by iceberg stored as orc tblproperties
('format-version'='2',
'write.merge.mode'='copy-on-write', 'write.delete.mode'='copy-on-write',
'write.update.mode'='copy-on-write') as
select b, count(*) cnt, sum(a) s from src_ice group by b;
create materialized view mv_mixed partitioned on (b) stored by iceberg stored
as orc tblproperties ('format-version'='2',
'write.merge.mode'='copy-on-write', 'write.delete.mode'='merge-on-read') as
select count(*) cnt, sum(a) s, b from src_ice group by b;
insert into src_ice values (4, 1), (5, 3);
alter materialized view mv_cow rebuild;
select * from mv_cow;
alter materialized view mv_mixed rebuild;
{code}
{{select * from mv_cow}}:
- expected: (1, 2, 5), (2, 2, 5), (3, 1, 5)
- actual: (1, 1, 1), (1, 1, 1), (1, 2, 5), (2, 2, 5), (3, 1, 5)
The same view with the default merge-on-read modes rebuilds correctly.
{{alter materialized view mv_mixed rebuild}}:
{noformat}
java.lang.IndexOutOfBoundsException: Index 9 out of bounds for length 7
at
org.apache.iceberg.mr.hive.IcebergTransformSortFunctionUtil.lambda$getCustomTransformExpr$12(IcebergTransformSortFunctionUtil.java:154)
at
org.apache.hadoop.hive.ql.optimizer.SortedDynPartitionOptimizer$SortedDynamicPartitionProc.allStaticPartitions(SortedDynPartitionOptimizer.java:435)
{noformat}
*Proposed fix*
Rebuild a materialized view with {{write.merge.mode=copy-on-write}} fully
instead of incrementally, or rewrite its incremental rebuild with the
copy-on-write MERGE rewriter. Either removes the DELETE branch
under a copy-on-write MERGE, and with it the compile failure.
> Iceberg: Incremental rebuild of a copy-on-write materialized view keeps the
> rows it replaces
> ---------------------------------------------------------------------------------------------
>
> Key: HIVE-30251
> URL: https://issues.apache.org/jira/browse/HIVE-30251
> Project: Hive
> Issue Type: Bug
> Reporter: Denys Kuzmenko
> Priority: Major
>
> *Problem*
>
>
> An incremental rebuild of an Iceberg materialized view with
> {{write.merge.mode=copy-on-write}} keeps the rows it replaces: every updated
> group keeps its old row, twice, next to its new one. There is no
> error.
>
>
>
>
>
> {{AlterMaterializedViewRebuildAnalyzer}} rewrites the incremental rebuild
> into a multi-insert with a DELETE branch, under {{Context.Operation.MERGE}},
> whatever the write mode of the view. With
> {{write.merge.mode=copy-on-write}} the file sink of that branch is
> copy-on-write, so {{HiveIcebergCopyOnWriteRecordWriter}} writes the rows to
> delete back as data and marks no data file as replaced: the old
> file stays, and its rows are written again.
>
>
>
>
>
> A partitioned view with {{write.merge.mode=copy-on-write}} and
> {{write.delete.mode=merge-on-read}} fails to compile since HIVE-30112: the
> partition expressions of the DELETE branch are positioned for a
> merge-on-read delete row, but the row has the copy-on-write layout. Before
> HIVE-30112 it compiled and kept the stale rows as above.
>
>
>
>
> *Reproducer*
>
>
> {code:sql}
>
>
> set hive.support.concurrency=true;
>
>
> set hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DbTxnManager;
>
>
> set hive.materializedview.rebuild.incremental.factor=0.01;
>
> create external table src_ice(a int, b int) stored by iceberg stored as orc
> tblproperties ('format-version'='2');
>
> insert into src_ice values (1, 1), (2, 2), (3, 2);
>
>
>
>
>
> create materialized view mv_cow stored by iceberg stored as orc
> tblproperties ('format-version'='2',
>
> 'write.merge.mode'='copy-on-write', 'write.delete.mode'='copy-on-write',
> 'write.update.mode'='copy-on-write') as
>
> select b, count(*) cnt, sum(a) s from src_ice group by b;
>
>
>
>
>
> create materialized view mv_mixed partitioned on (b) stored by iceberg
> stored as orc tblproperties ('format-version'='2',
>
> 'write.merge.mode'='copy-on-write', 'write.delete.mode'='merge-on-read')
> as
>
> select count(*) cnt, sum(a) s, b from src_ice group by b;
>
>
>
>
>
> insert into src_ice values (4, 1), (5, 3);
>
>
>
>
>
> alter materialized view mv_cow rebuild;
>
>
> select * from mv_cow;
>
>
>
>
>
> alter materialized view mv_mixed rebuild;
>
>
> {code}
>
>
>
>
>
> {{select * from mv_cow}}:
>
>
> - expected: (1, 2, 5), (2, 2, 5), (3, 1, 5)
>
>
> - actual: (1, 1, 1), (1, 1, 1), (1, 2, 5), (2, 2, 5), (3, 1, 5)
> The same view with the default merge-on-read modes rebuilds correctly.
>
>
>
>
>
> {{alter materialized view mv_mixed rebuild}}:
>
>
> {noformat}
>
>
> java.lang.IndexOutOfBoundsException: Index 9 out of bounds for length 7
>
>
> at
> org.apache.iceberg.mr.hive.IcebergTransformSortFunctionUtil.lambda$getCustomTransformExpr$12(IcebergTransformSortFunctionUtil.java:154)
>
> at
> org.apache.hadoop.hive.ql.optimizer.SortedDynPartitionOptimizer$SortedDynamicPartitionProc.allStaticPartitions(SortedDynPartitionOptimizer.java:435)
>
> {noformat}
>
>
>
>
>
> *Proposed fix*
>
>
> Rebuild a materialized view with {{write.merge.mode=copy-on-write}} fully
> instead of incrementally, or rewrite its incremental rebuild with the
> copy-on-write MERGE rewriter. Either removes the DELETE branch
> under a copy-on-write MERGE, and with it the compile failure.
>
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
