[ 
https://issues.apache.org/jira/browse/HIVE-30251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Denys Kuzmenko updated HIVE-30251:
----------------------------------
    Description: 
*Problem*                                                                       
                                                                                
                                               
  An incremental rebuild of an Iceberg materialized view with 
{{write.merge.mode=copy-on-write}} keeps the rows it replaces: every updated 
group keeps its old row, twice, next to its new one. There is no      
  error.                                                                        
                                                                                
                                                 
                                                                                
                                                                                
                                                 
  {{AlterMaterializedViewRebuildAnalyzer}} rewrites the incremental rebuild 
into a multi-insert with a DELETE branch, under {{Context.Operation.MERGE}}, 
whatever the write mode of the view. With               
  {{write.merge.mode=copy-on-write}} the file sink of that branch is 
copy-on-write, so {{HiveIcebergCopyOnWriteRecordWriter}} writes the rows to 
delete back as data and marks no data file as replaced: the old 
  file stays, and its rows are written again.                                   
                                                                                
                                                 
                                                                                
                                                                                
                                                 
  A partitioned view with {{write.merge.mode=copy-on-write}} and 
{{write.delete.mode=merge-on-read}} fails to compile since HIVE-30112: the 
partition expressions of the DELETE branch are positioned for a      
  merge-on-read delete row, but the row has the copy-on-write layout. Before 
HIVE-30112 it compiled and kept the stale rows as above.                        
                                                    
                                                                                
                                                                                
                                                 
  *Reproducer*                                                                  
                                                                                
                                                 
  {code:sql}                                                                    
                                                                                
                                                 
  set hive.support.concurrency=true;                                            
                                                                                
                                                 
  set hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DbTxnManager;          
                                                                                
                                                 
  set hive.materializedview.rebuild.incremental.factor=0.01;                    
     

create external table src_ice(a int, b int) stored by iceberg stored as orc 
tblproperties ('format-version'='2');                                           
                                                   
  insert into src_ice values (1, 1), (2, 2), (3, 2);                            
                                                                                
                                                 
                                                                                
                                                                                
                                                 
  create materialized view mv_cow stored by iceberg stored as orc tblproperties 
('format-version'='2',                                                          
                                                 
    'write.merge.mode'='copy-on-write', 'write.delete.mode'='copy-on-write', 
'write.update.mode'='copy-on-write') as                                         
                                                    
  select b, count(*) cnt, sum(a) s from src_ice group by b;                     
                                                                                
                                                 
                                                                                
                                                                                
                                                 
  create materialized view mv_mixed partitioned on (b) stored by iceberg stored 
as orc tblproperties ('format-version'='2',                                     
                                                 
    'write.merge.mode'='copy-on-write', 'write.delete.mode'='merge-on-read') as 
                                                                                
                                                 
  select count(*) cnt, sum(a) s, b from src_ice group by b;                     
                                                                                
                                                 
                                                                                
                                                                                
                                                 
  insert into src_ice values (4, 1), (5, 3);                                    
                                                                                
                                                 
                                                                                
                                                                                
                                                 
  alter materialized view mv_cow rebuild;                                       
                                                                                
                                                 
  select * from mv_cow;                                                         
                                                                                
                                                 
                                                                                
                                                                                
                                                 
  alter materialized view mv_mixed rebuild;                                     
                                                                                
                                                 
  {code}                                                                        
                                                                                
                                                 
                                                                                
                                                                                
                                                 
  {{select * from mv_cow}}:                                                     
                                                                                
                                                 
  - expected: (1, 2, 5), (2, 2, 5), (3, 1, 5)                                   
                                                                                
                                                 
  - actual: (1, 1, 1), (1, 1, 1), (1, 2, 5), (2, 2, 5), (3, 1, 5)         

 The same view with the default merge-on-read modes rebuilds correctly.         
                                                                                
                                                
                                                                                
                                                                                
                                                 
  {{alter materialized view mv_mixed rebuild}}:                                 
                                                                                
                                                 
  {noformat}                                                                    
                                                                                
                                                 
  java.lang.IndexOutOfBoundsException: Index 9 out of bounds for length 7       
                                                                                
                                                 
          at 
org.apache.iceberg.mr.hive.IcebergTransformSortFunctionUtil.lambda$getCustomTransformExpr$12(IcebergTransformSortFunctionUtil.java:154)
                                                             
          at 
org.apache.hadoop.hive.ql.optimizer.SortedDynPartitionOptimizer$SortedDynamicPartitionProc.allStaticPartitions(SortedDynPartitionOptimizer.java:435)
                                                
  {noformat}                                                                    
                                                                                
                                                 
                                                                                
                                                                                
                                                 
  *Proposed fix*                                                                
                                                                                
                                                 
  Rebuild a materialized view with {{write.merge.mode=copy-on-write}} fully 
instead of incrementally, or rewrite its incremental rebuild with the 
copy-on-write MERGE rewriter. Either removes the DELETE branch 
  under a copy-on-write MERGE, and with it the compile failure.                

> Iceberg: Incremental rebuild of a copy-on-write materialized view keeps the 
> rows it replaces 
> ---------------------------------------------------------------------------------------------
>
>                 Key: HIVE-30251
>                 URL: https://issues.apache.org/jira/browse/HIVE-30251
>             Project: Hive
>          Issue Type: Bug
>            Reporter: Denys Kuzmenko
>            Priority: Major
>
> *Problem*                                                                     
>                                                                               
>                                                    
>   An incremental rebuild of an Iceberg materialized view with 
> {{write.merge.mode=copy-on-write}} keeps the rows it replaces: every updated 
> group keeps its old row, twice, next to its new one. There is no      
>   error.                                                                      
>                                                                               
>                                                      
>                                                                               
>                                                                               
>                                                      
>   {{AlterMaterializedViewRebuildAnalyzer}} rewrites the incremental rebuild 
> into a multi-insert with a DELETE branch, under {{Context.Operation.MERGE}}, 
> whatever the write mode of the view. With               
>   {{write.merge.mode=copy-on-write}} the file sink of that branch is 
> copy-on-write, so {{HiveIcebergCopyOnWriteRecordWriter}} writes the rows to 
> delete back as data and marks no data file as replaced: the old 
>   file stays, and its rows are written again.                                 
>                                                                               
>                                                      
>                                                                               
>                                                                               
>                                                      
>   A partitioned view with {{write.merge.mode=copy-on-write}} and 
> {{write.delete.mode=merge-on-read}} fails to compile since HIVE-30112: the 
> partition expressions of the DELETE branch are positioned for a      
>   merge-on-read delete row, but the row has the copy-on-write layout. Before 
> HIVE-30112 it compiled and kept the stale rows as above.                      
>                                                       
>                                                                               
>                                                                               
>                                                      
>   *Reproducer*                                                                
>                                                                               
>                                                      
>   {code:sql}                                                                  
>                                                                               
>                                                      
>   set hive.support.concurrency=true;                                          
>                                                                               
>                                                      
>   set hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DbTxnManager;        
>                                                                               
>                                                      
>   set hive.materializedview.rebuild.incremental.factor=0.01;                  
>        
> create external table src_ice(a int, b int) stored by iceberg stored as orc 
> tblproperties ('format-version'='2');                                         
>                                                      
>   insert into src_ice values (1, 1), (2, 2), (3, 2);                          
>                                                                               
>                                                      
>                                                                               
>                                                                               
>                                                      
>   create materialized view mv_cow stored by iceberg stored as orc 
> tblproperties ('format-version'='2',                                          
>                                                                  
>     'write.merge.mode'='copy-on-write', 'write.delete.mode'='copy-on-write', 
> 'write.update.mode'='copy-on-write') as                                       
>                                                       
>   select b, count(*) cnt, sum(a) s from src_ice group by b;                   
>                                                                               
>                                                      
>                                                                               
>                                                                               
>                                                      
>   create materialized view mv_mixed partitioned on (b) stored by iceberg 
> stored as orc tblproperties ('format-version'='2',                            
>                                                           
>     'write.merge.mode'='copy-on-write', 'write.delete.mode'='merge-on-read') 
> as                                                                            
>                                                       
>   select count(*) cnt, sum(a) s, b from src_ice group by b;                   
>                                                                               
>                                                      
>                                                                               
>                                                                               
>                                                      
>   insert into src_ice values (4, 1), (5, 3);                                  
>                                                                               
>                                                      
>                                                                               
>                                                                               
>                                                      
>   alter materialized view mv_cow rebuild;                                     
>                                                                               
>                                                      
>   select * from mv_cow;                                                       
>                                                                               
>                                                      
>                                                                               
>                                                                               
>                                                      
>   alter materialized view mv_mixed rebuild;                                   
>                                                                               
>                                                      
>   {code}                                                                      
>                                                                               
>                                                      
>                                                                               
>                                                                               
>                                                      
>   {{select * from mv_cow}}:                                                   
>                                                                               
>                                                      
>   - expected: (1, 2, 5), (2, 2, 5), (3, 1, 5)                                 
>                                                                               
>                                                      
>   - actual: (1, 1, 1), (1, 1, 1), (1, 2, 5), (2, 2, 5), (3, 1, 5)         
>  The same view with the default merge-on-read modes rebuilds correctly.       
>                                                                               
>                                                     
>                                                                               
>                                                                               
>                                                      
>   {{alter materialized view mv_mixed rebuild}}:                               
>                                                                               
>                                                      
>   {noformat}                                                                  
>                                                                               
>                                                      
>   java.lang.IndexOutOfBoundsException: Index 9 out of bounds for length 7     
>                                                                               
>                                                      
>           at 
> org.apache.iceberg.mr.hive.IcebergTransformSortFunctionUtil.lambda$getCustomTransformExpr$12(IcebergTransformSortFunctionUtil.java:154)
>                                                              
>           at 
> org.apache.hadoop.hive.ql.optimizer.SortedDynPartitionOptimizer$SortedDynamicPartitionProc.allStaticPartitions(SortedDynPartitionOptimizer.java:435)
>                                                 
>   {noformat}                                                                  
>                                                                               
>                                                      
>                                                                               
>                                                                               
>                                                      
>   *Proposed fix*                                                              
>                                                                               
>                                                      
>   Rebuild a materialized view with {{write.merge.mode=copy-on-write}} fully 
> instead of incrementally, or rewrite its incremental rebuild with the 
> copy-on-write MERGE rewriter. Either removes the DELETE branch 
>   under a copy-on-write MERGE, and with it the compile failure.               
>  



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

Reply via email to