[ 
https://issues.apache.org/jira/browse/HIVE-30246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Denys Kuzmenko updated HIVE-30246:
----------------------------------
    Description: 
*Problem*                                                                       
                                                                                
                                               
  With {{hive.merge.tezfiles=true}}, a DELETE on a format-version 3 Iceberg 
table fails in the File Merge vertex. The merge stage for delete files 
(HIVE-28267) takes every delete file the DELETE wrote as merge
  input. On V3 these are deletion vectors in Puffin files, and 
{{IcebergMergeRecordReader}} reads only Avro, ORC and Parquet.                  
                                                                  
                                                                                
                                                                                
                                                 
  {noformat}                                                                    
                                                                                
                                                 
  TezRuntimeException: Vertex failed, vertexName=File Merge, ...                
                                                                                
                                                 
  Caused by: java.lang.UnsupportedOperationException: Cannot read PUFFIN file: 
file:/.../warehouse/v3_source/data/00000-0-...                                  
                                                  
          at 
org.apache.iceberg.mr.mapreduce.IcebergMergeRecordReader.openGeneric(IcebergMergeRecordReader.java:108)
                                                                                
             
          at 
org.apache.iceberg.mr.mapreduce.IcebergMergeRecordReader.nextTask(IcebergMergeRecordReader.java:59)
                                                                                
                 
          at 
org.apache.iceberg.mr.mapreduce.IcebergMergeRecordReader.initialize(IcebergMergeRecordReader.java:55)
                                                                                
               
          ...                                                                   
                                                                                
                                                 
          at 
org.apache.iceberg.mr.hive.HiveIcebergInputFormat.getRecordReader(HiveIcebergInputFormat.java:232)
                                                                                
                  
          at 
org.apache.hadoop.hive.ql.io.CombineHiveRecordReader.<init>(CombineHiveRecordReader.java:104)
                                                                                
                       
  {noformat}                                                                    
                                                                                
                                                 
                                                                                
                                                                                
                                                 
  Only DELETE is affected: UPDATE and MERGE never get a merge stage.   

  *Reproducer* (TestIcebergLlapLocalCliDriver; iceberg_merge_delete_files.q 
with format-version 3)                                                          
                                                     
  {code:sql}                                                                    
                                                                                
                                                 
  --! qt:dataset:src                                                            
                                                                                
                                                 
  set hive.merge.mapredfiles=true;                                              
                                                                                
                                                 
  set hive.merge.mapfiles=true;                                                 
                                                                                
                                                 
  set hive.merge.tezfiles=true;                                                 
                                                                                
                                                 
  set hive.optimize.sort.dynamic.partition.threshold=-1;                        
                                                                                
                                                 
  set mapred.reduce.tasks=5;                                                    
                                                                                
                                                 
  set hive.blobstore.supported.schemes=hdfs,file;                               
                                                                                
                                                 
                                                                                
                                                                                
                                                 
  create table v3_source(key string) stored by iceberg stored as parquet 
tblproperties('format-version'='3');                                            
                                                        
  insert overwrite table v3_source select key from src;                         
                                                                                
                                                 
  insert into table v3_source select key from src;                              
                                                                                
                                                 
  insert into table v3_source select key from src;                              
                                                                                
                                                 
  insert into table v3_source select key from src;                              
                                                                                
                                                 
                                                                                
                                                                                
                                                 
  delete from v3_source where key in (select key from src where key < '200');   
                                                                                
                                                 
  {code}                                                                        
                                                                                
                                                 
                                                                                
                                                                                
                                                 
  *Expected*                                                                    
                                                                                
                                                 
  The DELETE succeeds. A DV already holds every deleted position of its data 
file, so DVs have nothing to gain from the row-based merge.                     
                                                    
                                                                                
                                                                                
                                                 
  *Proposed fix*                                                                
                                                                                
                                                 
  Skip DVs ({{ContentFileUtil.isDV}}) when 
{{HiveIcebergOutputCommitter.getOutputFiles}} lists the merge input. For a V3 
DELETE the list is then empty, and {{ConditionalResolverMergeFiles}} runs only 
the move 
  task. V2 position-delete files and data files are merged as before.           
        

> Iceberg: V3 DELETE fails with hive.merge.tezfiles=true: the merge task can't 
> read DVs
> -------------------------------------------------------------------------------------
>
>                 Key: HIVE-30246
>                 URL: https://issues.apache.org/jira/browse/HIVE-30246
>             Project: Hive
>          Issue Type: Bug
>          Components: Iceberg integration
>            Reporter: Denys Kuzmenko
>            Priority: Major
>




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

Reply via email to