[ 
https://issues.apache.org/jira/browse/HIVE-30253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Denys Kuzmenko updated HIVE-30253:
----------------------------------
    Component/s: Iceberg integration

>   Iceberg: V2 merge-on-read deletes with write.fanout.enabled=false are 
> sorted, but always written by the fanout writer
> -----------------------------------------------------------------------------------------------------------------------
>
>                 Key: HIVE-30253
>                 URL: https://issues.apache.org/jira/browse/HIVE-30253
>             Project: Hive
>          Issue Type: Improvement
>          Components: Iceberg integration
>            Reporter: Denys Kuzmenko
>            Priority: Major
>
> *Problem*                                                                     
>                                                                               
>                                                    
>   On a V2 table with {{write.fanout.enabled=false}} (default: true), 
> {{HiveIcebergStorageHandler.acidSortColumns(DELETE)}} sorts the records that 
> a merge-on-read DELETE, split UPDATE or MERGE deletes by spec  
>   ID, partition, file path and position, so that they can be written by the 
> clustered position delete writer with bounded memory.                         
>                                                        
>                                                                               
>                                                                               
>                                                      
>   The delete writer is never told that its input is sorted. 
> {{FileSinkOperator}} marks the input sorted only when the file sink has a 
> {{DynamicPartitionCtx}} with custom partition or sort expressions, and the 
>   file sink of a merge-on-read delete has none. So 
> {{HiveIcebergWriterBase.newDeleteWriter}} always builds a 
> {{FanoutPositionOnlyDeleteWriter}}, which buffers the positions of every 
> partition in memory.       
>                                                                               
>                                                                               
>                                                      
>   As a result, every such statement pays for a reducer stage and a full sort 
> that nothing uses, and the {{ClusteredPositionDeleteWriter}} that the 
> property selects never runs.                                  
>                                                                               
>                                                                               
>                                                      
>   V3 is not affected: a DV writer needs no order, and {{acidSortColumns}} 
> returns none for V3.                                                          
>                                                          
>                                                                               
>                                                                               
>                                                      
>   *Reproducer*                                                                
>                                                                               
>                                                      
>   {code:sql}                                                                  
>                                                                               
>                                                      
>   create external table ice_sorted_v2 (id int, name string, amount int) 
> partitioned by spec (bucket(2, id))                                           
>                                                            
>   stored by iceberg tblproperties ('format-version'='2', 
> 'write.fanout.enabled'='false');                                              
>                                                                           
>   insert into ice_sorted_v2 values (1, 'a', 10), (2, 'b', 20), (3, 'c', 30), 
> (4, 'd', 40);                                                                 
>                                                       
>                                                                               
>                                                                               
>                                                      
>   explain delete from ice_sorted_v2 where id = 1;                             
>                                                                               
>                                                      
>   {code}                                                                      
>                                                                               
>                                                      
>   The plan has Map 1 -> Reducer 2 with a Reduce Output Operator sorted (++++) 
> by the spec ID, partition, file path and position, and the delete writer of 
> Reducer 2 is the fanout writer.                        
>                                                                               
>                                                                               
>                                                      
>   *Proposed fix*                                                              
>                                                                               
>                                                      
>   Either mark the input of the delete writer as sorted when 
> {{acidSortColumns(DELETE)}} is non-empty, so that the clustered writer is 
> used, or drop the sort and keep the fanout writer.                         



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

Reply via email to