[ 
https://issues.apache.org/jira/browse/HIVE-30253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Denys Kuzmenko updated HIVE-30253:
----------------------------------
    Description: 
*Problem*                                                                       
                                                                                
                                               
  On a V2 table with {{write.fanout.enabled=false}} (default: true), 
{{HiveIcebergStorageHandler.acidSortColumns(DELETE)}} sorts the records that a 
merge-on-read DELETE, split UPDATE or MERGE deletes by spec  
  ID, partition, file path and position, so that they can be written by the 
clustered position delete writer with bounded memory.                           
                                                     
                                                                                
                                                                                
                                                 
  The delete writer is never told that its input is sorted. 
{{FileSinkOperator}} marks the input sorted only when the file sink has a 
{{DynamicPartitionCtx}} with custom partition or sort expressions, and the 
  file sink of a merge-on-read delete has none. So 
{{HiveIcebergWriterBase.newDeleteWriter}} always builds a 
{{FanoutPositionOnlyDeleteWriter}}, which buffers the positions of every 
partition in memory.       
                                                                                
                                                                                
                                                 
  As a result, every such statement pays for a reducer stage and a full sort 
that nothing uses, and the {{ClusteredPositionDeleteWriter}} that the property 
selects never runs.                                  
                                                                                
                                                                                
                                                 
  V3 is not affected: a DV writer needs no order, and {{acidSortColumns}} 
returns none for V3.                                                            
                                                       
                                                                                
                                                                                
                                                 
  *Reproducer*                                                                  
                                                                                
                                                 
  {code:sql}                                                                    
                                                                                
                                                 
  create external table ice_sorted_v2 (id int, name string, amount int) 
partitioned by spec (bucket(2, id))                                             
                                                         
  stored by iceberg tblproperties ('format-version'='2', 
'write.fanout.enabled'='false');                                                
                                                                        
  insert into ice_sorted_v2 values (1, 'a', 10), (2, 'b', 20), (3, 'c', 30), 
(4, 'd', 40);                                                                   
                                                    
                                                                                
                                                                                
                                                 
  explain delete from ice_sorted_v2 where id = 1;                               
                                                                                
                                                 
  {code}                                                                        
                                                                                
                                                 
  The plan has Map 1 -> Reducer 2 with a Reduce Output Operator sorted (++++) 
by the spec ID, partition, file path and position, and the delete writer of 
Reducer 2 is the fanout writer.                        
                                                                                
                                                                                
                                                 
  *Proposed fix*                                                                
                                                                                
                                                 
  Either mark the input of the delete writer as sorted when 
{{acidSortColumns(DELETE)}} is non-empty, so that the clustered writer is used, 
or drop the sort and keep the fanout writer.                         


>   Iceberg: V2 merge-on-read deletes with write.fanout.enabled=false are 
> sorted, but always written by the fanout writer
> -----------------------------------------------------------------------------------------------------------------------
>
>                 Key: HIVE-30253
>                 URL: https://issues.apache.org/jira/browse/HIVE-30253
>             Project: Hive
>          Issue Type: Bug
>            Reporter: Denys Kuzmenko
>            Priority: Major
>
> *Problem*                                                                     
>                                                                               
>                                                    
>   On a V2 table with {{write.fanout.enabled=false}} (default: true), 
> {{HiveIcebergStorageHandler.acidSortColumns(DELETE)}} sorts the records that 
> a merge-on-read DELETE, split UPDATE or MERGE deletes by spec  
>   ID, partition, file path and position, so that they can be written by the 
> clustered position delete writer with bounded memory.                         
>                                                        
>                                                                               
>                                                                               
>                                                      
>   The delete writer is never told that its input is sorted. 
> {{FileSinkOperator}} marks the input sorted only when the file sink has a 
> {{DynamicPartitionCtx}} with custom partition or sort expressions, and the 
>   file sink of a merge-on-read delete has none. So 
> {{HiveIcebergWriterBase.newDeleteWriter}} always builds a 
> {{FanoutPositionOnlyDeleteWriter}}, which buffers the positions of every 
> partition in memory.       
>                                                                               
>                                                                               
>                                                      
>   As a result, every such statement pays for a reducer stage and a full sort 
> that nothing uses, and the {{ClusteredPositionDeleteWriter}} that the 
> property selects never runs.                                  
>                                                                               
>                                                                               
>                                                      
>   V3 is not affected: a DV writer needs no order, and {{acidSortColumns}} 
> returns none for V3.                                                          
>                                                          
>                                                                               
>                                                                               
>                                                      
>   *Reproducer*                                                                
>                                                                               
>                                                      
>   {code:sql}                                                                  
>                                                                               
>                                                      
>   create external table ice_sorted_v2 (id int, name string, amount int) 
> partitioned by spec (bucket(2, id))                                           
>                                                            
>   stored by iceberg tblproperties ('format-version'='2', 
> 'write.fanout.enabled'='false');                                              
>                                                                           
>   insert into ice_sorted_v2 values (1, 'a', 10), (2, 'b', 20), (3, 'c', 30), 
> (4, 'd', 40);                                                                 
>                                                       
>                                                                               
>                                                                               
>                                                      
>   explain delete from ice_sorted_v2 where id = 1;                             
>                                                                               
>                                                      
>   {code}                                                                      
>                                                                               
>                                                      
>   The plan has Map 1 -> Reducer 2 with a Reduce Output Operator sorted (++++) 
> by the spec ID, partition, file path and position, and the delete writer of 
> Reducer 2 is the fanout writer.                        
>                                                                               
>                                                                               
>                                                      
>   *Proposed fix*                                                              
>                                                                               
>                                                      
>   Either mark the input of the delete writer as sorted when 
> {{acidSortColumns(DELETE)}} is non-empty, so that the clustered writer is 
> used, or drop the sort and keep the fanout writer.                         



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

Reply via email to