[ 
https://issues.apache.org/jira/browse/HIVE-30252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Denys Kuzmenko updated HIVE-30252:
----------------------------------
    Component/s: Iceberg integration

> Iceberg: A filter that ORs a virtual column with a data column pushes down 
> only the data column and loses rows
> --------------------------------------------------------------------------------------------------------------
>
>                 Key: HIVE-30252
>                 URL: https://issues.apache.org/jira/browse/HIVE-30252
>             Project: Hive
>          Issue Type: Bug
>          Components: Iceberg integration
>            Reporter: Denys Kuzmenko
>            Priority: Minor
>
> *Problem*                                                                     
>                                                                               
>                                                    
>   {{HiveIcebergStorageHandler.decomposePredicate}} removes the children of 
> the top-level filter expression that reference {{FILE__PATH}}, 
> {{PARTITION__SPEC__ID}}, {{PARTITION__KEY}}, {{PARTITION__NAME}},      
>   {{ROW__LINEAGE__ID}} or {{LAST__UPDATED__SEQUENCE__NUMBER}}, without 
> checking that the expression is an AND. For an OR this narrows the pushed 
> predicate: Iceberg prunes the data files that match only the    
>   removed part, and their rows are missing from the result. There is no 
> error.                                                                        
>                                                            
>                                                                               
>                                                                               
>                                                      
>   The same narrowed predicate goes to 
> {{SessionStateUtil.setConflictDetectionFilter}}, so a DML with such a WHERE 
> clause checks too few files for conflicting concurrent writes.                
>                  
>                                                                               
>                                                                               
>                                                      
>   *Reproducer*                                                                
>                                                                               
>                                                      
>   {code:sql}                                                                  
>                                                                               
>                                                      
>   create table t_or(id int, v string) stored by iceberg stored as orc 
> tblproperties('format-version'='2');                                          
>                                                              
>   insert into t_or values (1, 'a');                                           
>                                                                               
>                                                      
>   insert into t_or values (2, 'b');                                           
>                                                                               
>                                                      
>                                                                               
>                                                                               
>                                                      
>   select * from t_or where id = 1 or PARTITION__SPEC__ID = 0;                 
>                                                                               
>                                                      
>   select * from t_or where id = 1 or FILE__PATH is not null;                  
>                                                                               
>                                                      
>   select * from t_or where id = 1 or v = 'b';                                 
>                                                                               
>                                                      
>   {code}                            
>  - expected for all three: (1, a), (2, b)                                     
>                                                                               
>                                                     
>   - actual: the first two return only (1, a); the third, without a virtual 
> column, is correct                                                            
>                                                         
>                                                                               
>                                                                               
>                                                      
>   The result is the same with {{hive.fetch.task.conversion=none}}. EXPLAIN of 
> the first query:                                                              
>                                                      
>   {noformat}                                                                  
>                                                                               
>                                                      
>   filterExpr: (id = 1) (type: boolean)                                        
>                                                                               
>                                                      
>   predicate: ((PARTITION__SPEC__ID = 0) or (id = 1)) (type: boolean)          
>                                                                               
>                                                      
>   {noformat}                                                                  
>                                                                               
>                                                      
>                                                                               
>                                                                               
>                                                      
>   A nested OR, such as {{id = 1 or (id = 2 and FILE__PATH = 'x')}}, is 
> narrowed the same way.                                                        
>                                                             
>                                                                               
>                                                                               
>                                                      
>   *Proposed fix*                                                              
>                                                                               
>                                                      
>   Remove only the conjuncts of a top-level AND that reference these columns. 
> Push nothing when any other expression references one.                        
>                                                       



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

Reply via email to