[ 
https://issues.apache.org/jira/browse/HIVE-30270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Denys Kuzmenko updated HIVE-30270:
----------------------------------
    Description: 
  A row without a stored _last_updated_sequence_number inherits the sequence 
number of its data file's manifest entry. Per the Iceberg spec (Row Lineage), 
that is the data sequence number. Iceberg 1.12 fixed its own reader constants 
to use it (apache/iceberg#17039, PartitionUtil.constantsMap). Hive's row-mode 
read path uses constantsMap, so it has followed the spec since HIVE-30127. The 
vectorized path (HiveBatchIterator) still uses 
task.file().fileSequenceNumber(), the Iceberg 1.11 behavior it copied in 
HIVE-29828.                                                                     
                       
                                                                                
                                                                                
                                                 
  The two numbers differ when a rewrite keeps the data sequence number of the 
rows it rewrites. One example is a compaction by another engine (Spark 
rewrite_data_files) on a V2 table that is later upgraded to 
  V3:                                                                           
                                                                                
                                                 
  - a vectorized query reports the sequence number of the rewrite;              
                                                                                
                                                 
  - a non-vectorized query reports the sequence number of the original write.   
                                                                                
                                                 
                                                                                
                                                                                
                                                 
  Hive's own compaction does not keep the data sequence number, so tables 
written and compacted only by Hive are not affected.                            
                                                       
                                                                                
                                                                                
                                                 
  Fix: HiveBatchIterator uses task.file().dataSequenceNumber().

> Iceberg: Vectorized reads use the file sequence number instead of the data 
> sequence number for LAST__UPDATED__SEQUENCE__NUMBER
> ------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HIVE-30270
>                 URL: https://issues.apache.org/jira/browse/HIVE-30270
>             Project: Hive
>          Issue Type: Bug
>          Components: Iceberg integration
>            Reporter: Denys Kuzmenko
>            Priority: Major
>
>   A row without a stored _last_updated_sequence_number inherits the sequence 
> number of its data file's manifest entry. Per the Iceberg spec (Row Lineage), 
> that is the data sequence number. Iceberg 1.12 fixed its own reader constants 
> to use it (apache/iceberg#17039, PartitionUtil.constantsMap). Hive's row-mode 
> read path uses constantsMap, so it has followed the spec since HIVE-30127. 
> The vectorized path (HiveBatchIterator) still uses 
> task.file().fileSequenceNumber(), the Iceberg 1.11 behavior it copied in 
> HIVE-29828.                                                                   
>                          
>                                                                               
>                                                                               
>                                                      
>   The two numbers differ when a rewrite keeps the data sequence number of the 
> rows it rewrites. One example is a compaction by another engine (Spark 
> rewrite_data_files) on a V2 table that is later upgraded to 
>   V3:                                                                         
>                                                                               
>                                                      
>   - a vectorized query reports the sequence number of the rewrite;            
>                                                                               
>                                                      
>   - a non-vectorized query reports the sequence number of the original write. 
>                                                                               
>                                                      
>                                                                               
>                                                                               
>                                                      
>   Hive's own compaction does not keep the data sequence number, so tables 
> written and compacted only by Hive are not affected.                          
>                                                          
>                                                                               
>                                                                               
>                                                      
>   Fix: HiveBatchIterator uses task.file().dataSequenceNumber().



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

Reply via email to