[ 
https://issues.apache.org/jira/browse/HIVE-30270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Denys Kuzmenko updated HIVE-30270:
----------------------------------
    Labels: correctness  (was: )

> Iceberg: Vectorized reads use the file sequence number instead of the data 
> sequence number for LAST__UPDATED__SEQUENCE__NUMBER
> ------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HIVE-30270
>                 URL: https://issues.apache.org/jira/browse/HIVE-30270
>             Project: Hive
>          Issue Type: Bug
>          Components: Iceberg integration
>            Reporter: Denys Kuzmenko
>            Priority: Major
>              Labels: correctness
>
>   A row without a stored _last_updated_sequence_number inherits the sequence 
> number of its data file's manifest entry. Per the Iceberg spec (Row Lineage), 
> that is the data sequence number. Iceberg 1.12 fixed its own reader constants 
> to use it (apache/iceberg#17039, PartitionUtil.constantsMap). Hive's row-mode 
> read path uses constantsMap, so it has followed the spec since HIVE-30127. 
> The vectorized path (HiveBatchIterator) still uses 
> task.file().fileSequenceNumber(), the Iceberg 1.11 behavior it copied in 
> HIVE-29828.                                                                   
>                          
>                                                                               
>                                                                               
>                                                      
> The two numbers differ when a rewrite keeps the data sequence number of the 
> rows it rewrites. One example is a compaction by another engine (Spark 
> rewrite_data_files) on a V2 table that is later upgraded to V3:               
>                                                                               
>                                                                               
>                                  
>   - a vectorized query reports the sequence number of the rewrite;            
>                                                                               
>                                                      
>   - a non-vectorized query reports the sequence number of the original write. 
>                                                                               
>                                                      
>                                                                               
>                                                                               
>                                                      
> Hive's own compaction does not keep the data sequence number, so tables 
> written and compacted only by Hive are not affected.                          
>                                                          
>                                                                               
>                                                                               
>                                                      
> Fix: HiveBatchIterator uses task.file().dataSequenceNumber().



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

Reply via email to