[
https://issues.apache.org/jira/browse/HIVE-30270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Denys Kuzmenko updated HIVE-30270:
----------------------------------
Labels: correctness (was: )
> Iceberg: Vectorized reads use the file sequence number instead of the data
> sequence number for LAST__UPDATED__SEQUENCE__NUMBER
> ------------------------------------------------------------------------------------------------------------------------------
>
> Key: HIVE-30270
> URL: https://issues.apache.org/jira/browse/HIVE-30270
> Project: Hive
> Issue Type: Bug
> Components: Iceberg integration
> Reporter: Denys Kuzmenko
> Priority: Major
> Labels: correctness
>
> A row without a stored _last_updated_sequence_number inherits the sequence
> number of its data file's manifest entry. Per the Iceberg spec (Row Lineage),
> that is the data sequence number. Iceberg 1.12 fixed its own reader constants
> to use it (apache/iceberg#17039, PartitionUtil.constantsMap). Hive's row-mode
> read path uses constantsMap, so it has followed the spec since HIVE-30127.
> The vectorized path (HiveBatchIterator) still uses
> task.file().fileSequenceNumber(), the Iceberg 1.11 behavior it copied in
> HIVE-29828.
>
>
>
>
> The two numbers differ when a rewrite keeps the data sequence number of the
> rows it rewrites. One example is a compaction by another engine (Spark
> rewrite_data_files) on a V2 table that is later upgraded to V3:
>
>
>
> - a vectorized query reports the sequence number of the rewrite;
>
>
> - a non-vectorized query reports the sequence number of the original write.
>
>
>
>
>
> Hive's own compaction does not keep the data sequence number, so tables
> written and compacted only by Hive are not affected.
>
>
>
>
> Fix: HiveBatchIterator uses task.file().dataSequenceNumber().
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)