[
https://issues.apache.org/jira/browse/HIVE-30270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Denys Kuzmenko updated HIVE-30270:
----------------------------------
Description:
A row without a stored _last_updated_sequence_number inherits the sequence
number of its data file's manifest entry. Per the Iceberg spec (Row Lineage),
that is the data sequence number. Iceberg 1.12 fixed its own reader constants
to use it (apache/iceberg#17039, PartitionUtil.constantsMap). Hive's row-mode
read path uses constantsMap, so it has followed the spec since HIVE-30127. The
vectorized path (HiveBatchIterator) still uses
task.file().fileSequenceNumber(), the Iceberg 1.11 behavior it copied in
HIVE-29828.
The two numbers differ when a rewrite keeps the data sequence number of the
rows it rewrites. One example is a compaction by another engine (Spark
rewrite_data_files) on a V2 table that is later upgraded to
V3:
- a vectorized query reports the sequence number of the rewrite;
- a non-vectorized query reports the sequence number of the original write.
Hive's own compaction does not keep the data sequence number, so tables
written and compacted only by Hive are not affected.
Fix: HiveBatchIterator uses task.file().dataSequenceNumber().
> Iceberg: Vectorized reads use the file sequence number instead of the data
> sequence number for LAST__UPDATED__SEQUENCE__NUMBER
> ------------------------------------------------------------------------------------------------------------------------------
>
> Key: HIVE-30270
> URL: https://issues.apache.org/jira/browse/HIVE-30270
> Project: Hive
> Issue Type: Bug
> Components: Iceberg integration
> Reporter: Denys Kuzmenko
> Priority: Major
>
> A row without a stored _last_updated_sequence_number inherits the sequence
> number of its data file's manifest entry. Per the Iceberg spec (Row Lineage),
> that is the data sequence number. Iceberg 1.12 fixed its own reader constants
> to use it (apache/iceberg#17039, PartitionUtil.constantsMap). Hive's row-mode
> read path uses constantsMap, so it has followed the spec since HIVE-30127.
> The vectorized path (HiveBatchIterator) still uses
> task.file().fileSequenceNumber(), the Iceberg 1.11 behavior it copied in
> HIVE-29828.
>
>
>
>
> The two numbers differ when a rewrite keeps the data sequence number of the
> rows it rewrites. One example is a compaction by another engine (Spark
> rewrite_data_files) on a V2 table that is later upgraded to
> V3:
>
>
> - a vectorized query reports the sequence number of the rewrite;
>
>
> - a non-vectorized query reports the sequence number of the original write.
>
>
>
>
>
> Hive's own compaction does not keep the data sequence number, so tables
> written and compacted only by Hive are not affected.
>
>
>
>
> Fix: HiveBatchIterator uses task.file().dataSequenceNumber().
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)