[
https://issues.apache.org/jira/browse/HIVE-30123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Denys Kuzmenko updated HIVE-30123:
----------------------------------
Description:
- HiveBatchIterator:137-188 marks "the file has no _row_id column" by setting
vector[0] = -1L once, in its constructor
(RowLineageUtils.initializeRowLineageColumns).
- The first batch overwrites vector[0] with a real ID. From the second batch
on, the iterator assumes the values came from the file and only fills nulls, so
the batch keeps the first batch's IDs.
LAST__UPDATED__SEQUENCE__NUMBER has the same pattern.
- It affects every V3 file written without _row_id, which includes every
freshly inserted file.
*Repro:*
a V3 table with more than 1,024 rows in one file. Compare count(distinct
ROW__LINEAGE__ID) with vectorization on and off
was:
- HiveBatchIterator marks "the file has no _row_id column" by setting
vector[0] = -1L once, in its constructor
(RowLineageUtils.initializeRowLineageColumns).
- The first batch overwrites vector[0] with a real ID. From the second batch
on, the iterator assumes the values came from the file and only fills nulls, so
the batch keeps the first batch's IDs.
LAST__UPDATED__SEQUENCE__NUMBER has the same pattern.
- It affects every V3 file written without _row_id, which includes every
freshly inserted file.
*Repro:*
insert 3,000 rows into a V3 table, then compare count(distinct
ROW__LINEAGE__ID) with vectorization on and off. Vectorized gives 1,024
(0–1023); non-vectorized gives 3,000.
> Iceberg: vectorized read returns wrong row lineage values after the first
> batch of a file
> -----------------------------------------------------------------------------------------
>
> Key: HIVE-30123
> URL: https://issues.apache.org/jira/browse/HIVE-30123
> Project: Hive
> Issue Type: Bug
> Components: Iceberg integration
> Reporter: Denys Kuzmenko
> Priority: Major
> Labels: correctness, v3
>
> - HiveBatchIterator:137-188 marks "the file has no _row_id column" by
> setting vector[0] = -1L once, in its constructor
> (RowLineageUtils.initializeRowLineageColumns).
> - The first batch overwrites vector[0] with a real ID. From the second
> batch on, the iterator assumes the values came from the file and only fills
> nulls, so the batch keeps the first batch's IDs.
> LAST__UPDATED__SEQUENCE__NUMBER has the same pattern.
>
>
> - It affects every V3 file written without _row_id, which includes every
> freshly inserted file.
>
>
> *Repro:*
> a V3 table with more than 1,024 rows in one file. Compare count(distinct
> ROW__LINEAGE__ID) with vectorization on and off
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
