deniskuzZ opened a new pull request, #6919: URL: https://github.com/apache/hive/pull/6919
### What changes were proposed in this pull request? - `HiveBatchIterator` arms the row lineage sentinel (`-1` at row 0 of the `_row_id` and `_last_updated_sequence_number` vectors) before every batch instead of once per file - `RowLineageUtils.initializeRowLineageColumns` also sets `noNulls`, and the iterator checks `noNulls && vector[0] == -1`: a vector the reader marked with nulls counts as read, so a stored column whose first row is null is not taken for a missing one ### Why are the changes needed? - Bug-Fix: the sentinel tells the iterator that the file stores no `_row_id` / `_last_updated_sequence_number`, so it fills the inherited values (the file's first row id plus the row position, the file's sequence number). Armed once per file, it was overwritten by the first batch, and every later batch of the file kept the first batch's values: a 3000-row file read vectorized returns 1024 distinct `ROW__LINEAGE__ID`. An UPDATE or MERGE persists the duplicates ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? - iceberg_row_lineage_many_batches.q: 3000-row V3 files read vectorized and row by row, with inherited ids, with ids stored by a copy-on-write UPDATE, and with a copy-on-write MERGE whose file mixes carried rows (stored ids) with inserted ones (NULL stored ids) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
