laskoviymishka opened a new issue, #1010: URL: https://github.com/apache/iceberg-go/issues/1010
Parent: #589 When a v3 data file pre-dates row lineage (the file has no `_row_id` / `_last_updated_sequence_number` columns), the scanner should project those reserved fields as all-null columns rather than erroring. Java fixed this in [apache/iceberg#15187](https://github.com/apache/iceberg/pull/15187) and [#15508](https://github.com/apache/iceberg/pull/15508). iceberg-go has a partial path today: `synthesizeRowLineageColumns` fills nulls with `first_row_id + row_position` and `data_sequence_number` *when the columns are already present in the batch*. The gap is the projection layer above it — if a scan requests these reserved fields against a file that lacks them entirely, the request needs to add the columns as null arrays, not error. Detect the reserved field IDs (`2147483546` for `_row_id`, `2147483545` for `_last_updated_sequence_number`) in the projection path; emit null columns of the right type when the underlying file lacks them. The existing synthesis path then handles the "file has them but the rows are null" case as before. Test: scan a v3 metadata that has `next-row-id` set but a data file written before row lineage existed — assert the projected columns come back all-null without error. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
