dina-kar commented on issue #1652: URL: https://github.com/apache/iceberg-rust/issues/1652#issuecomment-6321007856
I tested the write side of row lineage against Spark, and `first-row-id` assignment looks correct to me. **Versions:** iceberg-rust 0.10.1 and `main` @ 000e791 (v0.11.0-rc.2 has the same fast-append code), with Spark 4.0.4 on Iceberg 1.12.0 and Trino 483 as readers, through a Lakekeeper v0.13.6 REST catalog. **Fast append on a v3 table.** Spark created a v3 table and inserted 10 rows (`_row_id` 0–9). iceberg-rust then wrote 10 rows with `DataFileWriterBuilder` and committed them with `Transaction::fast_append().add_data_files(...)`, once from each build. - 0.10.1: `next-row-id` went from 10 to 20, and the snapshot's `first-row-id` was 10. - `main`: `next-row-id` went from 20 to 30, and the snapshot's `first-row-id` was 20. - Spark and Trino read `_row_id` 10–29 for the appended rows, in order. **Upgrade, then append.** Spark wrote a v2 table with 20 rows. `UpgradeFormatVersion` to v3 left `next-row-id` at 0. The next fast append (5 rows) set `next-row-id` from 0 to 25 and the snapshot's `first-row-id` to 0, so it assigned ids to the 20 pre-upgrade rows as well. Spark reads `_row_id` 0–19 for the old rows and 20–24 for the new ones. That matches the spec's rule that `next-row-id` covers both added and existing rows in manifests that get a `first_row_id`. The probe is in https://github.com/ostrium-labs/loams/tree/he0-iceberg-v3-spike/spikes/he0-iceberg-v3 (`reader-common/probe.rs`, functions `append` and `upgrade`; the outputs are in `results/spark-05.txt` and `results/upgrade-append.txt`). As far as I can tell, what this issue asks for is in place: `next_row_id` in `TableMetadata`, `first-row-id` on new snapshots, and initialization on upgrade. The read side is tracked in #2879. Can this issue be closed? If something is still missing, please let me know what it is. Carrying lineage through rewrites (#3401) seems to be a separate item. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
