timothyw553 opened a new pull request, #17446: URL: https://github.com/apache/iceberg/pull/17446
## What this prototype tests This draft explores whether an Iceberg v2-to-v3 upgrade can preserve the existing snapshot chain while assigning `first_row_id` to live data-file entries. It adds an opt-in `RewriteManifests.assignFirstRowIds` operation that: - rewrites live data manifests without changing the active data-file set; - writes each rewritten entry as `EXISTING`; - preserves the entry's snapshot ID, data sequence number, and file sequence number; - validates the reserved row-ID range, rejects overlaps and missing assignments, and fails if the current snapshot changes concurrently. The Spark test checkpoints a v2 streaming read, performs the prototype upgrade, appends another row, and resumes twice from the same checkpoint. The original snapshot remains resolvable and each logical row is emitted once. ## Why this is a research prototype This is **not** the production fix for a Delta-backed Managed Iceberg table and must not ship by itself. The prototype deliberately preserves every existing Iceberg data sequence number. That is required when rewriting an entry as `EXISTING`. However, a Managed Iceberg v3 file also needs its Iceberg `data_sequence_number` to match Delta's `defaultRowCommitVersion`. A v2 Iceberg sequence number is not generally the same as the corresponding Delta log version. Therefore this change proves the snapshot-ancestry and streaming-checkpoint behavior, but it does not solve the cross-format sequence-number mismatch. A production design needs either physical files at new paths, which may legally receive new sequence numbers, or a broader protocol design. ## Validation The prototype passed these checks on its original Iceberg 1.11 base: ```text ./gradlew :iceberg-core:test --tests org.apache.iceberg.TestRowLineageAssignment ./gradlew :iceberg-spark:iceberg-spark-4.0_2.13:test --tests org.apache.iceberg.spark.source.TestStructuredStreamingRead3 ./gradlew revapi git diff --check ``` The recorded results contain 25 passing `TestRowLineageAssignment` tests and 8 passing `TestStructuredStreamingRead3` tests, including `testResumeCheckpointAfterV2ToV3RowIdAssignment`. The branch has been rebased onto current `main`. A local rerun on `main` could not reach test execution because Gradle could not resolve `repo.maven.apache.org`; CI should provide the current-base result. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
