vbabenkoru opened a new pull request, #9245: URL: https://github.com/apache/paimon/pull/9245
Second PR of the 3-PR stack for #8972, **stacked on #9244**: GitHub cannot target a fork branch as base, so this PR shows the full 13-commit diff against master — **only the top 9 commits belong to this PR**; the first 4 are #9244 and will disappear from the diff once it merges and this branch is rebased. The VARIANT enablement PR sits on top of this one. ### Purpose #9244 made v3 *table metadata* carry row lineage. The Iceberg v3 spec also requires lineage at the manifest level: `first_row_id` in the manifest list (field 520) and in data-file entries (field 142), with defined inheritance — an ADDED entry with null `first_row_id` inherits from the manifest, and the inherited value must be materialized when an entry is rewritten as EXISTING/DELETED. Without these fields, GA readers cannot expose stable `_row_id` values, and row IDs silently change across snapshots. This PR completes writer-side v3 compliance with stable, synthetic row IDs: * Manifest lists gain `first_row_id` (field 520); manifest entries gain `first_row_id` on the data file (field 142). Both are written only for format version 3; v2 file shapes are unchanged (pinned by tests). * At manifest-list write time, every DATA manifest without an assigned `first_row_id` receives one from the table's `next-row-id` watermark, which advances by the rows that actually inherit an ID. DELETE manifests are never assigned, per spec. * When a v3 manifest is rewritten (delete compaction, metadata compaction), entries that inherited their `first_row_id` get the value materialized into the rewritten file, so IDs stay stable across rewrites. * Accounting is migration-safe: v3 manifests written before this PR contain EXISTING/DELETED entries without materialized IDs; the true number of inheriting rows is computed by reading such manifests once, so `added-rows` / `next-row-id` never double-assign a row-id range. * Snapshot expiration now decides manifest liveness by physical path instead of value equality of the manifest-list entry. Assigning `first_row_id` to a carried-over manifest re-lists the same file with a different list-level field; the old equality check deleted the shared file while newer snapshots still referenced it. * `paimon-iceberg` tests run against Iceberg 1.11 (GA row lineage) so compliance is validated by the reference implementation, not only by Paimon's own writer/reader. `iceberg-core` remains provided-scope; the runtime classpath still decides the deployed Iceberg version. Known follow-up (performance, not correctness): computing the true inheriting row count re-reads manifests that were rewritten in the same commit; this can be folded into the manifest write itself later. ### Tests `IcebergRowLineageCompatibilityTest` grows from 5 to 17 cases, including: * field 520/142 presence and round-trip for v3, absence for v2 (shape pin); * assignment starts at 0 and advances across commits; * multi-bucket (`bucket = 2`) assignment is stable across snapshots — IDs never shift when unrelated buckets commit; * rewrites (delete compaction, metadata compaction) materialize inherited IDs and keep them identical before/after; * migration from metadata-only lineage: legacy manifests get IDs exactly once, ranges stay unique, `added-rows` counts inherited legacy rows; * GA reader matrix: Apache Iceberg 1.11 reads the produced tables and observes the expected `first_row_id` per data file; * expiration regression: a manifest shared between a pre-assignment manifest list and its assigned successor survives snapshot expiration and all retained snapshots stay readable. `IcebergRestMetadataCommitterTest` covers v3 registration and recreation against a REST catalog with value-equality checks vs the local metadata. ### AI notice The code is generated using Fable 5 (with reviews from Codex) but has been verified to run on a real cluster with Flink, Paimon, Iceberg, StarRocks and Snowflake. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
