vbabenkoru opened a new pull request, #9244:
URL: https://github.com/apache/paimon/pull/9244
First PR of a 3-PR stack for #8972: metadata-level lineage (this PR) ->
manifest-level lineage () -> VARIANT enablement (), both stacked on this branch
in the same fork.
### Purpose
Since row lineage went GA in Iceberg v3 (Iceberg 1.10+), `next-row-id` is a
mandatory table-metadata field: `TableMetadataParser` fails on any v3
metadata
that omits it, and snapshots are expected to carry `first-row-id` /
`added-rows`.
Paimon's Iceberg compatibility layer wrote v3 metadata without these fields,
so
every GA reader — including AWS Glue's REST catalog and Iceberg 1.11 clients
—
rejects Paimon v3 tables outright (see #8972).
This PR makes the metadata level spec-compliant:
* `IcebergMetadata` gains a nullable `next-row-id` field (serialized only
when
set, so v2 metadata is byte-identical to before).
* Metadata created without a base starts the row-id space at 0 and records
the
snapshot's `first-row-id` / `added-rows`; subsequent commits advance
`next-row-id` by the committed record count.
* v3 base metadata written by older Paimon (no `next-row-id`) self-heals: the
next commit detects the missing field and recreates metadata from scratch,
preserving the table UUID.
* The REST metadata committer now propagates `format-version` when
registering
a table with a REST catalog (previously tables were always created as v2),
and upgrades the catalog table when the local format version is newer.
Scope notes:
* Manifest-level lineage (`first_row_id` in manifest lists and manifest
entries, Iceberg spec fields 520/142) is the next PR in the stack; until it
lands, v3 stays writer-side incomplete at the manifest level, which GA
readers tolerate for reads but which the follow-up makes spec-compliant.
* VARIANT publication remains gated off for all format versions in this PR;
it
is enabled for v3 at the top of the stack once the writer is fully
lineage-compliant.
* `EnableRowLineage` is deliberately never sent to REST catalogs: the update
was removed in Iceberg 1.10 GA and breaks Glue commits.
### Tests
* `IcebergMetadataNextRowIdTest`: JSON round-trip of the new field, absence
on
v2.
* `IcebergRowLineageCompatibilityTest`: fresh v3 metadata carries
`next-row-id` / `first-row-id` / `added-rows` and parses with the bundled
Iceberg `TableMetadataParser`; values advance across commits; v2 metadata
is unchanged (no lineage keys in JSON); self-heal regenerates valid
metadata
when the base lacks `next-row-id`; tags survive regeneration.
* `IcebergRestMetadataCommitterTest`: v3 tables register with format-version
3
via the REST `TableBuilder`; existing v2 catalog tables are upgraded when
Paimon moves to v3.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_012py5Xggbrgk9DMHz31WeK1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
