vbabenkoru opened a new pull request, #9246: URL: https://github.com/apache/paimon/pull/9246
Final PR of the 3-PR stack for #8972 — closes #8972. **Stacked on #9244 and #9245**: only the top commit belongs to this PR; the 13 commits below it are the two PRs underneath and will disappear from the diff as they merge. ### Purpose Iceberg supports the VARIANT type from format version 3 onward, but v3 also made row lineage mandatory. Paimon therefore rejected VARIANT columns in the Iceberg compatibility layer for *all* format versions, because the v3 metadata it produced was not lineage-compliant and GA readers would reject the whole table (#8972). With #9244 and #9245, Paimon's v3 writer emits complete row lineage (table metadata, manifest lists, manifest entries), so the reason to block VARIANT on v3 is gone. This PR lifts the guard for `metadata.iceberg.format-version` >= 3: * Tables with VARIANT columns publish Iceberg metadata when the format version is 3, mapping to the Iceberg `variant` type. * Format version 2 keeps the existing clear rejection (with a message pointing at the format-version option), since VARIANT does not exist in v2. The guard lift is intentionally the last commit of the stack: every v3 table that can publish VARIANT is guaranteed to have carried full row lineage from its first commit. ### Tests * `IcebergRowLineageCompatibilityTest#testVariantPublishableWithFormatVersion3`: a VARIANT column publishes on v3, the schema maps to Iceberg `variant`, and lineage fields are present. * `IcebergRowLineageCompatibilityTest#testVariantRejectedWithFormatVersion2`: v2 commits with VARIANT still fail with the explicit error. ### AI notice The code is generated using Fable 5 (with reviews from Codex) but has been verified to run on a real cluster with Flink, Paimon, Iceberg, StarRocks and Snowflake. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
