vbabenkoru opened a new pull request, #9348: URL: https://github.com/apache/paimon/pull/9348
### Purpose When Iceberg metadata is created from scratch, only the latest Paimon snapshot was available in Iceberg. This happens when Iceberg compatibility is enabled on a table that already has snapshots, or when the old metadata can no longer be used. Time travel and tags were lost (#6107). This PR has two commits. **1. `metadata.iceberg.sync-full-history`** (opt-in, default `false`): This option replays the full retained Paimon history. It creates new metadata for the earliest retained snapshot. It then applies each later snapshot on top of the previous one. This makes schemas, tags, and the row-id space for format version 3 build up in the same way as live commits. Design points: * **Resumable.** Each replay step saves its metadata file. If a rebuild stops, the next commit continues from the newest metadata already written. The resume candidate is checked first. It must include the start of the retained history. Otherwise, metadata left by a normal single-snapshot rebuild could silently cut off the replayed history. * **Single external transition.** Intermediate steps do not update the version hint, commit to the external catalog, or clean up. Only the final step publishes the result. This means an external catalog sees one change and never points to files deleted by an intermediate step. * **Correct time travel.** Replayed snapshots keep the original Paimon commit timestamps. Tags that point to any replayed snapshot become Iceberg refs. * **Row-id safety.** The rollback and self-heal floors from #9244/#9245 (`inheritUuid`, `lastColumnIdFloor`, `nextRowIdFloor`) are passed through every replay step. This prevents a rebuild from reusing row ids or column ids that abandoned metadata already assigned. * Snapshot retention (`snapshot.num-retained.*`, `snapshot.time-retained`) works on replayed history in the same way as on live commits. **2. Keep live-parity files when creating metadata from scratch.** Creating metadata from scratch used to drop every `DataSplit` that is not raw-convertible. For primary key tables, this removed entire buckets when they contained level-0 files or overlapping key ranges. This silently lost rows that the incremental commit path would have published. A full-history replay then included that loss in every replayed snapshot. Files are now collected one file at a time instead of one split at a time. Raw-convertible splits keep their exact export. Non-raw-convertible splits add every file that the incremental path would accept, along with its deletion vector. Only unmerged level-0 data is excluded. These files are counted, and a warning recommends full compaction as the fix. ### Tests * `IcebergSyncFullHistoryTest` (paimon-core): checks that the default still exposes only the latest snapshot; checks a full replay of retained snapshots with schema changes and a tag in the middle of the history that an Iceberg client can read; checks that an interrupted replay resumes from the newest metadata; checks that a resume candidate without the retained history prefix is rejected; checks that changing the format version rebuilds history while keeping v3 row lineage correct. * `IcebergBootstrapNonRawSplitsTest` (paimon-core): checks that creation from scratch exports compacted files from non-raw-convertible splits; checks full-history replay with non-raw splits. * `IcebergFullHistoryCompatibilityTest` (paimon-iceberg): checks that enabling the option on an existing v3 DV table rebuilds history correctly with the Iceberg 1.8/1.11 readers; checks that an uncompacted DV bucket exports its compacted files. * The full `paimon-iceberg` test suite passes on both configurations: JDK 11 / Iceberg 1.8.1 and JDK 17 / `-Piceberg-ga` (Iceberg 1.11). ### API and Format Adds the optional table option `metadata.iceberg.sync-full-history` with a default of `false`. The default behavior stays the same, except for the live-parity fix in commit 2. That fix only adds files that the incremental commit path would already publish. ### Documentation The option is documented through its description in the generated configuration docs. --- *AI notice: The code is generated using Fable 5 (with reviews from Codex) but has been verified to run on a real cluster with Flink, Paimon, Iceberg, StarRocks and Snowflake.* -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
