0dunay0 opened a new pull request, #10995: URL: https://github.com/apache/paimon/pull/10995
### Purpose Linked issue: close #10992 When the Iceberg metadata of the previous snapshot is missing, the commit callback rebuilds the metadata from every data file of the table. On a large table this can exhaust memory. This adds `metadata.iceberg.missing-base-strategy` (`rebuild` by default, or `replay`). With `replay`, the callback takes the newest older metadata version that was built from the Paimon snapshot it names and extends it one Paimon snapshot at a time through the existing incremental path. It falls back to the rebuild when the table is not eligible, no base qualifies, a limit is exceeded, or a check fails. The default behavior is unchanged. - Eligible tables: append tables without deletion vectors or data evolution, format version 2, with table-location, hadoop-catalog or hive-catalog storage. - The replay copies the manifests and lists it keeps, so the new metadata owns every file it reads. It checks the row count of each step against Paimon and writes only the final version file. - Work is bounded by `metadata.iceberg.replay.max-snapshots`, `max-delta-entries`, `max-base-manifest-bytes` and `max-duration`. - At the head, the usual retention cleanup runs for the older originals that the new metadata drops. Originals of retained snapshots are never deleted. The PR has three commits: 1. Split building Iceberg metadata from exposing it. No behavior change. 2. Skip an unreadable older metadata version when cleaning up after a rebuild. A truncated old file made every later rebuild fail. This also affects the default strategy, so it can move to its own PR if preferred. 3. The replay. ### Tests - New `IcebergGapReplayTest` (5 tests): replay over append, overwrite and compaction gaps; base selection; fallbacks and limits; concurrent changes; failure states. - New `IcebergCompatibilityTest#testExpireAllBeforeSkipsUnreadableMetadata`. - All 204 `Iceberg*Test` tests pass, with spotless and checkstyle. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
