0dunay0 opened a new pull request, #10995:
URL: https://github.com/apache/paimon/pull/10995

   ### Purpose
   
   Linked issue: close #10992
   
   When the Iceberg metadata of the previous snapshot is missing, the commit 
callback rebuilds the metadata from every data file of the table. On a large 
table this can exhaust memory.
   
   This adds `metadata.iceberg.missing-base-strategy` (`rebuild` by default, or 
`replay`). With `replay`, the callback takes the newest older metadata version 
that was built from the Paimon snapshot it names and extends it one Paimon 
snapshot at a time through the existing incremental path. It falls back to the 
rebuild when the table is not eligible, no base qualifies, a limit is exceeded, 
or a check fails. The default behavior is unchanged.
   
   - Eligible tables: append tables without deletion vectors or data evolution, 
format version 2, with table-location, hadoop-catalog or hive-catalog storage.
   - The replay copies the manifests and lists it keeps, so the new metadata 
owns every file it reads. It checks the row count of each step against Paimon 
and writes only the final version file.
   - Work is bounded by `metadata.iceberg.replay.max-snapshots`, 
`max-delta-entries`, `max-base-manifest-bytes` and `max-duration`.
   - At the head, the usual retention cleanup runs for the older originals that 
the new metadata drops. Originals of retained snapshots are never deleted.
   
   The PR has three commits:
   1. Split building Iceberg metadata from exposing it. No behavior change.
   2. Skip an unreadable older metadata version when cleaning up after a 
rebuild. A truncated old file made every later rebuild fail. This also affects 
the default strategy, so it can move to its own PR if preferred.
   3. The replay.
   
   ### Tests
   
   - New `IcebergGapReplayTest` (5 tests): replay over append, overwrite and 
compaction gaps; base selection; fallbacks and limits; concurrent changes; 
failure states.
   - New `IcebergCompatibilityTest#testExpireAllBeforeSkipsUnreadableMetadata`.
   - All 204 `Iceberg*Test` tests pass, with spotless and checkstyle.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to