0dunay0 opened a new issue, #10992:
URL: https://github.com/apache/paimon/issues/10992

   ### Search before asking
   
   - [x] I searched in the [issues](https://github.com/apache/paimon/issues) 
and found nothing similar.
   
   ### Motivation
   
   `IcebergCommitCallback` builds the metadata of snapshot N on top of 
`v(N-1).metadata.json`. When that file is missing, `createMetadata` falls back 
to `createMetadataWithoutBase`, which reads every data file of snapshot N and 
writes the manifests from scratch. The cost is O(all files) in memory and time, 
and it runs on the committing thread (a streaming sink task or a Spark driver).
   
   The base goes missing when two writers commit to one table, for example a 
streaming sink plus a batch job that clones or compacts. Each callback needs 
the other's metadata first. If one callback is slow or dies, the next commit 
finds no base and rebuilds. The same happens on every later commit until one 
rebuild succeeds. On a table with millions of files the rebuild exhausts the 
heap of the committing process, the process restarts, and the next attempt 
starts again from the same state.
   
   Paimon already protects the happy path well: commit identity in the Iceberg 
summary, rollback and abandoned-timeline handling, head-only publication. What 
is missing is a cheaper way to close a gap in the metadata chain than 
rebuilding it all.
   
   ### Solution
   
   Proposal: when the base is missing, replay the Iceberg metadata from the 
newest valid older version, one Paimon snapshot at a time, instead of 
rebuilding from every data file. It is opt in.
   
   #### Option
   
   A new option, name open for discussion:
   
   `metadata.iceberg.missing-base-strategy`
   
   - `rebuild` (default): today's behaviour, unchanged.
   - `replay`: try replay first and fall back to the existing rebuild when 
replay is not possible.
   
   Replay is attempted only for ordinary append tables: no primary key, no 
deletion vectors, no data evolution, Iceberg format version 2, and storage 
`hive-catalog`, `table-location` or `hadoop-catalog`. Everything else keeps the 
existing rebuild. `rest-catalog` is out of scope for this change.
   
   #### Base selection
   
   Search metadata versions strictly below N, newest first, over a small fixed 
number of candidates. A candidate `vK` is accepted only if:
   
   1. `N - K` is within a configured limit.
   2. Paimon snapshot K exists on disk.
   3. `vK` is readable and its current snapshot id is K.
   4. Its summary carries `paimon-commit-identity` and the value equals the 
identity of Paimon snapshot K. A base with no identity is not eligible. 
Row-count equality is not accepted as evidence.
   5. Its format version matches.
   
   #### Replay
   
   For each Paimon snapshot k in (K, N], build the Iceberg snapshot from that 
snapshot's own delta, using the in-memory metadata of step k-1 as the base. 
After each step, the live row count from the manifests is compared with the 
Paimon snapshot's `totalRecordCount()`. No intermediate `v` files are written. 
Only manifests and manifest lists are created along the way, and the final `vN` 
carries the intermediate snapshots in its snapshot list.
   
   To make this possible without risking a rebuild inside the loop, the 
existing method is split into a build step that returns `COMPLETED`, 
`NEEDS_REBUILD` or `STALE_SOURCE`, and the existing write, publish and cleanup 
tail. Today's method becomes a wrapper around both and behaves as before. This 
is a separate, behaviour-preserving commit.
   
   #### Keeping the output readable: copy on reuse
   
   The risk with reuse is that a manifest referenced by the base is deleted by 
another callback's cleanup while replay runs. An add-only step copies manifest 
references without opening the manifests, so the row-count check would still 
pass and the result would point at a missing file.
   
   Instead of coordinating with deletion, replay owns its inputs. Every 
snapshot retained in the resulting `vN` references manifest lists and manifests 
created by the replay itself:
   
   - Apply the existing retention rule to find the snapshots that belong in 
`vN`.
   - Copy each manifest of those snapshots once (deduplicated by original path) 
to a new name, check the length, and write a new manifest list per retained 
snapshot with only the manifest paths replaced. Snapshot ids, parent ids, 
sequence numbers, timestamps and summaries are preserved.
   - The base `vK`'s own list is always copied as the working base, even if 
retention drops snapshot K from the output.
   - Replay never deletes the original files. Readers that loaded older 
metadata can still need them. The existing retention-driven cleanup is 
unchanged, and retained originals can become unreachable garbage. 
Retention-aware cleanup is left as follow-up work.
   
   A broken snapshot that belongs in the output rejects the candidate. Replay 
reselects once and deletes only the rejected attempt's private files.
   
   The cost is that replay streams the unique manifest bytes through the client 
and writes them again. I plan to measure this on real tables (unique bytes, 
object count, copy time, peak heap) and would like to hear whether reviewers 
accept this approach.
   
   #### Outcomes
   
   | Replay | Source of N | Head | Action |
   |---|---|---|---|
   | completed | valid | N is head | write `vN`, publish hint and catalog, run 
the existing head-only cleanup |
   | completed | valid | head moved on | write `vN` as a reusable base, no 
hint, no catalog, no head-only cleanup |
   | any | replaced by rollback or recreation | any | abandon, delete private 
files, write nothing |
   | expected failure before `vN` is written | valid | N is head | delete 
private files, then run the existing rebuild |
   | expected failure before `vN` is written | valid | head moved on | delete 
private files, log why, return without rebuilding |
   
   Expected failures are a typed set: failed base validation, a step that needs 
a rebuild, a limit breach, a failed row-count assertion, and a missing or 
damaged input. Interruptions and unexpected exceptions propagate. Once the `vN` 
write has been issued, replay never deletes its files. If another writer wins 
`vN`, the durable winner is adopted.
   
   ### Anything else?
   
   - Questions for reviewers: option names and defaults, whether copy on reuse 
is acceptable as the dependency protocol, and whether one PR with the refactor 
in its own commit is the right shape.
   - A proof of concept is in progress and I will link it here.
   - Related: #6107 (a rebuild keeps only the latest snapshot) and PR #9348 
(`metadata.iceberg.sync-full-history`). Replay avoids the rebuild instead of 
changing what a rebuild contains, and it keeps the retained snapshots of the 
base.
   
   ### Are you willing to submit a PR?
   
   - [x] I'm willing to submit a PR!
   
   I will wait for the issue to be assigned before opening a PR, as the 
contributing guide asks.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to