LuciferYang commented on issue #9690: URL: https://github.com/apache/paimon/issues/9690#issuecomment-6435605951
Hi @tub, I'd like to help get this into Paimon. Are you already working on it? If so, I'm happy to review or split the work with you. If not, I can pick it up. The design I have in mind mostly follows Iceberg, so the same session config works for both formats: - Table option `write.wap.enabled` plus session config `spark.wap.branch`. Writes go to that branch, creating it from main's latest snapshot if it doesn't exist yet. Reads switch to the branch only when it already exists. - Publish with the existing `CALL sys.fast_forward(table, branch)`. Each commit on the branch stays a separate snapshot on main, so your downstream Flink job still gets small snapshots. - On WAP tables, publishing fails if main got new commits after the branch was created. Today `fast_forward` drops those commits without any error. Two things in core need work first: 1. `fast_forward` isn't atomic. `FileSystemBranchManager#fastForward` deletes main's snapshots from the branch's first snapshot onward, then copies the branch's snapshot files over one at a time, without taking the commit lock. If it fails halfway, main can lose its latest snapshot or end up with only part of the batch. 2. A branch can only be created from a tag or as an empty branch. WAP needs one created from a snapshot, which for the REST catalog means an optional `fromSnapshot` in `CreateBranchRequest`. @JingsongLi does this direction look right to you? Since it touches the REST spec and how `fast_forward` writes snapshot files, I plan to write it up as a PIP first, and keep the publish semantics consistent with the database-level forward in #9879. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
