LuciferYang opened a new pull request, #10845: URL: https://github.com/apache/paimon/pull/10845
### Purpose On a filesystem-managed table, `fast_forward` deletes main's snapshots, schemas and tags from the branch's earliest snapshot onwards together with the LATEST hint, then copies the branch's snapshot, schema and tag directories in listing order. It does this without the lock that commits use. A failure part way leaves main without the snapshot the branch was created from, with missing or truncated snapshot files, and without the schemas its remaining snapshots use. A commit to main can also interleave with the delete and copy. With this change: 1. `fast_forward` holds the lock that commits to main use for the whole operation, and checks that the lock is still valid before each file it writes or deletes, so it stops once a lease-based lock such as the Hive lock has expired. Without a configured lock this is a no-op, as for commits. 2. When main is a prefix of the branch, it only appends. Main is a prefix when every main snapshot from the branch's earliest one (or main's own earliest, if later) to main's latest exists with the branch's file size, main's latest snapshot has the branch's content, and main's schemas are the branch's. It then writes the branch's newer schemas and snapshots in id order with `tryToWriteAtomic`, the tags that differ, and the LATEST hint. Nothing on main is deleted, so after a failure main ends at one of the branch's snapshots and running `fast_forward` again continues from there. Main's latest snapshot is found by listing, not from the LATEST hint. 3. Otherwise main is replaced from the branch's earliest snapshot as before, but schemas are replaced one file at a time in id order instead of deleted up front, and snapshots are copied in id order with `tryToWriteAtomic`. The states that an interrupted `fast_forward` of an earlier version leaves (a LATEST hint naming a missing snapshot, a gap, a truncated snapshot) fail the prefix check, so the next `fast_forward` replaces main and repairs them. Publication is still not atomic. Readers can see some of the branch's snapshots before the rest are copied, and when the branch has newer schemas, main's latest schema switches to the branch's before its snapshots arrive. Other differences from the current behavior: tags on main that point at snapshots it shares with the branch are kept instead of deleted and copied back, and main's EARLIEST hint is no longer overwritten with the branch's. Replacing main moves it back to the branch's earliest snapshot only when main had expired past it. Tables whose catalog manages versions, such as REST, use `CatalogBranchManager` and are not changed. Related to #9690. ### Tests New cases in `FileSystemBranchManagerTest`, with a test FileIO that lists files in name order like HDFS and object stores do, and can fail a chosen write: - `testFailedFastForwardKeepsMainSnapshots`, `testFastForwardResumesAfterPartialFailure`: a snapshot write into main fails. Main stays readable at snapshot 2 or 3, and the retry writes only the missing snapshots and deletes nothing. - `testFastForwardStopsWhenLockLeaseExpires`, `testFastForwardRunsUnderCommitLock`: on both paths, every change to main's snapshot files happens while the lock is held, on the same identifier as main's commits, and an expired lease stops the next write. - `testFastForwardRepairsInterruptedReplace`: four states an interrupted earlier version can leave in main's snapshot directory are repaired to the branch's bytes. - `testFailedReplaceKeepsMainSchemas`, `testFailedSchemaReplaceKeepsMainReadable`: a snapshot or schema write fails while main is replaced, and the snapshot main keeps can still be read. - `testFastForwardCopiesSchemasInIdOrder`, `testFastForwardCopiesBranchSchemas`, `testFastForwardAfterMainSchemaChangeReplacesMain`: schemas 2 to 10 are written in id order; a column added on the branch reaches main; when main and the branch both created schema 2, main ends with the branch's. - `testFastForwardAfterMainDivergedReplacesMain`, `testFastForwardKeepsMainTagsUnchanged`: main's own snapshot 3 is replaced by the branch's; a tag main shares with the branch is not rewritten. - `testFastForwardAppendsAfterMainExpiredSharedSnapshots`, `testReplaceMovesEarliestHintBack`: after main expires the snapshots it shares with the branch, the next `fast_forward` still only appends; replacing a main that expired past the branch's earliest snapshot moves EARLIEST back, also when the first attempt fails and the retry appends. ### API and Format no ### Documentation no -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
