JingsongLi commented on code in PR #11017:
URL: https://github.com/apache/paimon/pull/11017#discussion_r4561893355
##########
paimon-core/src/main/java/org/apache/paimon/operation/FileStoreCommitImpl.java:
##########
@@ -1817,6 +1946,24 @@ private Pair<Boolean, CommitCallback.Context>
rollbackToAsLatestOnceWithoutLock(
// the snapshots between the target and the previous latest, breaking
the global uniqueness
// of _ROW_ID. Keep the larger of the previous latest and the target
nextRowId.
Long nextRowId = maxNextRowId(latest.nextRowId(),
targetSnapshot.nextRowId());
+ Map<String, String> rollbackProperties =
+ SequenceSnapshotProperties.mergeGuardMaxSequenceNumber(
+ targetSnapshot.properties(), latest, deltaFiles,
false);
+ rollbackProperties =
+ DeleteOrderingGuardGeneration.merge(
+ rollbackProperties,
+ latest,
+ options.deleteOrderingGuardsEnabled(),
+ true,
+ null,
+ partitionType);
+ rollbackProperties =
+ DeleteOrderingGuardPartitions.merge(
+ rollbackProperties,
+ targetSnapshot,
Review Comment:
[P1] Validate restored guard policy before publishing rollback-as-latest
This path restores the target guard metadata/index manifest without checking
it against the current persisted ordering policy. I followed the documented
migration sequence: capture an ascending delete/version4, append hidden
version2 and tag it, truncate the entire table, persist descending order,
reopen and successfully write/read new data. `rollbackToAsLatest` of the old
guarded target then returns true and advances the head, but a fresh public read
of the latest table fails with `Delete ordering guard ordering/deletion policy
changed since schema 0; an ordering migration is required.` Ordinary commits
validate the segment headers; this bypass publishes an incompatible state
first. Reject an incompatible target before publication, or atomically restore
a compatible effective policy with its guards.
##########
paimon-core/src/main/java/org/apache/paimon/table/AbstractFileStoreTable.java:
##########
@@ -747,6 +756,35 @@ public void rollbackTo(String tagName) {
}
}
+ private void validateDestructiveRollback() {
+ SnapshotManager snapshotManager = snapshotManager();
+ Snapshot latest;
+ try {
+ latest = snapshotManager.latestSnapshot();
+ } catch (RuntimeException e) {
+ if (!(e.getCause() instanceof FileNotFoundException)) {
+ throw e;
+ }
+ // Rollback can recover from a latest hint pointing to a missing
snapshot. Check the
+ // latest existing snapshot so that recovery still respects
retained guard history.
+ try {
+ latest =
+ snapshotManager.safelyGetAllSnapshots().stream()
+ .max(Comparator.comparingLong(Snapshot::id))
+ .orElse(null);
+ } catch (IOException ex) {
+ throw new UncheckedIOException(ex);
+ }
+ }
+ if (coreOptions().deleteOrderingGuardsEnabled()
+ || DeleteOrderingGuardGeneration.isTracked(latest)
+ || DeleteOrderingGuardFile.mayContainGuards(latest)) {
+ throw new UnsupportedOperationException(
Review Comment:
[P1] Apply destructive-history fencing to fastForward as well
The new rejection protects `rollbackTo`, but public `fastForward` still
copies an old branch's generation onto main through FileSystemBranchManager. I
reproduced this sequentially, with the writer already closed: commit key1,
create an old-history branch, prepare key2 under generation0, truncate main,
and verify that the exact prepared committable is rejected with `generation
changed`. After `table.fastForward("old-history")`, retrying that same
committable succeeds and a real Java read returns keys1/2. The branch
legitimately restores key1, but must not revive the separately fenced key2
message. Reject guarded fast-forward too, or publish the branch contents with a
fresh destination generation that cannot rewind; no concurrent writer is
required for this failure.
##########
paimon-core/src/main/java/org/apache/paimon/table/source/snapshot/SnapshotReaderImpl.java:
##########
@@ -530,12 +614,33 @@ private Plan toIncrementalPlan(
@Nullable Snapshot afterSnapshot,
@Nullable Long afterWatermark,
Map<BinaryRow, Map<Integer, List<ManifestEntry>>> afterFiles) {
+ // Diff must resolve the complete logical state at each endpoint.
Removing a shared
+ // data file can hide a before-image needed to compare the visible
rows after guard
+ // resolution.
+ boolean resolveGuards =
+ !isStreaming ||
DeleteOrderingGuardFile.mayContainGuards(afterSnapshot);
+ Map<Pair<BinaryRow, Integer>, List<IndexFileMeta>> beforeGuards =
+ resolveGuards ? incrementalGuards(beforeSnapshot.get()) :
Collections.emptyMap();
Review Comment:
[P1] Avoid requiring expired predecessor metadata for current APPEND deltas
This unconditionally resolves `beforeSnapshot` whenever the after-snapshot
carries guard history, even when an APPEND delta has no before files. With a
full-compacted delete guard, I appended a stale key1/version2 plus fresh
key2/version5, expired all predecessors while retaining current snapshot3, and
verified that its data/guard files and batch read remain valid. A public
file-monitor stream restored at3 then fails here loading expired snapshot2; the
identical unguarded control succeeds. Flink's default PK lookup
FILE_MONITOR/AllDelta path also calls this code. Current-delta consumption
should not require a deleted predecessor solely to discover an empty before
side. Avoid that load when unnecessary, or preserve the required predecessor
guard references independently of expired snapshot metadata.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]