JingsongLi commented on code in PR #11017: URL: https://github.com/apache/paimon/pull/11017#discussion_r4522937843
########## paimon-core/src/main/java/org/apache/paimon/index/DeleteOrderingGuardFile.java: ########## @@ -0,0 +1,320 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.paimon.index; + +import org.apache.paimon.Snapshot; +import org.apache.paimon.fs.FileIO; +import org.apache.paimon.fs.Path; +import org.apache.paimon.fs.PositionOutputStream; +import org.apache.paimon.fs.SeekableInputStream; +import org.apache.paimon.io.cache.CacheManager; +import org.apache.paimon.lookup.sort.SortLookupStoreReader; +import org.apache.paimon.lookup.sort.SortLookupStoreWriter; +import org.apache.paimon.memory.MemorySlice; +import org.apache.paimon.reader.RecordReader; +import org.apache.paimon.sst.BlockIterator; +import org.apache.paimon.sst.SstFileReader; + +import javax.annotation.Nullable; + +import java.io.Closeable; +import java.io.DataInputStream; +import java.io.DataOutputStream; +import java.io.IOException; +import java.util.Comparator; +import java.util.Map; + +import static org.apache.paimon.utils.Preconditions.checkArgument; +import static org.apache.paimon.utils.Preconditions.checkState; + +/** + * Immutable bucket-local SST segments of {@link DeleteOrderingGuard}s. + * + * <p>A segment starts with magic, format version, and table schema id, followed by the native SST + * blocks and footer. Keys consist of primary-key fields; values contain the internal sequence + * number and projected ordering fields. Native SST blocks provide checksums and a block index. + * Reads stream the segment for merge processing, with no per-event guard lookup or ingestion-time + * rewrite of previous segments. + * + * <p>The path factory defines the bucket scope. Callers must commit returned index references + * atomically with the corresponding data changes. Finishing a writer only produces an uncommitted + * file; it does not publish ordering protection. No bound is retired by this storage layer. + * Instances and their serializers are not thread-safe. + */ +public class DeleteOrderingGuardFile extends IndexFile { + + public static final String DELETE_ORDERING_GUARD = "DELETE_ORDERING_GUARD"; + public static final String SNAPSHOT_MAX_SEQUENCE_NUMBER = + "delete-ordering-guards.max-sequence-number"; + + /** Conservative hint, retained through resets along with the internal sequence watermark. */ + public static boolean mayContainGuards(@Nullable Snapshot snapshot) { Review Comment: **[P1] Prevent unsupported clients from exposing guarded rows or dropping guard discovery** I reproduced this with one filesystem PK table and the Java and PyPaimon clients from `31cf7f2cd`. Java compacted a delete for key 1 at version 4, then appended version 2. Java returned no rows. The normal PyPaimon scan returned the stale row. I then appended an unrelated key from Python. The new snapshot kept the same index manifest and lost the guard properties, so Java now returned the stale row too. This property alone gates discovery of the still-present guard. Please reject guarded table reads/writes in unsupported clients before they can expose data or publish metadata, and account for older clients during rollout. ########## paimon-core/src/main/java/org/apache/paimon/operation/FileStoreCommitImpl.java: ########## @@ -1817,6 +1938,24 @@ private Pair<Boolean, CommitCallback.Context> rollbackToAsLatestOnceWithoutLock( // the snapshots between the target and the previous latest, breaking the global uniqueness // of _ROW_ID. Keep the larger of the previous latest and the target nextRowId. Long nextRowId = maxNextRowId(latest.nextRowId(), targetSnapshot.nextRowId()); + Map<String, String> rollbackProperties = + SequenceSnapshotProperties.mergeGuardMaxSequenceNumber( + targetSnapshot.properties(), latest, deltaFiles, false); + rollbackProperties = + DeleteOrderingGuardGeneration.merge( Review Comment: **[P1] Keep a destructive rollback from reviving fenced writers** On a guarded fixed-bucket table I committed key 1, staged key 2 under generation 0, and truncated. The staged commit was rejected. Then I used the public `table.rollbackTo(snapshot1)` API and retried the exact staged message. It succeeded and Java read `[1,2]`. The ordinary rollback path used by `sys.rollback_to` restores the old snapshot generation, while the generation change here only covers rollback-as-latest. Please fence the public path with state that survives a head rewind, or reject destructive rollback on guarded histories and route users to an atomic as-latest reset. ########## paimon-core/src/main/java/org/apache/paimon/KeyValueFileStore.java: ########## @@ -105,17 +111,30 @@ public BucketMode bucketMode() { @Override public MergeFileSplitRead newRead() { + validateDeleteOrderingGuardReadScope(); Review Comment: **[P2] Avoid loading the current snapshot per reader on guard-disabled tables** This runs before the reader sees its split. `validateDeleteOrderingGuardReadScope()` unconditionally fetches `latestSnapshot`, even for an ordinary PK table and even if the split is pinned to an older snapshot. Spark constructs a reader per partition, so a REST-backed query adds a current-snapshot request for each task instead of using the coordinator's planned metadata. A live catalog failure can also block a read whose pinned split and files are already available. Could this check use the planned split's guard metadata and stay out of the default-off reader initialization path? ########## paimon-core/src/main/java/org/apache/paimon/operation/FileStoreCommitImpl.java: ########## @@ -1241,6 +1304,27 @@ private CommitResult tryCommitOnceWithoutLock( List<ManifestEntry> deltaFiles = changes.tableFiles; List<ManifestEntry> changelogFiles = changes.changelogFiles; List<IndexManifestEntry> indexFiles = changes.indexFiles; + boolean replacesGuards = + indexFiles.stream() + .anyMatch( + entry -> + entry.kind() == FileKind.DELETE + && DELETE_ORDERING_GUARD.equals( + entry.indexFile().indexType())); + boolean publishesGuards = Review Comment: **[P1] Fence compactions built without a newly committed guard** Using real writers with both lookup and full-compaction changelog producers, I opened writer B and buffered key 1/version 2. Writer A then deleted version 4, fully compacted and published a guard-only snapshot. B compacted the buffered version 2 with its older view and committed successfully. A native stream resumed after A's snapshot emitted `+I(1,2)`, while a batch scan of the same table returned no rows. This validation checks active guards only when the commit retires guards. Please validate the guard source set used to build data/changelog output for the affected bucket, and refresh it before retrying a changed set. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
