JingsongLi commented on code in PR #11017:
URL: https://github.com/apache/paimon/pull/11017#discussion_r4522937843


##########
paimon-core/src/main/java/org/apache/paimon/index/DeleteOrderingGuardFile.java:
##########
@@ -0,0 +1,320 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.paimon.index;
+
+import org.apache.paimon.Snapshot;
+import org.apache.paimon.fs.FileIO;
+import org.apache.paimon.fs.Path;
+import org.apache.paimon.fs.PositionOutputStream;
+import org.apache.paimon.fs.SeekableInputStream;
+import org.apache.paimon.io.cache.CacheManager;
+import org.apache.paimon.lookup.sort.SortLookupStoreReader;
+import org.apache.paimon.lookup.sort.SortLookupStoreWriter;
+import org.apache.paimon.memory.MemorySlice;
+import org.apache.paimon.reader.RecordReader;
+import org.apache.paimon.sst.BlockIterator;
+import org.apache.paimon.sst.SstFileReader;
+
+import javax.annotation.Nullable;
+
+import java.io.Closeable;
+import java.io.DataInputStream;
+import java.io.DataOutputStream;
+import java.io.IOException;
+import java.util.Comparator;
+import java.util.Map;
+
+import static org.apache.paimon.utils.Preconditions.checkArgument;
+import static org.apache.paimon.utils.Preconditions.checkState;
+
+/**
+ * Immutable bucket-local SST segments of {@link DeleteOrderingGuard}s.
+ *
+ * <p>A segment starts with magic, format version, and table schema id, 
followed by the native SST
+ * blocks and footer. Keys consist of primary-key fields; values contain the 
internal sequence
+ * number and projected ordering fields. Native SST blocks provide checksums 
and a block index.
+ * Reads stream the segment for merge processing, with no per-event guard 
lookup or ingestion-time
+ * rewrite of previous segments.
+ *
+ * <p>The path factory defines the bucket scope. Callers must commit returned 
index references
+ * atomically with the corresponding data changes. Finishing a writer only 
produces an uncommitted
+ * file; it does not publish ordering protection. No bound is retired by this 
storage layer.
+ * Instances and their serializers are not thread-safe.
+ */
+public class DeleteOrderingGuardFile extends IndexFile {
+
+    public static final String DELETE_ORDERING_GUARD = "DELETE_ORDERING_GUARD";
+    public static final String SNAPSHOT_MAX_SEQUENCE_NUMBER =
+            "delete-ordering-guards.max-sequence-number";
+
+    /** Conservative hint, retained through resets along with the internal 
sequence watermark. */
+    public static boolean mayContainGuards(@Nullable Snapshot snapshot) {

Review Comment:
   **[P1] Prevent unsupported clients from exposing guarded rows or dropping 
guard discovery**
   
   I reproduced this with one filesystem PK table and the Java and PyPaimon 
clients from `31cf7f2cd`. Java compacted a delete for key 1 at version 4, then 
appended version 2. Java returned no rows. The normal PyPaimon scan returned 
the stale row. I then appended an unrelated key from Python. The new snapshot 
kept the same index manifest and lost the guard properties, so Java now 
returned the stale row too. This property alone gates discovery of the 
still-present guard. Please reject guarded table reads/writes in unsupported 
clients before they can expose data or publish metadata, and account for older 
clients during rollout.



##########
paimon-core/src/main/java/org/apache/paimon/operation/FileStoreCommitImpl.java:
##########
@@ -1817,6 +1938,24 @@ private Pair<Boolean, CommitCallback.Context> 
rollbackToAsLatestOnceWithoutLock(
         // the snapshots between the target and the previous latest, breaking 
the global uniqueness
         // of _ROW_ID. Keep the larger of the previous latest and the target 
nextRowId.
         Long nextRowId = maxNextRowId(latest.nextRowId(), 
targetSnapshot.nextRowId());
+        Map<String, String> rollbackProperties =
+                SequenceSnapshotProperties.mergeGuardMaxSequenceNumber(
+                        targetSnapshot.properties(), latest, deltaFiles, 
false);
+        rollbackProperties =
+                DeleteOrderingGuardGeneration.merge(

Review Comment:
   **[P1] Keep a destructive rollback from reviving fenced writers**
   
   On a guarded fixed-bucket table I committed key 1, staged key 2 under 
generation 0, and truncated. The staged commit was rejected. Then I used the 
public `table.rollbackTo(snapshot1)` API and retried the exact staged message. 
It succeeded and Java read `[1,2]`. The ordinary rollback path used by 
`sys.rollback_to` restores the old snapshot generation, while the generation 
change here only covers rollback-as-latest. Please fence the public path with 
state that survives a head rewind, or reject destructive rollback on guarded 
histories and route users to an atomic as-latest reset.



##########
paimon-core/src/main/java/org/apache/paimon/KeyValueFileStore.java:
##########
@@ -105,17 +111,30 @@ public BucketMode bucketMode() {
 
     @Override
     public MergeFileSplitRead newRead() {
+        validateDeleteOrderingGuardReadScope();

Review Comment:
   **[P2] Avoid loading the current snapshot per reader on guard-disabled 
tables**
   
   This runs before the reader sees its split. 
`validateDeleteOrderingGuardReadScope()` unconditionally fetches 
`latestSnapshot`, even for an ordinary PK table and even if the split is pinned 
to an older snapshot. Spark constructs a reader per partition, so a REST-backed 
query adds a current-snapshot request for each task instead of using the 
coordinator's planned metadata. A live catalog failure can also block a read 
whose pinned split and files are already available. Could this check use the 
planned split's guard metadata and stay out of the default-off reader 
initialization path?



##########
paimon-core/src/main/java/org/apache/paimon/operation/FileStoreCommitImpl.java:
##########
@@ -1241,6 +1304,27 @@ private CommitResult tryCommitOnceWithoutLock(
         List<ManifestEntry> deltaFiles = changes.tableFiles;
         List<ManifestEntry> changelogFiles = changes.changelogFiles;
         List<IndexManifestEntry> indexFiles = changes.indexFiles;
+        boolean replacesGuards =
+                indexFiles.stream()
+                        .anyMatch(
+                                entry ->
+                                        entry.kind() == FileKind.DELETE
+                                                && 
DELETE_ORDERING_GUARD.equals(
+                                                        
entry.indexFile().indexType()));
+        boolean publishesGuards =

Review Comment:
   **[P1] Fence compactions built without a newly committed guard**
   
   Using real writers with both lookup and full-compaction changelog producers, 
I opened writer B and buffered key 1/version 2. Writer A then deleted version 
4, fully compacted and published a guard-only snapshot. B compacted the 
buffered version 2 with its older view and committed successfully. A native 
stream resumed after A's snapshot emitted `+I(1,2)`, while a batch scan of the 
same table returned no rows. This validation checks active guards only when the 
commit retires guards. Please validate the guard source set used to build 
data/changelog output for the affected bucket, and refresh it before retrying a 
changed set.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to