Akash3121 commented on code in PR #11252:
URL: https://github.com/apache/paimon/pull/11252#discussion_r4618877730


##########
paimon-core/src/main/java/org/apache/paimon/append/dataevolution/DataEvolutionCompactionCommitPreparation.java:
##########
@@ -41,9 +46,6 @@ public List<CommitMessage> prepare(List<CommitMessage> 
messages) {
         result.addAll(

Review Comment:
   Now that `DataEvolutionCompactGlobalIndexDropper` is removed and preparation 
only emits `DELETION_VECTORS_INDEX` changes, nothing prepares `FileKind.DELETE` 
global-index entries against an older snapshot anymore 
(`DataEvolutionCompactDeletionVectorRewriter#checkNoCompactIndexChanges` also 
rejects pre-existing compact index changes).
    
    Should we also clean up the leftover filter in 
`FileStoreCommitImpl#dataEvolutionChangesProvider` (lines 1031–1043, `// 
Replace global-index deletions prepared against an older snapshot.`) to `new 
ArrayList<>(changes.indexFiles)` and update 
`FileStoreCommitTest#testMaterializedCompactionOnlyRefreshesGlobalIndexInSameBucket`
 accordingly?



##########
paimon-core/src/test/java/org/apache/paimon/table/DataEvolutionDeletionVectorTest.java:
##########
@@ -1039,6 +1044,80 @@ public void testMaterializeDeletionVectors() throws 
Exception {
                 .doesNotContainAnyElementsOf(oldAnchorFiles);
     }
 
+    @ParameterizedTest
+    @ValueSource(strings = {"before", "after", "retry", "fully-deleted"})
+    public void testMaterializeInvalidatesGlobalIndexesAtCommit(String 
indexTiming)
+            throws Exception {
+        FileStoreTable table = 
createPartitionedReassignTable("materialize_global_indexes", false);
+        writePartitionRows(table, "a", 0, 1, 2);
+        writePartitionRows(table, "b", 3, 4, 5);
+        BinaryRow partitionA = partition(table, "a");
+        boolean fullyDeleted = "fully-deleted".equals(indexTiming);
+        commitDeletionVectors(
+                table,
+                partitionA,
+                Collections.singletonList(
+                        new DvSpec(
+                                new Range(0, 2),
+                                fullyDeleted ? new long[] {0, 1, 2} : new 
long[] {1})));
+        commitGlobalIndex(
+                table, partition(table, "b"), 
globalIndexFile("untouched-index", new Range(3, 5)));
+        if ("before".equals(indexTiming) || fullyDeleted) {
+            commitGlobalIndex(table, partitionA, globalIndexFile("old-index", 
new Range(0, 2)));
+        }

Review Comment:
   In the `"retry"` case, `partitionA` currently has no global index when 
`snapshot` is taken (`"old-index"` is only committed for `"before"` and 
`"fully-deleted"`). As a result:
    1. The assertion at lines 1071–1079 is vacuous for `"retry"` because 
`partitionA` has no global index at `snapshot`.
    2. Commit attempt 1 does not produce any global-index `DELETE` entry for 
`partitionA`, so the retry does not verify that attempt 1's `DELETE` entries 
are discarded/rebuilt cleanly on attempt 2 without causing a duplicate or 
missing-file `IllegalStateException` in `GlobalIndexCombiner#combine`.
    
    If we also commit `"old-index"` before preparation when 
`"retry".equals(indexTiming)` and compact it into `"concurrent-index"` inside 
`tryToWriteAtomic` (via `indexCompactMessage(partitionA, oldIndex, 
globalIndexFile("concurrent-index", new Range(0, 2)))`), the `"retry"` case 
will verify both that attempt 1's stale `DELETE old-index` is dropped and that 
`"concurrent-index"` from the retried snapshot is invalidated.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to