JingsongLi commented on code in PR #11102:
URL: https://github.com/apache/paimon/pull/11102#discussion_r4561385171


##########
paimon-core/src/main/java/org/apache/paimon/operation/FileStoreCommitImpl.java:
##########
@@ -333,6 +348,120 @@ public List<ManifestCommittable> 
filterCommitted(List<ManifestCommittable> commi
         }
     }
 
+    @Nullable
+    private ManifestCommittable recoverCompact(ManifestCommittable 
committable) {
+        List<CommitMessage> messages = new ArrayList<>();
+        Set<Path> outputs = new HashSet<>();
+        DataFilePathFactories dataFactories = new 
DataFilePathFactories(pathFactory);
+        IndexFilePathFactories indexFactories = new 
IndexFilePathFactories(pathFactory);
+        for (CommitMessage message : committable.fileCommittables()) {
+            CompactIncrement compact = ((CommitMessageImpl) 
message).compactIncrement();
+            if (compact.isEmpty()) {
+                continue;
+            }
+            messages.add(
+                    new CommitMessageImpl(
+                            message.partition(),
+                            message.bucket(),
+                            message.totalBuckets(),
+                            DataIncrement.emptyIncrement(),
+                            compact,
+                            message.checkFromSnapshot()));
+            DataFilePathFactory dataFactory =
+                    dataFactories.get(message.partition(), message.bucket());
+            compact.compactAfter().forEach(file -> 
outputs.addAll(file.collectFiles(dataFactory)));
+            compact.changelogFiles()
+                    .forEach(file -> 
outputs.addAll(file.collectFiles(dataFactory)));
+            IndexPathFactory indexFactory =
+                    indexFactories.get(message.partition(), message.bucket());
+            compact.newIndexFiles().forEach(file -> 
outputs.add(indexFactory.toPath(file)));
+        }
+        if (messages.isEmpty()) {
+            return null;
+        }
+        // An expired or orphan-cleaned output cannot be replayed. Skip the 
whole identifier,
+        // rather than publishing only some of its buckets. I/O failures must 
still fail recovery.
+        for (Path output : outputs) {
+            Path physicalPath =
+                    
CompactedChangelogPathResolver.resolveCompactedChangelogPath(output);
+            try {
+                if (!fileIO.exists(physicalPath)) {
+                    LOG.warn(
+                            "Skipping compaction recovery for table {}, user 
{}, identifier {}: output {} is missing",
+                            tableName,
+                            commitUser,
+                            committable.identifier(),
+                            physicalPath);
+                    return null;
+                }
+            } catch (IOException e) {
+                throw new UncheckedIOException(e);
+            }
+        }
+        if (!compactInputsStillLive(committable)) {
+            return null;
+        }
+        return new CompactRecoveryCommittable(committable, messages);
+    }
+
+    /**
+     * A concurrent compaction may already have removed some compact-before 
files. Replaying such an
+     * increment would fail conflict detection on every restart, so only 
recover it when all of its
+     * inputs are still live in the latest snapshot.
+     */
+    private boolean compactInputsStillLive(ManifestCommittable committable) {
+        Set<List<Object>> inputs = new HashSet<>();
+        Set<String> inputNames = new HashSet<>();
+        Set<BinaryRow> partitions = new HashSet<>();
+        for (CommitMessage message : committable.fileCommittables()) {
+            for (DataFileMeta file :
+                    ((CommitMessageImpl) 
message).compactIncrement().compactBefore()) {
+                inputs.add(Arrays.asList(message.partition(), 
message.bucket(), file.fileName()));

Review Comment:
   [P1] Compare complete manifest identities when checking live compact inputs
   
   A concurrent compaction can upgrade a file from level 0 to level 1 without 
changing its physical filename (`MergeTreeCompactTask.upgrade` / 
`AbstractCompactRewriter.upgrade`). This tuple still considers the old input 
live, but conflict detection uses `FileEntry.Identifier`, including level and 
auxiliary metadata. Replaying the old level-0 deletion then fails with `File 
deletion conflicts detected`. The latest snapshot still contains the same 
filename, so every restart reconstructs the same invalid recovery instead of 
skipping it.
   
   I reproduced this on `49ef4c4`: publish the APPEND phase of identifier 2, 
commit DELETE(old level 0)/ADD(same file upgraded to level 1) as another user, 
then restore identifier 2. The new regression 
`testFilterHalfCommittedIdentifierWithConcurrentlyUpgradedInput` confirms the 
replay conflicts and fails the expected-empty recovery assertion. Please 
compare the full manifest identity (or explicitly rebase), rather than just 
partition/bucket/name.



##########
paimon-core/src/main/java/org/apache/paimon/operation/FileStoreCommitImpl.java:
##########
@@ -333,6 +348,120 @@ public List<ManifestCommittable> 
filterCommitted(List<ManifestCommittable> commi
         }
     }
 
+    @Nullable
+    private ManifestCommittable recoverCompact(ManifestCommittable 
committable) {
+        List<CommitMessage> messages = new ArrayList<>();
+        Set<Path> outputs = new HashSet<>();
+        DataFilePathFactories dataFactories = new 
DataFilePathFactories(pathFactory);
+        IndexFilePathFactories indexFactories = new 
IndexFilePathFactories(pathFactory);
+        for (CommitMessage message : committable.fileCommittables()) {
+            CompactIncrement compact = ((CommitMessageImpl) 
message).compactIncrement();
+            if (compact.isEmpty()) {
+                continue;
+            }
+            messages.add(
+                    new CommitMessageImpl(
+                            message.partition(),
+                            message.bucket(),
+                            message.totalBuckets(),
+                            DataIncrement.emptyIncrement(),
+                            compact,
+                            message.checkFromSnapshot()));
+            DataFilePathFactory dataFactory =
+                    dataFactories.get(message.partition(), message.bucket());
+            compact.compactAfter().forEach(file -> 
outputs.addAll(file.collectFiles(dataFactory)));
+            compact.changelogFiles()
+                    .forEach(file -> 
outputs.addAll(file.collectFiles(dataFactory)));
+            IndexPathFactory indexFactory =
+                    indexFactories.get(message.partition(), message.bucket());
+            compact.newIndexFiles().forEach(file -> 
outputs.add(indexFactory.toPath(file)));
+        }
+        if (messages.isEmpty()) {
+            return null;
+        }
+        // An expired or orphan-cleaned output cannot be replayed. Skip the 
whole identifier,
+        // rather than publishing only some of its buckets. I/O failures must 
still fail recovery.
+        for (Path output : outputs) {
+            Path physicalPath =
+                    
CompactedChangelogPathResolver.resolveCompactedChangelogPath(output);
+            try {
+                if (!fileIO.exists(physicalPath)) {
+                    LOG.warn(
+                            "Skipping compaction recovery for table {}, user 
{}, identifier {}: output {} is missing",
+                            tableName,
+                            commitUser,
+                            committable.identifier(),
+                            physicalPath);
+                    return null;
+                }
+            } catch (IOException e) {
+                throw new UncheckedIOException(e);
+            }
+        }
+        if (!compactInputsStillLive(committable)) {
+            return null;
+        }
+        return new CompactRecoveryCommittable(committable, messages);
+    }
+
+    /**
+     * A concurrent compaction may already have removed some compact-before 
files. Replaying such an
+     * increment would fail conflict detection on every restart, so only 
recover it when all of its
+     * inputs are still live in the latest snapshot.
+     */
+    private boolean compactInputsStillLive(ManifestCommittable committable) {
+        Set<List<Object>> inputs = new HashSet<>();
+        Set<String> inputNames = new HashSet<>();
+        Set<BinaryRow> partitions = new HashSet<>();
+        for (CommitMessage message : committable.fileCommittables()) {
+            for (DataFileMeta file :
+                    ((CommitMessageImpl) 
message).compactIncrement().compactBefore()) {
+                inputs.add(Arrays.asList(message.partition(), 
message.bucket(), file.fileName()));
+                inputNames.add(file.fileName());
+                partitions.add(message.partition());
+            }
+        }
+        if (inputs.isEmpty()) {

Review Comment:
   [P1] Validate pending index deletions before accepting recovery
   
   Compact increments can also contain `deletedIndexFiles`, including an 
index-only message produced by `DataEvolutionCompactGlobalIndexDropper`. If a 
concurrent `DROP GLOBAL INDEX` removes that index after the APPEND phase, this 
check still succeeds: the index-only message has no `compactBefore`, and index 
liveness is never checked. `GlobalIndexCombiner` then rejects the stale 
deletion, while each subsequent restart reconstructs it again.
   
   The added `testRecoveryAfterConcurrentGlobalIndexDrop` reproduces two fresh 
writer recoveries on this head; both explicitly assert the root cause 
`IllegalStateException: Trying to delete global index file recovery-index which 
does not exist.` Spark/Flink DROP GLOBAL INDEX perform the same metadata-only 
removal. Please validate the compact increment's deleted index identities 
against the current index manifest as well, and discard/rebase an invalid 
replay.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to