mvanhorn commented on code in PR #11102:
URL: https://github.com/apache/paimon/pull/11102#discussion_r4568837038


##########
paimon-core/src/main/java/org/apache/paimon/operation/FileStoreCommitImpl.java:
##########
@@ -333,6 +348,120 @@ public List<ManifestCommittable> 
filterCommitted(List<ManifestCommittable> commi
         }
     }
 
+    @Nullable
+    private ManifestCommittable recoverCompact(ManifestCommittable 
committable) {
+        List<CommitMessage> messages = new ArrayList<>();
+        Set<Path> outputs = new HashSet<>();
+        DataFilePathFactories dataFactories = new 
DataFilePathFactories(pathFactory);
+        IndexFilePathFactories indexFactories = new 
IndexFilePathFactories(pathFactory);
+        for (CommitMessage message : committable.fileCommittables()) {
+            CompactIncrement compact = ((CommitMessageImpl) 
message).compactIncrement();
+            if (compact.isEmpty()) {
+                continue;
+            }
+            messages.add(
+                    new CommitMessageImpl(
+                            message.partition(),
+                            message.bucket(),
+                            message.totalBuckets(),
+                            DataIncrement.emptyIncrement(),
+                            compact,
+                            message.checkFromSnapshot()));
+            DataFilePathFactory dataFactory =
+                    dataFactories.get(message.partition(), message.bucket());
+            compact.compactAfter().forEach(file -> 
outputs.addAll(file.collectFiles(dataFactory)));
+            compact.changelogFiles()
+                    .forEach(file -> 
outputs.addAll(file.collectFiles(dataFactory)));
+            IndexPathFactory indexFactory =
+                    indexFactories.get(message.partition(), message.bucket());
+            compact.newIndexFiles().forEach(file -> 
outputs.add(indexFactory.toPath(file)));
+        }
+        if (messages.isEmpty()) {
+            return null;
+        }
+        // An expired or orphan-cleaned output cannot be replayed. Skip the 
whole identifier,
+        // rather than publishing only some of its buckets. I/O failures must 
still fail recovery.
+        for (Path output : outputs) {
+            Path physicalPath =
+                    
CompactedChangelogPathResolver.resolveCompactedChangelogPath(output);
+            try {
+                if (!fileIO.exists(physicalPath)) {
+                    LOG.warn(
+                            "Skipping compaction recovery for table {}, user 
{}, identifier {}: output {} is missing",
+                            tableName,
+                            commitUser,
+                            committable.identifier(),
+                            physicalPath);
+                    return null;
+                }
+            } catch (IOException e) {
+                throw new UncheckedIOException(e);
+            }
+        }
+        if (!compactInputsStillLive(committable)) {
+            return null;
+        }
+        return new CompactRecoveryCommittable(committable, messages);
+    }
+
+    /**
+     * A concurrent compaction may already have removed some compact-before 
files. Replaying such an
+     * increment would fail conflict detection on every restart, so only 
recover it when all of its
+     * inputs are still live in the latest snapshot.
+     */
+    private boolean compactInputsStillLive(ManifestCommittable committable) {
+        Set<List<Object>> inputs = new HashSet<>();
+        Set<String> inputNames = new HashSet<>();
+        Set<BinaryRow> partitions = new HashSet<>();
+        for (CommitMessage message : committable.fileCommittables()) {
+            for (DataFileMeta file :
+                    ((CommitMessageImpl) 
message).compactIncrement().compactBefore()) {
+                inputs.add(Arrays.asList(message.partition(), 
message.bucket(), file.fileName()));
+                inputNames.add(file.fileName());
+                partitions.add(message.partition());
+            }
+        }
+        if (inputs.isEmpty()) {

Review Comment:
   Good catch. In 6cd7bd9, recovery now also checks every deleted index file in 
the compact increment against the latest index manifest (same partition, bucket 
and file meta) and discards the replay when one is missing or changed, so a 
concurrent DROP GLOBAL INDEX no longer makes each restart fail in the combiner. 
The new FileStoreCommitTest case replays an index-only message after the index 
was dropped by another commit.
   
   <!-- r:b719df74464d.11d1184e9072 -->



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to