smengcl commented on code in PR #10821:
URL: https://github.com/apache/ozone/pull/10821#discussion_r3756201090
##########
hadoop-hdds/rocksdb-checkpoint-differ/src/main/java/org/apache/ozone/rocksdiff/RocksDBCheckpointDiffer.java:
##########
@@ -1161,6 +1162,46 @@ private synchronized void
removeKeyFromCompactionLogTable(
}
}
+ /**
+ * Removes SST files from the backup directory that were hard-linked during
+ * {@code onCompactionBegin} but never recorded in the compaction log because
+ * the compaction did not complete (for example, due to an OM crash/restart).
+ */
+ @VisibleForTesting
+ void cleanupOrphanedSstBackupFiles() {
+ Path sstBackupDirPath = Paths.get(sstBackupDir);
+ if (!Files.isDirectory(sstBackupDirPath)) {
+ return;
+ }
+
+ Set<String> referencedFiles = compactionDag.getCompactionMap().keySet();
Review Comment:
Could we identify referenced backup files from the inputs of completed
compactions instead of every node in the DAG? Consider a completed `A → B`
compaction followed by a `B → C` compaction that hard-links `B.sst` but
terminates before recording its log. On restart, `B` is already in
`compactionMap` as the output of `A → B`, so this check retains the orphaned
`B.sst` indefinitely. A regression test covering this sequence leaves `B.sst`
present when it should be removed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
