Copilot commented on code in PR #9169:
URL: https://github.com/apache/ozone/pull/9169#discussion_r2440792938
##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OMDBCheckpointServletInodeBasedXfer.java:
##########
@@ -486,13 +487,28 @@ private boolean writeDBToArchive(Set<String>
sstFilesToExclude, Path dbDir, Atom
*/
private DBCheckpoint createAndPrepareCheckpoint(Path tmpdir, boolean flush)
throws IOException {
// make tmp directories to contain the copies
- Path tmpCompactionLogDir =
tmpdir.resolve(getCompactionLogDir().getFileName());
Path tmpSstBackupDir = tmpdir.resolve(getSstBackupDir().getFileName());
+ Files.createDirectories(tmpSstBackupDir);
- // Create checkpoint and then copy the files so that it has all the
compaction entries and files.
+ // Create checkpoint.
DBCheckpoint dbCheckpoint = getDbStore().getCheckpoint(flush);
- FileUtils.copyDirectory(getCompactionLogDir().toFile(),
tmpCompactionLogDir.toFile());
- OmSnapshotUtils.linkFiles(getSstBackupDir().toFile(),
tmpSstBackupDir.toFile());
+
+ try {
+ RocksDBCheckpointDiffer differ =
getDbStore().getRocksDBCheckpointDiffer();
+ List<String> sstFiles =
+
differ.getCompactionLogSstFiles(dbCheckpoint.getCheckpointLocation());
+ for (String sstFile : sstFiles) {
+ Path sstFileToLink = getSstBackupDir().resolve(sstFile);
+ if (Files.exists(sstFileToLink)) {
+ Files.createLink(tmpSstBackupDir.resolve(sstFile), sstFileToLink);
+ } else {
+ LOG.warn("SST file {} from compaction log not found in backup " +
+ "directory {}", sstFile, getSstBackupDir());
+ }
+ }
+ } catch (Exception e) {
Review Comment:
Catching generic `Exception` is too broad. Catch specific exceptions like
`RocksDBException` or `IOException` to handle different failure scenarios
appropriately and avoid masking unexpected errors.
##########
hadoop-hdds/rocksdb-checkpoint-differ/src/main/java/org/apache/ozone/rocksdiff/RocksDBCheckpointDiffer.java:
##########
@@ -1426,6 +1431,79 @@ public static void invalidateCacheEntry(String cacheKey)
{
}
}
+ public List<String> getCompactionLogSstFiles(Path checkpointPath)
+ throws IOException {
+ List<String> sstFiles = new ArrayList<>();
+ List<ColumnFamilyDescriptor> cfDescriptors = new ArrayList<>();
+ List<byte[]> cfs;
+ try (Options options = new Options()) {
+ cfs = RocksDB.listColumnFamilies(options, checkpointPath.toString());
+ } catch (RocksDBException e) {
+ throw new IOException(e);
+ }
+
+ if (cfs == null || cfs.isEmpty()) {
+ cfs = Collections.singletonList(RocksDB.DEFAULT_COLUMN_FAMILY);
+ }
+
+ List<ColumnFamilyOptions> cfOptions = new ArrayList<>();
+ try {
+ for (byte[] cf : cfs) {
+ ColumnFamilyOptions opts = new ColumnFamilyOptions();
+ cfOptions.add(opts);
+ cfDescriptors.add(new ColumnFamilyDescriptor(cf, opts));
+ }
+
+ List<ColumnFamilyHandle> cfHandles = new ArrayList<>();
+ try (DBOptions dbOptions = new DBOptions()
+ .setCreateIfMissing(false).setCreateMissingColumnFamilies(true);
Review Comment:
Opening a database in read-only mode with
`setCreateMissingColumnFamilies(true)` is contradictory. Read-only mode should
not create new column families. Change to
`.setCreateMissingColumnFamilies(false)` to align with the read-only intent.
```suggestion
.setCreateIfMissing(false).setCreateMissingColumnFamilies(false);
```
##########
hadoop-hdds/rocksdb-checkpoint-differ/src/main/java/org/apache/ozone/rocksdiff/RocksDBCheckpointDiffer.java:
##########
@@ -1426,6 +1431,79 @@ public static void invalidateCacheEntry(String cacheKey)
{
}
}
+ public List<String> getCompactionLogSstFiles(Path checkpointPath)
+ throws IOException {
+ List<String> sstFiles = new ArrayList<>();
+ List<ColumnFamilyDescriptor> cfDescriptors = new ArrayList<>();
+ List<byte[]> cfs;
+ try (Options options = new Options()) {
+ cfs = RocksDB.listColumnFamilies(options, checkpointPath.toString());
+ } catch (RocksDBException e) {
+ throw new IOException(e);
+ }
+
+ if (cfs == null || cfs.isEmpty()) {
+ cfs = Collections.singletonList(RocksDB.DEFAULT_COLUMN_FAMILY);
+ }
+
+ List<ColumnFamilyOptions> cfOptions = new ArrayList<>();
+ try {
+ for (byte[] cf : cfs) {
+ ColumnFamilyOptions opts = new ColumnFamilyOptions();
+ cfOptions.add(opts);
+ cfDescriptors.add(new ColumnFamilyDescriptor(cf, opts));
+ }
+
+ List<ColumnFamilyHandle> cfHandles = new ArrayList<>();
+ try (DBOptions dbOptions = new DBOptions()
+ .setCreateIfMissing(false).setCreateMissingColumnFamilies(true);
+ RocksDB db = RocksDB.openReadOnly(dbOptions,
+ checkpointPath.toString(), cfDescriptors, cfHandles)) {
+
+ ColumnFamilyHandle compactionLogCf = null;
+ for (int i = 0; i < cfs.size(); i++) {
+ if (Arrays.equals(cfs.get(i),
+ "compactionLogTable".getBytes(UTF_8))) {
+ compactionLogCf = cfHandles.get(i);
+ break;
+ }
+ }
+
+ if (compactionLogCf != null) {
+ try (RocksIterator iterator = db.newIterator(compactionLogCf)) {
+ for (iterator.seekToFirst(); iterator.isValid(); iterator.next()) {
+ try {
+ CompactionLogEntry logEntry =
CompactionLogEntry.getFromProtobuf(
+ CompactionLogEntryProto.parseFrom(iterator.value()));
+ logEntry.getInputFileInfoList().forEach(f ->
+ sstFiles.add(f.getFileName()));
+ logEntry.getOutputFileInfoList().forEach(f ->
+ sstFiles.add(f.getFileName()));
Review Comment:
Using `List.add()` in a loop can be inefficient if the list grows large.
Consider using a `Set` for `sstFiles` to automatically handle duplicates and
improve lookup performance, or use `addAll()` if the file info lists can be
converted to filename collections.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
