klsince commented on a change in pull request #7969:
URL: https://github.com/apache/pinot/pull/7969#discussion_r778478593
##########
File path:
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/LoaderUtils.java
##########
@@ -161,4 +160,36 @@ public static void reloadFailureRecovery(File indexDir)
FileUtils.forceDelete(segmentTempDir);
}
}
+
+ public static void createBackup(File indexDir)
+ throws IOException {
+ if (!indexDir.exists()) {
+ return;
+ }
+ File parentDir = indexDir.getParentFile();
+ File segmentBackupDir = new File(parentDir, indexDir.getName() +
CommonConstants.Segment.SEGMENT_BACKUP_DIR_SUFFIX);
+ // Rename index directory to segment backup directory (atomic)
+ Preconditions.checkState(indexDir.renameTo(segmentBackupDir),
+ "Failed to rename index directory: %s to segment backup directory:
%s", indexDir, segmentBackupDir);
+ // Copy the backup dir back to proceed.
+ FileUtils.copyDirectory(segmentBackupDir, indexDir);
Review comment:
It's on purpose to rename and then copy back, so that the subsequent
segment preprocessing happens on the copy dir w/o affecting the ongoing
queries, which if any are accessing the renamed dir which is kept intact even
if segment reloading fails on the copy dir. As context, renameTo is an atomic
operation.
> prior to this PR, we just downloaded the segment
Previously, downloading doesn't always happens. It happens if segment CRC
changes or forceDownload option is set to true, when one requests to reload
segment (so downloading raw segment is relatively rare). In this PR, I made
copy always happens, and if downloading happens, it overwrites the copy dir.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
