klsince commented on a change in pull request #7969:
URL: https://github.com/apache/pinot/pull/7969#discussion_r782516626
##########
File path:
pinot-core/src/main/java/org/apache/pinot/core/data/manager/BaseTableDataManager.java
##########
@@ -285,49 +290,36 @@ public void addSegmentError(String segmentName,
SegmentErrorInfo segmentErrorInf
public void reloadSegment(String segmentName, IndexLoadingConfig
indexLoadingConfig, SegmentZKMetadata zkMetadata,
SegmentMetadata localMetadata, @Nullable Schema schema, boolean
forceDownload)
throws Exception {
- File indexDir = localMetadata.getIndexDir();
- Preconditions.checkState(indexDir.isDirectory(), "Index directory: %s is
not a directory", indexDir);
-
- File parentFile = indexDir.getParentFile();
- File segmentBackupDir =
- new File(parentFile, indexDir.getName() +
CommonConstants.Segment.SEGMENT_BACKUP_DIR_SUFFIX);
-
+ File indexDir = getSegmentDataDir(segmentName);
try {
- // First rename index directory to segment backup directory so that
original segment have all file descriptors
- // point to the segment backup directory to ensure original segment
serves queries properly
+ // Create backup directory to handle failure of segment reloading.
+ createBackup(indexDir);
- // Rename index directory to segment backup directory (atomic)
- Preconditions.checkState(indexDir.renameTo(segmentBackupDir),
- "Failed to rename index directory: %s to segment backup directory:
%s", indexDir, segmentBackupDir);
-
- // Download from remote or copy from local backup directory into index
directory,
- // and then continue to load the segment from index directory.
+ // Download segment from deep store if CRC changes or forced to download;
+ // otherwise, copy backup directory back to the original index directory.
+ // And then continue to load the segment from the index directory.
boolean shouldDownload = forceDownload || !hasSameCRC(zkMetadata,
localMetadata);
if (shouldDownload && allowDownload(segmentName, zkMetadata)) {
if (forceDownload) {
LOGGER.info("Segment: {} of table: {} is forced to download",
segmentName, _tableNameWithType);
} else {
- LOGGER.info("Download segment:{} of table: {} as local crc: {}
mismatches remote crc: {}", segmentName,
+ LOGGER.info("Download segment:{} of table: {} as crc changes from:
{} to: {}", segmentName,
_tableNameWithType, localMetadata.getCrc(), zkMetadata.getCrc());
}
indexDir = downloadSegment(segmentName, zkMetadata);
} else {
- LOGGER.info("Reload the local copy of segment: {} of table: {}",
segmentName, _tableNameWithType);
- FileUtils.copyDirectory(segmentBackupDir, indexDir);
+ LOGGER.info("Reload existing segment: {} of table: {}", segmentName,
_tableNameWithType);
+ try (SegmentDirectory segmentDirectory =
initSegmentDirectory(segmentName, indexLoadingConfig)) {
+ segmentDirectory.copyTo(indexDir);
+ }
}
// Load from index directory and replace the old segment in memory.
- addSegment(ImmutableSegmentLoader.load(indexDir, indexLoadingConfig,
schema));
-
- // Rename segment backup directory to segment temporary directory
(atomic)
- // The reason to first rename then delete is that, renaming is an atomic
operation, but deleting is not. When we
- // rename the segment backup directory to segment temporary directory,
we know the reload already succeeded, so
- // that we can safely delete the segment temporary directory
- File segmentTempDir = new File(parentFile, indexDir.getName() +
CommonConstants.Segment.SEGMENT_TEMP_DIR_SUFFIX);
- Preconditions.checkState(segmentBackupDir.renameTo(segmentTempDir),
- "Failed to rename segment backup directory: %s to segment temporary
directory: %s", segmentBackupDir,
- segmentTempDir);
- FileUtils.deleteDirectory(segmentTempDir);
+ ImmutableSegment segment = ImmutableSegmentLoader.load(indexDir,
indexLoadingConfig, schema);
Review comment:
hmm.. like what kind of issues? As to what happens at these two places:
The 1st time to initialize SegmentDirectory, the indexDir is empty, so
essentially it's an empty SegmentLocalFSDirectory object to be created. But
it's able to make a copy at indexDir with knowledge about where the source is.
Other SegmentDirectory impls may do copyTo with their source location, like
remote fs.
The 2nd time to create SegmentDirectory via
ImmutableSegmentLoader.load(indexDir) here, the indexDir is non empty now. It's
preprocessed, and then used to initialize the SegmentDirectory object to create
the ImmutableSegment object in the end.
Would love to understand your concerns, so feel free to comment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
