mcvsubbu commented on a change in pull request #7969:
URL: https://github.com/apache/pinot/pull/7969#discussion_r784280438
##########
File path:
pinot-core/src/main/java/org/apache/pinot/core/data/manager/BaseTableDataManager.java
##########
@@ -371,9 +360,9 @@ public void addOrReplaceSegment(String segmentName,
IndexLoadingConfig indexLoad
// Download segment and replace the local one, either due to failure to
recover local segment,
// or the segment data is updated and has new CRC now.
if (localMetadata == null) {
- LOGGER.info("Download segment: {} of table: {} as no good one exists
locally", segmentName, _tableNameWithType);
+ LOGGER.info("Download segment: {} of table: {} as it doesn't exist",
segmentName, _tableNameWithType);
} else {
- LOGGER.info("Download segment: {} of table: {} as local crc: {}
mismatches remote crc: {}.", segmentName,
+ LOGGER.info("Download segment: {} of table: {} as crc changes from: {}
to: {}", segmentName,
Review comment:
will this log message happen twice when crc changes? is there a way to
avoid that? It is already logged in tryLoadExisting() ... method
##########
File path:
pinot-core/src/main/java/org/apache/pinot/core/data/manager/BaseTableDataManager.java
##########
@@ -493,12 +445,139 @@ protected File getTmpSegmentDataDir(String segmentName) {
return new File(_resourceTmpDir, segmentName);
}
- @VisibleForTesting
- static boolean isNewSegment(SegmentZKMetadata zkMetadata, @Nullable
SegmentMetadata localMetadata) {
- return localMetadata == null || !hasSameCRC(zkMetadata, localMetadata);
+ /**
+ * Create a backup directory to handle failure of segment reloading.
+ * First rename index directory to segment backup directory so that original
segment have all file
+ * descriptors point to the segment backup directory to ensure original
segment serves queries properly.
+ * The original index directory is restored lazily, as depending on the
conditions,
+ * it may be restored from the backup directory or segment downloaded from
deep store.
+ */
+ private void createBackup(File indexDir) {
+ if (!indexDir.exists()) {
+ return;
+ }
+ File parentDir = indexDir.getParentFile();
+ File segmentBackupDir = new File(parentDir, indexDir.getName() +
CommonConstants.Segment.SEGMENT_BACKUP_DIR_SUFFIX);
+ // Rename index directory to segment backup directory (atomic).
+ Preconditions.checkState(indexDir.renameTo(segmentBackupDir),
+ "Failed to rename index directory: %s to segment backup directory:
%s", indexDir, segmentBackupDir);
+ }
+
+ /**
+ * Remove the backup directory to mark the completion of segment reloading.
+ * First rename then delete is as renaming is an atomic operation, but
deleting is not.
+ * When we rename the segment backup directory to segment temporary
directory, we know the reload
+ * already succeeded, so that we can safely delete the segment temporary
directory.
+ */
+ private void removeBackup(File indexDir)
+ throws IOException {
+ File parentDir = indexDir.getParentFile();
+ File segmentBackupDir = new File(parentDir, indexDir.getName() +
CommonConstants.Segment.SEGMENT_BACKUP_DIR_SUFFIX);
+ if (!segmentBackupDir.exists()) {
+ return;
+ }
+ // Rename segment backup directory to segment temporary directory (atomic).
+ File segmentTempDir = new File(parentDir, indexDir.getName() +
CommonConstants.Segment.SEGMENT_TEMP_DIR_SUFFIX);
+ Preconditions.checkState(segmentBackupDir.renameTo(segmentTempDir),
+ "Failed to rename segment backup directory: %s to segment temporary
directory: %s", segmentBackupDir,
+ segmentTempDir);
+ FileUtils.deleteDirectory(segmentTempDir);
+ }
+
+ private boolean tryLoadExistingSegment(String segmentName,
IndexLoadingConfig indexLoadingConfig,
Review comment:
Can u add a comment here stating the conditions when it returns true or
false. As I understand, true means it existed locally and has been loaded into
memory. false means it may exist locally with a different CRC or it may not
exist at all. Is the SegmentDirectory always closed upon return?
##########
File path:
pinot-server/src/main/java/org/apache/pinot/server/starter/helix/HelixInstanceDataManager.java
##########
@@ -307,6 +306,30 @@ private void reloadSegment(String tableNameWithType,
SegmentMetadata segmentMeta
}
}
+ /**
+ * Try to reload a mutable segment.
+ * @return true if the segment is mutable; false if the segment is immutable.
Review comment:
```suggestion
* @return true if the segment is mutable and loaded; false if the segment
is immutable.
```
##########
File path:
pinot-server/src/main/java/org/apache/pinot/server/starter/helix/HelixInstanceDataManager.java
##########
@@ -307,6 +306,30 @@ private void reloadSegment(String tableNameWithType,
SegmentMetadata segmentMeta
}
}
+ /**
+ * Try to reload a mutable segment.
+ * @return true if the segment is mutable; false if the segment is immutable.
+ */
+ @VisibleForTesting
+ boolean reloadMutableSegment(String tableNameWithType, String segmentName,
Review comment:
Maybe rename to reloadIfMutabelSegment() ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
