danny0405 commented on code in PR #13603:
URL: https://github.com/apache/hudi/pull/13603#discussion_r2234467577
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -957,22 +1032,28 @@ private List<DirectoryInfo>
listAllPartitionsFromMDT(String initializationTime,
* Let's say we configure 10 file groups for record level index partition,
and prefix as "record-index-bucket-"
* File groups will be named as :
* record-index-bucket-0000, .... -> ..., record-index-bucket-0009
+ *
+ * @param tryErasePartition During initialization, we want to delete any
existing files in the partition. However, for partitioned rli we call this
method
+ * multiple times. We guard the deletion code with
tryErasePartition and only set it to true the first time this method is called.
+ * Otherwise, each repeated call would erase the
files created by the previous call
*/
private void initializeFileGroups(HoodieTableMetaClient dataMetaClient,
MetadataPartitionType metadataPartition, String instantTime,
- int fileGroupCount, String partitionName)
throws IOException {
- // Remove all existing file groups or leftover files in the partition
- final StoragePath partitionPath = new
StoragePath(metadataWriteConfig.getBasePath(), partitionName);
- HoodieStorage storage = metadataMetaClient.getStorage();
- try {
- final List<StoragePathInfo> existingFiles =
storage.listDirectEntries(partitionPath);
- if (existingFiles.size() > 0) {
- LOG.warn("Deleting all existing files found in MDT partition {}",
partitionName);
- storage.deleteDirectory(partitionPath);
- ValidationUtils.checkState(!storage.exists(partitionPath),
- "Failed to delete MDT partition " + partitionName);
+ int fileGroupCount, String
relativePartitionPath, Option<String> dataPartitionName, boolean
tryErasePartition) throws IOException {
+ if (tryErasePartition) {
Review Comment:
Can we move the cleaning logic out as a separate utility method, so that we
can get rid of this confusing flag and control the cleaning more flexibly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
