codope commented on a change in pull request #4693:
URL: https://github.com/apache/hudi/pull/4693#discussion_r839747121
##########
File path:
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java
##########
@@ -645,12 +681,34 @@ private void initializeFileGroups(HoodieTableMetaClient
dataMetaClient, Metadata
}
}
+ public void dropMetadataPartitions(List<MetadataPartitionType>
metadataPartitions) throws IOException {
+ Set<String> completedIndexes =
getCompletedMetadataPartitions(dataMetaClient.getTableConfig());
+ Set<String> inflightIndexes =
getInflightMetadataPartitions(dataMetaClient.getTableConfig());
+
+ for (MetadataPartitionType partitionType : metadataPartitions) {
+ String partitionPath = partitionType.getPartitionPath();
+ // first update table config
+ if (inflightIndexes.contains(partitionPath)) {
+ inflightIndexes.remove(partitionPath);
+
dataMetaClient.getTableConfig().setValue(HoodieTableConfig.TABLE_METADATA_PARTITIONS_INFLIGHT.key(),
String.join(",", inflightIndexes));
+ } else if (completedIndexes.contains(partitionPath)) {
+ completedIndexes.remove(partitionPath);
+
dataMetaClient.getTableConfig().setValue(HoodieTableConfig.TABLE_METADATA_PARTITIONS.key(),
String.join(",", completedIndexes));
+ }
+ HoodieTableConfig.update(dataMetaClient.getFs(), new
Path(dataMetaClient.getMetaPath()), dataMetaClient.getTableConfig().getProps());
+ LOG.warn("Deleting Metadata Table partitions: " + partitionPath);
+ dataMetaClient.getFs().delete(new
Path(metadataWriteConfig.getBasePath(), partitionPath), true);
Review comment:
yes, this will be replaced by DELETE_PARTITION path. we just got the
[lazy deletion of partitions](#4489) landed.
indeed there are multiple point of failure but unlike schedule/run index
delete is a bit safer in terms of partial failures. we would be in trouble if
partition gets deleted but table cnfig is not updated.. so we update the table
config first.. if table config is updated but partitions is not fully deleted,
users can re-trigger drop.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
