codope commented on code in PR #10860:
URL: https://github.com/apache/hudi/pull/10860#discussion_r1535123166
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -534,10 +537,17 @@ private Pair<Integer, HoodieData<HoodieRecord>>
initializeFunctionalIndexPartiti
int fileGroupCount =
dataWriteConfig.getMetadataConfig().getFunctionalIndexFileGroupCount();
int parallelism = Math.min(partitionFileSlicePairs.size(),
dataWriteConfig.getMetadataConfig().getFunctionalIndexParallelism());
- Schema readerSchema =
addMetadataFields(getProjectedSchemaForFunctionalIndex(indexDefinition,
dataMetaClient), dataWriteConfig.allowOperationMetadataField());
+ Schema readerSchema =
getProjectedSchemaForFunctionalIndex(indexDefinition, dataMetaClient);
return Pair.of(fileGroupCount,
getFunctionalIndexRecords(partitionFileSlicePairs, indexDefinition,
dataMetaClient, parallelism, readerSchema, hadoopConf));
}
+ private Set<String> getFunctionalIndexPartitionsToInit() {
+ Set<String> functionalIndexPartitions =
dataMetaClient.getFunctionalIndexMetadata().get().getIndexDefinitions().keySet();
+ Set<String> completedMetadataPartitions =
dataMetaClient.getTableConfig().getMetadataPartitions();
Review Comment:
this is handled in `getFunctionalIndexMetadata` method itself. If It is
empty and the index definition path is defined then we load the file. The
loading ideally happens just once when instantiating the metaclient. As long as
we're not recreating meta client in metadata writer we should be good.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
