codope commented on code in PR #12525:
URL: https://github.com/apache/hudi/pull/12525#discussion_r1901488846
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -1181,16 +1182,13 @@ private HoodieData<HoodieRecord>
getSecondaryIndexUpdates(HoodieCommitMetadata c
long targetPartitionSize = 100 * 1024 * 1024;
int parallelism = (int) Math.max(1, (totalWriteBytesForSecondaryIndex +
targetPartitionSize - 1) / targetPartitionSize);
- return readSecondaryKeysFromBaseFiles(
- engineContext,
- partitionFilePairs,
- parallelism,
- this.getClass().getSimpleName(),
- dataMetaClient,
- getEngineType(),
- indexDefinition)
- .union(deletedRecords)
- .distinctWithKey(HoodieRecord::getKey, parallelism);
+ // Load file system view for only the affected partitions on the driver.
+ // By loading on the driver one time, we avoid loading the same metadata
multiple times on the executors.
+ HoodieMetadataFileSystemView fsView = getMetadataView();
+
fsView.loadPartitions(partitionFilePairs.stream().map(Pair::getKey).collect(Collectors.toList()));
+ HoodieData<HoodieRecord> insertRecords =
+ readSecondaryKeysFromBaseFiles(engineContext, partitionFilePairs,
parallelism, this.getClass().getSimpleName(), dataMetaClient, getEngineType(),
indexDefinition, fsView);
+ return
insertRecords.union(deleteRecords).distinctWithKey(HoodieRecord::getKey,
parallelism);
Review Comment:
right.. that patch has landed. I will rebase.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
