codope commented on code in PR #12525:
URL: https://github.com/apache/hudi/pull/12525#discussion_r1896428904
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -1181,16 +1182,13 @@ private HoodieData<HoodieRecord>
getSecondaryIndexUpdates(HoodieCommitMetadata c
long targetPartitionSize = 100 * 1024 * 1024;
int parallelism = (int) Math.max(1, (totalWriteBytesForSecondaryIndex +
targetPartitionSize - 1) / targetPartitionSize);
- return readSecondaryKeysFromBaseFiles(
- engineContext,
- partitionFilePairs,
- parallelism,
- this.getClass().getSimpleName(),
- dataMetaClient,
- getEngineType(),
- indexDefinition)
- .union(deletedRecords)
- .distinctWithKey(HoodieRecord::getKey, parallelism);
+ // Load file system view for only the affected partitions on the driver.
+ // By loading on the driver one time, we avoid loading the same metadata
multiple times on the executors.
+ HoodieMetadataFileSystemView fsView = getMetadataView();
+
fsView.loadPartitions(partitionFilePairs.stream().map(Pair::getKey).collect(Collectors.toList()));
+ HoodieData<HoodieRecord> insertRecords =
+ readSecondaryKeysFromBaseFiles(engineContext, partitionFilePairs,
parallelism, this.getClass().getSimpleName(), dataMetaClient, getEngineType(),
indexDefinition, fsView);
Review Comment:
HUDI-8793. IMO, we should take it up with metadata DAG rewrite -
https://github.com/apache/hudi/pull/12269
Even now, I am not very happy with the double work of getting
inserts/updates/deletes, one in the write handle and then again while updating
metadata table. It's basically merging all log records in every commit, which
neutralizes the advantages of MOR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
