cshuo commented on code in PR #18372:
URL: https://github.com/apache/hudi/pull/18372#discussion_r3541894836
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -1116,102 +1110,15 @@ public BatchMetadataConversionFunction(String
instantTime, HoodieCommitMetadata
}
@Override
- public Map<String, HoodieData<HoodieRecord>> convertMetadata() {
- Map<String, HoodieData<HoodieRecord>> partitionToRecordMap =
- HoodieMetadataWriteUtils.convertMetadataToRecords(
- engineContext, dataWriteConfig, commitMetadata, instantTime,
dataMetaClient, getTableMetadata(),
- dataWriteConfig.getMetadataConfig(),
- partitionsToUpdate, dataWriteConfig.getBloomFilterType(),
- dataWriteConfig.getBloomIndexParallelism(),
dataWriteConfig.getWritesFileIdEncoding(), getEngineType(),
- Option.of(dataWriteConfig.getRecordMerger().getRecordType()));
-
- // Updates for record index are created by parsing the WriteStatus which
is a hudi-client object. Hence, we cannot yet move this code
- // to the HoodieTableMetadataUtil class in hudi-common.
- if (partitionsToUpdate.contains(RECORD_INDEX.getPartitionPath())) {
- HoodieData<HoodieRecord> additionalUpdates =
getRecordIndexAdditionalUpserts(partitionToRecordMap.get(RECORD_INDEX.getPartitionPath()),
commitMetadata);
- partitionToRecordMap.put(RECORD_INDEX.getPartitionPath(),
partitionToRecordMap.get(RECORD_INDEX.getPartitionPath()).union(additionalUpdates));
- }
- if (partitionsToUpdate.stream().anyMatch(partition ->
partition.startsWith(EXPRESSION_INDEX.getPartitionPath()))) {
- updateExpressionIndexIfPresent(commitMetadata, instantTime,
partitionToRecordMap);
- }
- if (partitionsToUpdate.stream().anyMatch(partition ->
partition.startsWith(SECONDARY_INDEX.getPartitionPath()))) {
- updateSecondaryIndexIfPresent(commitMetadata, partitionToRecordMap,
instantTime);
- }
- return partitionToRecordMap;
- }
- }
-
- /**
- * Update expression index from {@link HoodieCommitMetadata}.
- */
- private void updateExpressionIndexIfPresent(HoodieCommitMetadata
commitMetadata, String instantTime,
- Map<String,
HoodieData<HoodieRecord>> partitionToRecordMap) {
- if
(!MetadataPartitionType.EXPRESSION_INDEX.isMetadataPartitionAvailable(dataMetaClient))
{
- return;
+ public List<IndexPartitionAndRecords> convertMetadata() {
+ return partitionsToUpdate.stream().flatMap(indexPartition -> {
Review Comment:
Fixed by de-duplicating `partitionsToUpdate` by `MetadataPartitionType`
before invoking `buildUpdate`.
This keeps the update orchestration aligned with the old writer behavior:
SI/EI remain type-level update helpers, so each `SecondaryIndexer` /
`ExpressionIndexer` is invoked only once when any matching concrete metadata
partition is present, while the indexer still returns records for the concrete
partitions of that type.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
