bhat-vinay commented on code in PR #10625:
URL: https://github.com/apache/hudi/pull/10625#discussion_r1505573458
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -546,6 +562,39 @@ private HoodieFunctionalIndexDefinition
getFunctionalIndexDefinition(String inde
}
}
+ private Pair<Integer, HoodieData<HoodieRecord>>
initializeSecondaryIndexPartition() throws IOException {
+ HoodieMetadataFileSystemView fsView = new
HoodieMetadataFileSystemView(dataMetaClient,
dataMetaClient.getActiveTimeline(), metadata);
+ // Collect the list of latest file slices present in each partition
+ List<String> partitions = metadata.getAllPartitionPaths();
+ fsView.loadAllPartitions();
+
+ List<Pair<String, FileSlice>> partitionFileSlicePairs = new ArrayList<>();
+ partitions.forEach(partition -> {
Review Comment:
This is actually borrowed from how the existing indexes/partitions are built
in metadata table (like RLI partion in `initializeRecordIndexPartition()`)
##########
hudi-common/src/main/java/org/apache/hudi/common/table/log/HoodieMergedLogRecordScanner.java:
##########
@@ -82,6 +88,9 @@ public class HoodieMergedLogRecordScanner extends
AbstractHoodieLogRecordReader
public final HoodieTimer timer = HoodieTimer.create();
// Map of compacted/merged records
private final ExternalSpillableMap<String, HoodieRecord> records;
+
+ private final ExternalSpillableMap<String, HashMap<String, HoodieRecord>>
nonUniqueKeyRecords;
Review Comment:
Guava and Chronicle supports persistent multi-map structures (just from the
documentation of the library)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
