manojpec commented on a change in pull request #4352:
URL: https://github.com/apache/hudi/pull/4352#discussion_r796975544
##########
File path:
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadata.java
##########
@@ -233,38 +249,69 @@ private void initIfNeeded() {
}
/**
- * Returns a new pair of readers to the base and log files.
+ * Get the file slice details for the given key in a partition.
+ *
+ * @param partitionName - Metadata partition name
+ * @param key - Key to get the file slice for
+ * @return Partition and file slice pair for the given key
*/
- private Pair<HoodieFileReader, HoodieMetadataMergedLogRecordReader>
openReadersIfNeeded(String key, String partitionName) {
- return partitionReaders.computeIfAbsent(partitionName, k -> {
- try {
- final long baseFileOpenMs;
- final long logScannerOpenMs;
- HoodieFileReader baseFileReader = null;
- HoodieMetadataMergedLogRecordReader logRecordScanner = null;
+ private Pair<String, FileSlice> getPartitionFileSlice(final String
partitionName, final String key) {
+ // Metadata is in sync till the latest completed instant on the dataset
+ List<FileSlice> latestFileSlices =
+
HoodieTableMetadataUtil.getPartitionLatestMergedFileSlices(metadataMetaClient,
partitionName);
+
+ final FileSlice slice =
latestFileSlices.get(HoodieTableMetadataUtil.mapRecordKeyToFileGroupIndex(key,
+ latestFileSlices.size()));
+ return Pair.of(partitionName, slice);
+ }
+
+ /**
+ * Get the latest file slices for the interested keys in a given partition.
+ *
+ * @param partitionName - Partition to get the file slices from
+ * @param keys - Interested keys
+ * @return FileSlices for the keys
+ */
+ private Map<Pair<String, FileSlice>, List<String>>
getPartitionFileSlices(final String partitionName, final List<String> keys) {
+ // Metadata is in sync till the latest completed instant on the dataset
+ List<FileSlice> latestFileSlices =
Review comment:
yes, we have more perf work to do and it starts at the
HoodieBackedTableMetadata::getRecordsByKeys() level.
HoodieBackedTableMetadata::getRecordsByKeys() ->
getPartitionFileSliceToKeysMapping() -> getPartitionLatestMergedFileSlices().
But we need to address HUDI-3301 before this. Added this comment to 3301.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
