nsivabalan commented on code in PR #12447:
URL: https://github.com/apache/hudi/pull/12447#discussion_r1889493647
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadata.java:
##########
@@ -919,4 +886,42 @@ private Map<String, HoodieRecord<HoodieMetadataPayload>>
fetchBaseFileAllRecords
return
SecondaryIndexKeyUtils.getRecordKeyFromSecondaryIndexKey(record.getRecordKey());
}, record -> record));
}
+
+ @VisibleForTesting
+ public static Map<String, String>
reverseLookupSecondaryKeysInternal(List<String> recordKeys,
+
Map<String, HoodieRecord<HoodieMetadataPayload>> baseFileRecords,
+
HoodieMetadataLogRecordReader logRecordScanner) {
+ Map<String, String> recordKeyMap = new HashMap<>();
+ Set<String> keySet = new TreeSet<>(recordKeys);
+ Set<String> deletedRecordsFromLogs = new HashSet<>();
+ Map<String, HoodieRecord<HoodieMetadataPayload>> logRecordsMap = new
HashMap<>();
+ // Note that: we read the log records from the oldest to the latest!!!
+ // If we change the read order, we need update the following logic
accordingly.
+ logRecordScanner.getRecords().forEach(record -> {
+ String recordKey =
SecondaryIndexKeyUtils.getRecordKeyFromSecondaryIndexKey(record.getRecordKey());
+ HoodieMetadataPayload payload = record.getData();
+ if (!payload.isDeleted()) { // process only valid records.
+ if (keySet.contains(recordKey)) {
+ logRecordsMap.put(recordKey, record);
Review Comment:
do we have test for this scenario ?
i.e.
add a record to logfile1, delete in logfile2 and then add it back in
logfile3.
if the test works, I am good.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
