codope commented on code in PR #12214:
URL: https://github.com/apache/hudi/pull/12214#discussion_r1831364578
##########
hudi-common/src/main/java/org/apache/hudi/common/table/log/HoodieMetadataMergedLogRecordScanner.java:
##########
@@ -78,24 +78,27 @@ public Map<String, HoodieRecord> getRecords() {
@Override
public <T> void processNextRecord(HoodieRecord<T> newRecord) throws
IOException {
- // Merge the new record with the existing record in the map
- HoodieRecord<T> oldRecord = (HoodieRecord<T>)
records.get(newRecord.getRecordKey());
- if (oldRecord != null) {
- LOG.debug("Merging new record with existing record in the map. Key: {}",
newRecord.getRecordKey());
- recordMerger.fullOuterMerge(oldRecord, readerSchema, newRecord,
readerSchema, this.getPayloadProps()).forEach(
- mergedRecord -> {
- HoodieRecord<T> combinedRecord = mergedRecord.getLeft();
- if (combinedRecord.getData() != oldRecord.getData()) {
- HoodieRecord latestHoodieRecord =
getLatestHoodieRecord(newRecord, combinedRecord, newRecord.getRecordKey());
- records.put(newRecord.getRecordKey(), latestHoodieRecord.copy());
- }
- });
+ String key = newRecord.getRecordKey();
Review Comment:
The logic for processNextRecord is now same as HoodieMergedLogRecordScanner,
so we won't need separate HoodieMetadataMergedLogRecordScanner. However, I
would prefer to do this cleanup in a separate patch.
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieMetadataPayload.java:
##########
@@ -155,7 +156,7 @@ public class HoodieMetadataPayload implements
HoodieRecordPayload<HoodieMetadata
/**
* HoodieMetadata secondary index payload field ids
*/
- public static final String SECONDARY_INDEX_FIELD_RECORD_KEY = "recordKey";
+ public static final String SECONDARY_INDEX_RECORD_KEY_SEPARATOR = "$";
Review Comment:
the key for the secondary index record has to be a string and it is in the
format `secondaryKey$recordKey`. Even though the record key is extracted using
lastIndexOf($), still it won;t work if the record key itself contains the
separator. Is there a better way to encode the key?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
