danny0405 commented on code in PR #13699:
URL: https://github.com/apache/hudi/pull/13699#discussion_r2280080203
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/FileGroupReaderBasedMergeHandle.java:
##########
@@ -198,8 +270,12 @@ public void doMerge() {
}
// Writes the record
try {
- writeToFile(record.getKey(), record, writeSchemaWithMetaFields,
- config.getPayloadConfig().getProps(), preserveMetadata);
+ // For Compaction operations, the preserveMetadata flag is always
true as we want to preserve the existing record metadata.
+ // For other updates, we only want to preserve the metadata if the
record is not being modified by this update. If the record already exists in
the base file and is not updated,
+ // the operation will be null. Records that are being updated or
records being added to the file group for the first time will have an operation
set and must generate new metadata.
+ boolean shouldPreserveRecordMetadata = preserveMetadata ||
record.getOperation() == null;
+ Schema recordSchema = shouldPreserveRecordMetadata ?
writeSchemaWithMetaFields : writeSchema;
+ writeToFile(record.getKey(), record, recordSchema,
config.getPayloadConfig().getProps(), shouldPreserveRecordMetadata);
Review Comment:
Currently we collect SI and RLI no matter wheter the record has been written
successfully, this could incur inconsistencies.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
