danny0405 commented on a change in pull request #5088:
URL: https://github.com/apache/hudi/pull/5088#discussion_r833877211
##########
File path:
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/HoodieMergeHandle.java
##########
@@ -294,13 +292,11 @@ protected boolean writeRecord(HoodieRecord<T>
hoodieRecord, Option<IndexedRecord
try {
if (indexedRecord.isPresent() && !isDelete) {
// Convert GenericRecord to GenericRecord with hoodie commit metadata
in schema
- IndexedRecord recordWithMetadataInSchema =
rewriteRecord((GenericRecord) indexedRecord.get(), preserveMetadata, oldRecord);
- if (preserveMetadata && useWriterSchema) { // useWriteSchema will be
true only incase of compaction.
- // do not preserve FILENAME_METADATA_FIELD
- recordWithMetadataInSchema.put(FILENAME_METADATA_FIELD_POS,
newFilePath.getName());
- fileWriter.writeAvro(hoodieRecord.getRecordKey(),
recordWithMetadataInSchema);
+ if (preserveMetadata && useWriterSchema) { // useWriteSchema will be
true only in case of compaction.
Review comment:
Currently we read the old records from base file with metadata fields in
schema, see:
https://github.com/apache/hudi/blob/52f0498330cc6e2112d9e5228386436babdb9133/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/commit/HoodieMergeHelper.java#L76
And for new records in the in-coming dataset, the schema also includes the
metadata fields.
The `combineAndGetUpdateValue` method also handle the records with write
schema(schema with metadata fields)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
