lokeshj1703 commented on code in PR #18132:
URL: https://github.com/apache/hudi/pull/18132#discussion_r2924820878
##########
hudi-common/src/main/java/org/apache/hudi/common/table/log/AbstractHoodieLogRecordReader.java:
##########
@@ -182,6 +188,8 @@ protected AbstractHoodieLogRecordReader(FileSystem fs,
String basePath, List<Str
this.forceFullScan = forceFullScan;
this.internalSchema = internalSchema == null ?
InternalSchema.getEmptyInternalSchema() : internalSchema;
this.enableOptimizedLogBlocksScan = enableOptimizedLogBlocksScan;
+ this.enableLogicalTimestampFieldRepair =
!hoodieTableMetaClient.isMetadataTable() &&
fs.getConf().getBoolean(HoodieFileReader.ENABLE_LOGICAL_TIMESTAMP_REPAIR,
+ readerSchema != null &&
AvroSchemaUtils.hasTimestampMillisField(readerSchema));
Review Comment:
This is also getting executed in the executor code path. We can optimise by
adding a flag in the caller instead. Validated in the logical repair tests
added in TestHoodieDeltaStreamer.
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/HoodieMergedReadHandle.java:
##########
@@ -59,9 +60,12 @@ public HoodieMergedReadHandle(HoodieWriteConfig config,
HoodieTable<T, I, K, O> hoodieTable,
Pair<String, String> partitionPathFileIDPair) {
super(config, instantTime, hoodieTable, partitionPathFileIDPair);
- readerSchema = HoodieAvroUtils.addMetadataFields(new
Schema.Parser().parse(config.getSchema()),
config.allowOperationMetadataField());
+ Schema orignalReaderSchema = HoodieAvroUtils.addMetadataFields(new
Schema.Parser().parse(config.getSchema()),
config.allowOperationMetadataField());
// config.getSchema is not canonicalized, while config.getWriteSchema is
canonicalized. So, we have to use the canonicalized schema to read the existing
data.
baseFileReaderSchema = HoodieAvroUtils.addMetadataFields(new
Schema.Parser().parse(config.getWriteSchema()),
config.allowOperationMetadataField());
+ // Repair reader schema.
Review Comment:
This would also be executed in executor and we can probably optimise by
adding a flag in the caller. Could not validate though.
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/commit/HoodieMergeHelper.java:
##########
@@ -86,7 +87,8 @@ public void runMerge(HoodieTable<?, ?, ?, ?> table,
HoodieFileReader bootstrapFileReader = null;
Schema writerSchema = mergeHandle.getWriterSchemaWithMetaFields();
- Schema readerSchema = baseFileReader.getSchema();
+ Schema readerSchema =
AvroSchemaUtils.getRepairedSchema(baseFileReader.getSchema(), writerSchema);
Review Comment:
I can see this code getting executed in executor. Ran test:
org.apache.hudi.functional.TestRecordLevelIndex#testRLIUpsert
This might be a problem with branch-0.x as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
