jonvex commented on code in PR #13654:
URL: https://github.com/apache/hudi/pull/13654#discussion_r2269899145
##########
hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/HiveHoodieReaderContext.java:
##########
@@ -121,6 +128,12 @@ public ClosableIterator<ArrayWritable>
getFileRecordIterator(
private ClosableIterator<ArrayWritable> getFileRecordIterator(StoragePath
filePath, String[] hosts, long start, long length, Schema dataSchema,
Schema
requiredSchema, HoodieStorage storage) throws IOException {
+ // mdt file schema irregular and does not work with this logic. Also, log
file evolution is handled inside the log block
+ boolean isParquetOrOrc =
filePath.getFileExtension().equals(HoodieFileFormat.PARQUET.getFileExtension())
+ ||
filePath.getFileExtension().equals(HoodieFileFormat.ORC.getFileExtension());
+ Schema avroFileSchema = isParquetOrOrc ?
HoodieIOFactory.getIOFactory(storage)
+ .getFileFormatUtils(filePath).readAvroSchema(storage, filePath) :
dataSchema;
Review Comment:
The data schema passed in is usually the tableSchema. But parquet log block
will pass in the schema from the header. And bootstrap the schema is different
than the table schema.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
