voonhous commented on code in PR #17763:
URL: https://github.com/apache/hudi/pull/17763#discussion_r2656172313
##########
hudi-hadoop-common/src/main/java/org/apache/hudi/io/storage/hadoop/HoodieAvroParquetReader.java:
##########
@@ -181,21 +181,21 @@ private static StorageConfiguration<?>
tryOverrideDefaultConfigs(StorageConfigur
return conf;
}
- private ClosableIterator<IndexedRecord>
getIndexedRecordIteratorInternal(Schema schema, Map<String, String>
renamedColumns) throws IOException {
+ private ClosableIterator<IndexedRecord>
getIndexedRecordIteratorInternal(HoodieSchema schema, Map<String, String>
renamedColumns) throws IOException {
// NOTE: We have to set both Avro read-schema and projection schema to make
// sure that in case the file-schema is not equal to read-schema
we'd still
// be able to read that file (in case projection is a proper one)
Configuration hadoopConf =
storage.getConf().unwrapCopyAs(Configuration.class);
//TODO boundary for now to revisit in later pr to use HoodieSchema
- Schema repairedFileSchema =
AvroSchemaRepair.repairLogicalTypes(getSchema().getAvroSchema(), schema);
+ Schema repairedFileSchema =
AvroSchemaRepair.repairLogicalTypes(getSchema().toAvroSchema(),
schema.toAvroSchema());
Option<Schema> promotedSchema = Option.empty();
- if (!renamedColumns.isEmpty() ||
HoodieAvroUtils.recordNeedsRewriteForExtendedAvroTypePromotion(repairedFileSchema,
schema)) {
+ if (!renamedColumns.isEmpty() ||
HoodieAvroUtils.recordNeedsRewriteForExtendedAvroTypePromotion(repairedFileSchema,
schema.toAvroSchema())) {
Review Comment:
It's not handled in any of the current PRs now. I am still systematically
trying to migrate what is easily migrated first.
Whatever that is left, will then evaluate if we should migrate them over. I
haven't really analysed what this method entails.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
