CTTY commented on code in PR #11450:
URL: https://github.com/apache/hudi/pull/11450#discussion_r1640637334
##########
hudi-hadoop-common/src/main/java/org/apache/parquet/avro/HoodieAvroReadSupport.java:
##########
@@ -62,6 +58,23 @@ public ReadContext init(Configuration configuration,
Map<String, String> keyValu
return new ReadContext(requestedSchema,
readContext.getReadSupportMetadata());
}
+ /**
+ * Here we want set config with which file has been written.
+ * Even though user may have overwritten {@link
AvroWriteSupport.WRITE_OLD_LIST_STRUCTURE},
+ * it's only applicable to how to produce new files(here is a read path).
+ * Later the config value {@link AvroWriteSupport.WRITE_OLD_LIST_STRUCTURE}
will still be used
+ * to write new file according to the user preferences.
+ **/
+ private void adjustConfToReadWithFileProduceMode(Boolean
isLegacyModeWrittenFile, Configuration configuration) {
+ if (isLegacyModeWrittenFile) {
+ configuration.set(AvroWriteSupport.WRITE_OLD_LIST_STRUCTURE,
+ "true", "support reading avro from legacy map/list in parquet file");
+ } else {
+ configuration.set(AvroWriteSupport.WRITE_OLD_LIST_STRUCTURE,
+ "false", "support reading avro from non-legacy map/list in parquet
file");
+ }
Review Comment:
I personally think this is an OK fix. An alternative may be adding something
like `HoodieAvroSchemaConverter` to rewrite schema within Hudi. But it would be
too much for such a corner case and could become hard to maintain in the future
when Parquet changes their `AvroSchemaConverter`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
