[GitHub] [parquet-mr] wgtmac commented on a diff in pull request #1140: allow read old parquet file which is maked by old api with old avro version which allow wrong default value in schema

[via GitHub]

wgtmac commented on code in PR #1140:
URL: https://github.com/apache/parquet-mr/pull/1140#discussion_r1330857433


##########
parquet-avro/src/main/java/org/apache/parquet/avro/AvroReadSupport.java:
##########
@@ -129,10 +129,10 @@ public RecordMaterializer<T> prepareForRead(
       avroSchema = new 
Schema.Parser().parse(metadata.get(AVRO_READ_SCHEMA_METADATA_KEY));
     } else if (keyValueMetaData.get(AVRO_SCHEMA_METADATA_KEY) != null) {
       // use the Avro schema from the file metadata if present
-      avroSchema = new 
Schema.Parser().parse(keyValueMetaData.get(AVRO_SCHEMA_METADATA_KEY));
+      avroSchema = new 
Schema.Parser().setValidateDefaults(false).parse(keyValueMetaData.get(AVRO_SCHEMA_METADATA_KEY));

Review Comment:
   Blindly disabling validation at all times may introduce new issues silently. 
Is it safer to add a flag to work around this? In the default case, we should 
always enable validation.



##########
parquet-avro/src/main/java/org/apache/parquet/avro/AvroReadSupport.java:
##########
@@ -129,10 +129,10 @@ public RecordMaterializer<T> prepareForRead(
       avroSchema = new 
Schema.Parser().parse(metadata.get(AVRO_READ_SCHEMA_METADATA_KEY));
     } else if (keyValueMetaData.get(AVRO_SCHEMA_METADATA_KEY) != null) {
       // use the Avro schema from the file metadata if present
-      avroSchema = new 
Schema.Parser().parse(keyValueMetaData.get(AVRO_SCHEMA_METADATA_KEY));
+      avroSchema = new 
Schema.Parser().setValidateDefaults(false).parse(keyValueMetaData.get(AVRO_SCHEMA_METADATA_KEY));

Review Comment:
   Does this issue exist with both `AVRO_SCHEMA_METADATA_KEY` and 
`OLD_AVRO_SCHEMA_METADATA_KEY`?



##########
parquet-avro/src/main/java/org/apache/parquet/avro/AvroReadSupport.java:
##########
@@ -129,10 +129,10 @@ public RecordMaterializer<T> prepareForRead(
       avroSchema = new 
Schema.Parser().parse(metadata.get(AVRO_READ_SCHEMA_METADATA_KEY));
     } else if (keyValueMetaData.get(AVRO_SCHEMA_METADATA_KEY) != null) {
       // use the Avro schema from the file metadata if present
-      avroSchema = new 
Schema.Parser().parse(keyValueMetaData.get(AVRO_SCHEMA_METADATA_KEY));
+      avroSchema = new 
Schema.Parser().setValidateDefaults(false).parse(keyValueMetaData.get(AVRO_SCHEMA_METADATA_KEY));

Review Comment:
   cc Avro experts @Fokko @gszadovszky @clairemcginty 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@parquet.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org