islamismailov commented on PR #957: URL: https://github.com/apache/parquet-mr/pull/957#issuecomment-1137996922
This is a "list" AND a "map" issue, not just list. If you're using Iceberg, good news: just apply this PR to your iceberg branch https://github.com/apache/iceberg/pull/3309 Link to the original issue: https://github.com/apache/iceberg/issues/2962 This worked for us. If you still want to fix it in parquet you might be interested in this change, or something along those lines (not recommended as I didn't fully test this change): commit 1918276ec7f01279cb9906b9378cb8986f6ad3ea Author: Islam Ismailov <[email protected]> Date: Wed May 25 19:03:33 2022 +0000 Attempt a fix on avro-parquet conversion diff --git a/parquet-avro/src/main/java/org/apache/parquet/avro/AvroSchemaConverter.java b/parquet-avro/src/main/java/org/apache/parquet/avro/AvroSchemaConverter.java index 7d1f3cab..960aae22 100644 --- a/parquet-avro/src/main/java/org/apache/parquet/avro/AvroSchemaConverter.java +++ b/parquet-avro/src/main/java/org/apache/parquet/avro/AvroSchemaConverter.java @@ -190,7 +190,7 @@ public class AvroSchemaConverter { } else if (type.equals(Schema.Type.ARRAY)) { if (writeOldListStructure) { return ConversionPatterns.listType(repetition, fieldName, - convertField("array", schema.getElementType(), REPEATED, schemaPath)); + convertField("list", schema.getElementType(), REPEATED, schemaPath)); } else { return ConversionPatterns.listOfElements(repetition, fieldName, convertField(AvroWriteSupport.LIST_ELEMENT_NAME, schema.getElementType(), schemaPath)); diff --git a/parquet-column/src/main/java/org/apache/parquet/schema/ConversionPatterns.java b/parquet-column/src/main/java/org/apache/parquet/schema/ConversionPatterns.java index 8ae66f00..db1b546d 100644 --- a/parquet-column/src/main/java/org/apache/parquet/schema/ConversionPatterns.java +++ b/parquet-column/src/main/java/org/apache/parquet/schema/ConversionPatterns.java @@ -84,8 +84,7 @@ public abstract class ConversionPatterns { LogicalTypeAnnotation.mapType(), new GroupType( Repetition.REPEATED, - mapAlias, - LogicalTypeAnnotation.MapKeyValueTypeAnnotation.getInstance(), + "map", keyType, valueType) ); -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
