Claire McGinty created PARQUET-2450:
---------------------------------------
Summary: ParquetAvroReader throws exception projecting a single
field of a repeated record type
Key: PARQUET-2450
URL: https://issues.apache.org/jira/browse/PARQUET-2450
Project: Parquet
Issue Type: Bug
Reporter: Claire McGinty
Given an Avro schema with a repeated record type, i.e.:
{code:java}
[
{
"name": "RecordWithNestedFieldTypes",
"namespace": "org.apache.parquet.avro",
"type": "record",
"fields" : [
{
"name" : "nested_record_array",
"type": {
"type": "array",
"items": {
"name": "NestedRecord",
"namespace": "org.apache.parquet.avro",
"type": "record",
"fields": [
{
"name": "int_field",
"type": "int"
},
{
"name": "string_field",
"type": ["null", "string"]
}
]
}
}
}
]
}
] {code}
ParquetAvroReader will fail if you try to project a single field of the nested
array type, with:
{code:java}
java.lang.ClassCastException: optional binary string_field (STRING) is not a
group
at org.apache.parquet.schema.Type.asGroupType(Type.java:247)
at
org.apache.parquet.avro.AvroRecordConverter.newConverter(AvroRecordConverter.java:359)
at
org.apache.parquet.avro.AvroRecordConverter.newConverter(AvroRecordConverter.java:303)
at
org.apache.parquet.avro.AvroRecordConverter.access$100(AvroRecordConverter.java:76)
at
org.apache.parquet.avro.AvroRecordConverter$AvroCollectionConverter$ElementConverter.<init>(AvroRecordConverter.java:613)
at
org.apache.parquet.avro.AvroRecordConverter$AvroCollectionConverter.<init>(AvroRecordConverter.java:562)
at
org.apache.parquet.avro.AvroRecordConverter.newConverter(AvroRecordConverter.java:367)
at
org.apache.parquet.avro.AvroRecordConverter.<init>(AvroRecordConverter.java:143)
{code}
Running a debugger, it looks like this is happening because the #isElementType
check tries to guess if the array element type is a record or not based on
checking if fieldCount > 1:
[https://github.com/apache/parquet-mr/blob/945836c79b5bd3003512ace9e2d30d4cd03422f3/parquet-avro/src/main/java/org/apache/parquet/avro/AvroRecordConverter.java#L932]
. So, if you project a record with fieldCount == 1, it tries to collapse it
into its single field type.
Repro: you can run the following test in `TestSpecificReadWrite`:
{code:java}
@Test
public void testNestedProjectionSingleField() throws IOException {
Path path = writeCarsToParquetFile(1, CompressionCodecName.UNCOMPRESSED,
false);
Configuration conf = new Configuration(testConf);
Schema schema = Car.getClassSchema();
// Project a single field from nested schema
List<Schema.Field> projectedFields = new ArrayList<Schema.Field>();
projectedFields.add(new Schema.Field(
"serviceHistory",
Schema.createUnion(
Schema.create(Schema.Type.NULL),
Schema.createArray(
SchemaBuilder.builder(schema.getNamespace())
.record("Service")
.fields()
.requiredString("mechanic")
.endRecord()))));
Schema projectedSchema =
Schema.createRecord(schema.getName(), schema.getDoc(),
schema.getNamespace(), schema.isError());
projectedSchema.setFields(projectedFields);
AvroReadSupport.setRequestedProjection(conf, projectedSchema);
try (ParquetReader<Car> reader = new AvroParquetReader<Car>(conf, path)) {
for (Car car = reader.read(); car != null; car = reader.read()) {
assertNotNull(car.getServiceHistory());
}
}
} {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
