[
https://issues.apache.org/jira/browse/PARQUET-651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Cheng Lian updated PARQUET-651:
-------------------------------
Description:
Found this issue while investigating SPARK-16344.
For the following Parquet schema
{noformat}
message root {
optional group f (LIST) {
repeated group list {
optional group element {
optional int64 element;
}
}
}
}
{noformat}
parquet-avro decodes it as something like this:
{noformat}
record SingleElement {
int element;
}
record NestedSingleElement {
SingleElement element;
}
record Spark16344Wrong {
array<NestedSingleElement> f;
}
{noformat}
while correct interpretation should be:
{noformat}
record SingleElement {
int element;
}
record Spark16344 {
array<SingleElement> f;
}
{noformat}
The reason is that the {{element}} syntactic group for LIST in
{noformat}
<list-repetition> group <name> (LIST) {
repeated group list {
<element-repetition> <element-type> element;
}
}
{noformat}
is recognized as a record field named {{element}}. The problematic code lies in
[{{AvroRecordConverter.isElementType()}}|https://github.com/apache/parquet-mr/blob/bd0b5af025fab9cad8f94260138741c252f45fc8/parquet-avro/src/main/java/org/apache/parquet/avro/AvroRecordConverter.java#L858].
We should probably check the standard 3-level layout first before falling back
to the legacy 2-level layout.
was:
Found this issue while investigating SPARK-16344.
For the following Parquet schema
{noformat}
message root {
optional group f (LIST) {
repeated group list {
optional group element {
optional int64 element;
}
}
}
}
{noformat}
parquet-avro decodes it as something like this:
{noformat}
record SingleElement {
int element;
}
record NestedSingleElement {
SingleElement element;
}
record Spark16344Wrong {
array<NestedSingleElement> f;
}
{noformat}
while correct interpretation should be:
{noformat}
record SingleElement {
int element;
}
record Spark16344 {
array<SingleElement> f;
}
{noformat}
The reason is that the {{element}} syntactic group for LIST in
{noformat}
<list-repetition> group <name> (LIST) {
repeated group list {
<element-repetition> <element-type> element;
}
}
{noformat}
is recognized as record field {{SingleElement.element}}. The problematic code
lies in
[{{AvroRecordConverter.isElementType()}}|https://github.com/apache/parquet-mr/blob/bd0b5af025fab9cad8f94260138741c252f45fc8/parquet-avro/src/main/java/org/apache/parquet/avro/AvroRecordConverter.java#L858].
We should probably check the standard 3-level layout first before falling back
to the legacy 2-level layout.
> Parquet-avro fails to decode array of record with a single field name
> "element" correctly
> -----------------------------------------------------------------------------------------
>
> Key: PARQUET-651
> URL: https://issues.apache.org/jira/browse/PARQUET-651
> Project: Parquet
> Issue Type: Bug
> Components: parquet-avro
> Affects Versions: 1.7.0, 1.8.0, 1.8.1
> Reporter: Cheng Lian
>
> Found this issue while investigating SPARK-16344.
> For the following Parquet schema
> {noformat}
> message root {
> optional group f (LIST) {
> repeated group list {
> optional group element {
> optional int64 element;
> }
> }
> }
> }
> {noformat}
> parquet-avro decodes it as something like this:
> {noformat}
> record SingleElement {
> int element;
> }
> record NestedSingleElement {
> SingleElement element;
> }
> record Spark16344Wrong {
> array<NestedSingleElement> f;
> }
> {noformat}
> while correct interpretation should be:
> {noformat}
> record SingleElement {
> int element;
> }
> record Spark16344 {
> array<SingleElement> f;
> }
> {noformat}
> The reason is that the {{element}} syntactic group for LIST in
> {noformat}
> <list-repetition> group <name> (LIST) {
> repeated group list {
> <element-repetition> <element-type> element;
> }
> }
> {noformat}
> is recognized as a record field named {{element}}. The problematic code lies
> in
> [{{AvroRecordConverter.isElementType()}}|https://github.com/apache/parquet-mr/blob/bd0b5af025fab9cad8f94260138741c252f45fc8/parquet-avro/src/main/java/org/apache/parquet/avro/AvroRecordConverter.java#L858].
> We should probably check the standard 3-level layout first before falling
> back to the legacy 2-level layout.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
