Bil Bingham created PARQUET-2039:
------------------------------------
Summary: AvroReadSupport.setRequestedProjection in 1.11.1+ not
backwards compatible with MAPS
Key: PARQUET-2039
URL: https://issues.apache.org/jira/browse/PARQUET-2039
Project: Parquet
Issue Type: Bug
Components: parquet-avro, parquet-mr
Affects Versions: 1.11.1, 1.11.0
Reporter: Bil Bingham
Attachments: PQMapTest.java
using AvroReadSupport.setRequestedProjection in 1.11.1 reading a 1.10.0
generated parquet file sets MAP<STRING,STRING> fields to null (and vice versa
1.10.0 reader against a 1.11.1 generated file)
- Not using a projected schema works, the map fields are converted correctly.
In my case. Parquet file is generated by hive
{code:java}
CREATE TABLE parquetmaptest (
a string,
b MAP<string,string>
)
STORED AS PARQUET
tblproperties(
"parquet.compression"="SNAPPY"
);
insert into parquetmaptest select "a",map("k1","v1","k2","v2");{code}
Using parquet-avro 1.11.1 (and appropriate dependencies) result in field "b"
being null.
{code:java}
data:null
row:{"a": "a", "b": null}{code}
Using parquet-avro 1.11.0 (and appropriate dependencies) result in field "b"
being the right map value.
{code:java}
data:{k1=v1, k2=v2}
row:{"a": "a", "b": {"k1": "v1", "k2": "v2"}}{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
