[
https://issues.apache.org/jira/browse/PARQUET-2039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17335535#comment-17335535
]
Bil Bingham commented on PARQUET-2039:
--------------------------------------
Sorry for the confusion.
The writer is using parquet 1.10.0 (bundled with hive-exec)
The bug I found was using 1.11.0 and 1.11.1 as described above . (The reader at
1.10.0 , 1,11,0 works, at 1.11.1 it breaks)
In the test case I included I simply used the parquet schema of the file
converted with the AvroSchemeConverter.
{code:java}
{"type":"record","name":"hive_schema","fields":[{"name":"a","type":["null","string"],"default":null},{"name":"b","type":["null",{"type":"map","values":["null","string"]}],"default":null}]}{code}
> AvroReadSupport.setRequestedProjection in 1.11.1+ not backwards compatible
> with MAPS
> ------------------------------------------------------------------------------------
>
> Key: PARQUET-2039
> URL: https://issues.apache.org/jira/browse/PARQUET-2039
> Project: Parquet
> Issue Type: Bug
> Components: parquet-avro, parquet-mr
> Affects Versions: 1.11.0, 1.11.1
> Reporter: Bil Bingham
> Priority: Minor
> Attachments: PQMapTest.java
>
>
>
> using AvroReadSupport.setRequestedProjection in 1.11.1 reading a 1.10.0
> generated parquet file sets MAP<STRING,STRING> fields to null (and vice versa
> 1.10.0 reader against a 1.11.1 generated file)
> - Not using a projected schema works, the map fields are converted correctly.
> In my case. Parquet file is generated by hive
> {code:java}
> CREATE TABLE parquetmaptest (
> a string,
> b MAP<string,string>
> )
> STORED AS PARQUET
> tblproperties(
> "parquet.compression"="SNAPPY"
> );
> insert into parquetmaptest select "a",map("k1","v1","k2","v2");{code}
> Using parquet-avro 1.11.1 (and appropriate dependencies) result in field "b"
> being null.
> {code:java}
> data:null
> row:{"a": "a", "b": null}{code}
> Using parquet-avro 1.11.0 (and appropriate dependencies) result in field "b"
> being the right map value.
> {code:java}
> data:{k1=v1, k2=v2}
> row:{"a": "a", "b": {"k1": "v1", "k2": "v2"}}{code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
