[
https://issues.apache.org/jira/browse/PARQUET-357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Ivanfi updated PARQUET-357:
----------------------------------
Fix Version/s: 1.10.0
> Parquet-thrift generates wrong schema for Thrift binary fields
> --------------------------------------------------------------
>
> Key: PARQUET-357
> URL: https://issues.apache.org/jira/browse/PARQUET-357
> Project: Parquet
> Issue Type: Bug
> Components: parquet-mr
> Affects Versions: 1.5.0, 1.6.0, 1.7.0, 1.8.0
> Reporter: Cheng Lian
> Assignee: Nandor Kollar
> Fix For: 1.10.0
>
>
> Thrift [doesn't have true {{BINARY}}
> type|https://github.com/apache/thrift/blob/0.9.0/lib/java/src/org/apache/thrift/protocol/TType.java#L26-L39].
> The {{BINARY}} type is actually just an unencoded {{STRING}}. Quoted from
> [Thrift Types|https://thrift.apache.org/docs/types] section of official
> Thrift documentation:
> {quote}
> binary: a sequence of unencoded bytes
> N.B.: This is currently a specialized form of the string type above, added to
> provide better interoperability with Java. The current plan-of-record is to
> elevate this to a base type at some point.
> {quote}
> The consequence is that, Thrift {{BINARY}} and {{STRING}} are both passed to
> {{parquet-thrift}} as {{STRING}}, and are [always encoded as {{BINARY
> (UTF8)}}|https://github.com/apache/parquet-mr/blob/apache-parquet-1.7.0/parquet-thrift/src/main/java/org/apache/parquet/thrift/ThriftSchemaConvertVisitor.java#L249-L252].
> This is really a problem on Thrift side. One possible workaround is to
> inspect binary fields in the actual generated Java classes to see whether the
> type is {{ByteBuffer}}.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
