[
https://issues.apache.org/jira/browse/PARQUET-281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14576697#comment-14576697
]
Dong Chen commented on PARQUET-281:
-----------------------------------
Sounds good. Thanks [~rdblue].
Last question is shall we need to consider the possibility that user code and
Parquet have different conversion for logical type and 'Type', such as Decimal
<-> Binary?
Maybe it will not happen, then Parquet can build the custom comparator inside
for any types based on their logical type.
> Statistic and Filter need a mechanism to get customized comparator from high
> layer user
> ---------------------------------------------------------------------------------------
>
> Key: PARQUET-281
> URL: https://issues.apache.org/jira/browse/PARQUET-281
> Project: Parquet
> Issue Type: Improvement
> Reporter: Dong Chen
> Assignee: Dong Chen
>
> As discussed in HIVE-10254, we might need a customized comparator from high
> layer user for generating statistic when writing and applying filter when
> reading.
> The problem is that (use Decimal type in Hive as an example):
> Decimal in Hive is mapped to Binary in Parquet. When using predicate and
> statistic to filter values, comparing Binary values in Parquet cannot reflect
> the correct relationship of Decimal values in Hive. This type mapping causes
> 2 problems:
> 1. When writing Decimal column, Binary.compareTo() is used to judge and set
> the column statistic (min, max). The generated statistic value is not correct
> from a Decimal perspective.
> 2. When reading with Predicate (also Filter), in which the expected Decimal
> value is converted to Binary type, Binary.compareTo() is used to compare the
> expected value and column statistic value. They are Binary perspective, and
> also the result is not right.
> We could add an interface for customized comparator, and high level user like
> Hive provides the comparator to Parquet, since Hive knows how to decode the
> binary to Decimal and compare. Then Parquet could switch between customized
> and original comparison method.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
