[
https://issues.apache.org/jira/browse/PARQUET-281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16835065#comment-16835065
]
Ruslan Dautkhanov commented on PARQUET-281:
-------------------------------------------
Does this affect also `timestamp` and `string` logical data types?
Those are also stored as `binary` / byte array in parquet.
> Statistic and Filter need a mechanism to get customized comparator from high
> layer user
> ---------------------------------------------------------------------------------------
>
> Key: PARQUET-281
> URL: https://issues.apache.org/jira/browse/PARQUET-281
> Project: Parquet
> Issue Type: Improvement
> Reporter: Dong Chen
> Assignee: Dong Chen
> Priority: Major
>
> As discussed in HIVE-10254, we might need a customized comparator from high
> layer user for generating statistic when writing and applying filter when
> reading.
> The problem is that (use Decimal type in Hive as an example):
> Decimal in Hive is mapped to Binary in Parquet. When using predicate and
> statistic to filter values, comparing Binary values in Parquet cannot reflect
> the correct relationship of Decimal values in Hive. This type mapping causes
> 2 problems:
> 1. When writing Decimal column, Binary.compareTo() is used to judge and set
> the column statistic (min, max). The generated statistic value is not correct
> from a Decimal perspective.
> 2. When reading with Predicate (also Filter), in which the expected Decimal
> value is converted to Binary type, Binary.compareTo() is used to compare the
> expected value and column statistic value. They are Binary perspective, and
> also the result is not right.
> We could add an interface for customized comparator, and high level user like
> Hive provides the comparator to Parquet, since Hive knows how to decode the
> binary to Decimal and compare. Then Parquet could switch between customized
> and original comparison method.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
