[
https://issues.apache.org/jira/browse/PARQUET-686?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Duffy updated PARQUET-686:
---------------------------------
Description:
BinaryStatistics currently only have a min/max, which are compared as signed
{{byte[]}}. However, for real UTF8-friendly lexicographic comparison, e.g. for
string columns, we would want to calculate the BinaryStatistics based off of a
comparator that treats the bytes as unsigned.
was:FromStringBinary is the Binary-type for UTF-8 strings, but uses the old
comparison method of byte-by-byte instead of being UTF-8 aware like Avro and
core Spark. This should be fixed so that the BinaryStatistics generated for
String columns that contain non-ASCII text will actually be useful.
> Allow for Unsigned Statistics in Binary Type
> --------------------------------------------
>
> Key: PARQUET-686
> URL: https://issues.apache.org/jira/browse/PARQUET-686
> Project: Parquet
> Issue Type: Bug
> Reporter: Andrew Duffy
>
> BinaryStatistics currently only have a min/max, which are compared as signed
> {{byte[]}}. However, for real UTF8-friendly lexicographic comparison, e.g.
> for string columns, we would want to calculate the BinaryStatistics based off
> of a comparator that treats the bytes as unsigned.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
