Dain Sundstrom created ORC-222:
----------------------------------
Summary: StringStatisticsImpl munges min/max during write
Key: ORC-222
URL: https://issues.apache.org/jira/browse/ORC-222
Project: ORC
Issue Type: Bug
Components: encoding
Reporter: Dain Sundstrom
Priority: Critical
String statistics are collected using Text which compares using raw bytes which
are assumed to be UTF-8. When the input contains invalid UTF-8 sequences and
these are the min or max value, the writer converts these invalid sequences to
a java.lang.String, which replaces invalid UTF-8 sequences with the replacement
character (0xFFFD). This conversion happens here:
https://github.com/apache/orc/blob/master/java/core/src/java/org/apache/orc/impl/ColumnStatisticsImpl.java#L611
To work around this issue, the writer should use the `setMinimumBytes` Protocol
Buffers API instead.
The same issue exists during read, where the bytes are round tripped through
java.lang.String. The read code is here:
https://github.com/apache/orc/blob/master/java/core/src/java/org/apache/orc/impl/ColumnStatisticsImpl.java#L528
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
