[
https://issues.apache.org/jira/browse/IMPALA-15413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dániel Gábor Vankó updated IMPALA-15413:
----------------------------------------
Labels: parquet utf8 (was: )
> Truncate Parquet page-index min/max values on UTF-8 character boundaries
> ------------------------------------------------------------------------
>
> Key: IMPALA-15413
> URL: https://issues.apache.org/jira/browse/IMPALA-15413
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
> Reporter: Dániel Gábor Vankó
> Priority: Major
> Labels: parquet, utf8
>
> When Impala writes the Parquet page index, it truncates per-page min/max
> string statistics to PAGE_INDEX_MAX_STRING_LENGTH via TruncateDown() /
> TruncateUp() in be/src/exec/parquet/hdfs-parquet-table-writer.cc. These
> truncate at an arbitrary byte offset, which can split a multi-byte UTF-8
> sequence and produce a min/max value that is not itself valid UTF-8.
> The Parquet spec requires the truncated min/max in the column/page index to
> remain valid values of the column's logical type:
> https://github.com/apache/parquet-format/blob/219e3f12a62f9476e830c21e26d030d231f7c017/src/main/thrift/parquet.thrift#L300
> When Impala annotates STRING columns as UTF-8 by default (IMPALA-12675),
> writing an invalid-UTF-8 min/max into a UTF-8-annotated column produces
> internally inconsistent metadata and can confuse readers that trust the
> annotation.
> Proposal:
> - Add UTF-8-aware variants of TruncateDown/TruncateUp that truncate on
> character boundaries (and, for TruncateUp, increment the last whole code
> point correctly). Reference implementation: parquet-java's BinaryTruncator:
> https://github.com/apache/parquet-java/blob/master/parquet-column/src/main/java/org/apache/parquet/internal/column/columnindex/BinaryTruncator.java
> - Check other engines' behavior for parity.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
