Dániel Gábor Vankó created IMPALA-15413:
-------------------------------------------
Summary: Truncate Parquet page-index min/max values on UTF-8
character boundaries
Key: IMPALA-15413
URL: https://issues.apache.org/jira/browse/IMPALA-15413
Project: IMPALA
Issue Type: Bug
Components: Backend
Reporter: Dániel Gábor Vankó
When Impala writes the Parquet page index, it truncates per-page min/max string
statistics to PAGE_INDEX_MAX_STRING_LENGTH via TruncateDown() / TruncateUp() in
be/src/exec/parquet/hdfs-parquet-table-writer.cc. These truncate at an
arbitrary byte offset, which can split a multi-byte UTF-8 sequence and produce
a min/max value that is not itself valid UTF-8.
The Parquet spec requires the truncated min/max in the column/page index to
remain valid values of the column's logical type:
https://github.com/apache/parquet-format/blob/219e3f12a62f9476e830c21e26d030d231f7c017/src/main/thrift/parquet.thrift#L300
When Impala annotates STRING columns as UTF-8 by default (IMPALA-12675),
writing an invalid-UTF-8 min/max into a UTF-8-annotated column produces
internally inconsistent metadata and can confuse readers that trust the
annotation.
Proposal:
- Add UTF-8-aware variants of TruncateDown/TruncateUp that truncate on
character boundaries (and, for TruncateUp, increment the last whole code point
correctly). Reference implementation: parquet-java's BinaryTruncator:
https://github.com/apache/parquet-java/blob/master/parquet-column/src/main/java/org/apache/parquet/internal/column/columnindex/BinaryTruncator.java
- Check other engines' behavior for parity.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
