Dániel Gábor Vankó created IMPALA-15414:
-------------------------------------------
Summary: Validate CHAR/VARCHAR values as UTF-8 when writing Parquet
Key: IMPALA-15414
URL: https://issues.apache.org/jira/browse/IMPALA-15414
Project: IMPALA
Issue Type: Improvement
Components: Backend
Reporter: Dániel Gábor Vankó
IMPALA-12675 makes Impala annotate STRING Parquet columns as UTF-8 by default
and validate their values on the write path, rejecting non-UTF-8 bytes with
PARQUET_INVALID_UTF8_STRING. CHAR and VARCHAR columns are always annotated as
UTF-8 (unconditionally, in ParquetMetadataUtils::FillSchemaElement()), but
unlike STRING they are not validated — so it is currently possible to write
non-UTF-8 bytes into a CHAR/VARCHAR Parquet column while still emitting a UTF-8
annotation, i.e. a file whose metadata does not match its contents.
This asymmetry is deliberate for now: removing the escape hatch for
CHAR/VARCHAR would be a larger behavior change and deserves its own patch. This
ticket tracks doing that consistently.
Proposal:
- Validate CHAR/VARCHAR values as UTF-8 on the Parquet write path, consistent
with STRING (BaseColumnWriter::ShouldValidateUtf8() in
be/src/exec/parquet/hdfs-parquet-table-writer.cc currently restricts validation
to TYPE_STRING).
- Decide the opt-out story: unlike STRING there is no per-column annotation
toggle for CHAR/VARCHAR, so validation must not leave users without a way to
store legacy data. Consider gating behind an option or documenting the
migration path.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
