[
https://issues.apache.org/jira/browse/IMPALA-15414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dániel Gábor Vankó updated IMPALA-15414:
----------------------------------------
Labels: UTF-8 parquet (was: )
> Validate CHAR/VARCHAR values as UTF-8 when writing Parquet
> ----------------------------------------------------------
>
> Key: IMPALA-15414
> URL: https://issues.apache.org/jira/browse/IMPALA-15414
> Project: IMPALA
> Issue Type: Improvement
> Components: Backend
> Reporter: Dániel Gábor Vankó
> Priority: Major
> Labels: UTF-8, parquet
>
> IMPALA-12675 makes Impala annotate STRING Parquet columns as UTF-8 by default
> and validate their values on the write path, rejecting non-UTF-8 bytes with
> PARQUET_INVALID_UTF8_STRING. CHAR and VARCHAR columns are always annotated as
> UTF-8 (unconditionally, in ParquetMetadataUtils::FillSchemaElement()), but
> unlike STRING they are not validated — so it is currently possible to write
> non-UTF-8 bytes into a CHAR/VARCHAR Parquet column while still emitting a
> UTF-8 annotation, i.e. a file whose metadata does not match its contents.
> This asymmetry is deliberate for now: removing the escape hatch for
> CHAR/VARCHAR would be a larger behavior change and deserves its own patch.
> This ticket tracks doing that consistently.
> Proposal:
> - Validate CHAR/VARCHAR values as UTF-8 on the Parquet write path, consistent
> with STRING (BaseColumnWriter::ShouldValidateUtf8() in
> be/src/exec/parquet/hdfs-parquet-table-writer.cc currently restricts
> validation to TYPE_STRING).
> - Decide the opt-out story: unlike STRING there is no per-column annotation
> toggle for CHAR/VARCHAR, so validation must not leave users without a way to
> store legacy data. Consider gating behind an option or documenting the
> migration path.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
