Hello Arnab Karmakar, Zoltan Borok-Nagy, Mihaly Szjatinya, Csaba Ringhofer,
Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24570
to look at the new patch set (#13).
Change subject: IMPALA-12675: Set PARQUET_ANNOTATE_STRINGS_UTF8 to true by
default and turn on validation
......................................................................
IMPALA-12675: Set PARQUET_ANNOTATE_STRINGS_UTF8 to true by default and
turn on validation
Now that Impala has a BINARY type for binary data and broader UTF-8
support, STRING columns written to Parquet are annotated with the
UTF-8 logical type by default (PARQUET_ANNOTATE_STRINGS_UTF8 now
defaults to true).
To keep the annotation truthful, STRING values are validated as UTF-8
on the Parquet write path (using the simdutf library) whenever the
annotation is enabled. A value that is not valid UTF-8 aborts the write
with PARQUET_INVALID_UTF8_STRING. Validation is an inseparable
consequence of the annotation: there is no option to keep the
annotation while writing unvalidated values, since that would produce a
file whose metadata lies about its contents. To store non-UTF-8 bytes,
set PARQUET_ANNOTATE_STRINGS_UTF8 to false to omit the annotation, or
use a BINARY column. CHAR/VARCHAR (always text) and BINARY columns are
not validated. Iceberg tables always annotate STRING columns as UTF-8
(a spec requirement that is not configurable), so their STRING values
are always validated; use a BINARY column to store non-UTF-8 bytes
there.
Details:
* Consume the simdutf 9.0.0 library from the native toolchain:
add IMPALA_SIMDUTF_VERSION and bump the toolchain build ids in
bin/impala-config.sh, register the package in
bin/bootstrap_toolchain.py, locate it with a new
cmake_modules/FindSimdutf.cmake, and link it via IMPALA_DEPENDENCIES.
It is wrapped by IsValidUtf8() in be/src/util/utf8-util.{h,cc} (with
a unit test).
* Validate once per non-null value in
HdfsParquetTableWriter::AppendRow().
* Update the docs, including a new Impala 5.0.x entry in
impala_incompatible_changes.xml documenting the default flip as a
behavior change.
Testing:
* added new be test: Utf8UtilTest
* set PARQUET_ANNOTATE_STRINGS_UTF8=false in test_datasketches.py so it
can write the binary data into STRING columns
* add new tests to test_iceberg.py and test_scanners.py
* modified existing e2e testcases to match new estimations
Change-Id: Ia95bcba978863ffb5d603fdf81bb2c75ba06f7da
Assisted-by: Claude Opus 4.8 (Claude Code)
---
M CMakeLists.txt
M be/CMakeLists.txt
M be/src/exec/parquet/hdfs-parquet-table-writer.cc
M be/src/exec/parquet/parquet-metadata-utils.cc
M be/src/util/CMakeLists.txt
A be/src/util/utf8-util-test.cc
A be/src/util/utf8-util.cc
A be/src/util/utf8-util.h
M bin/bootstrap_toolchain.py
M bin/impala-config.sh
A cmake_modules/FindSimdutf.cmake
M common/thrift/Query.thrift
M common/thrift/generate_error_codes.py
M docs/topics/impala_incompatible_changes.xml
M docs/topics/impala_parquet_annotate_strings_utf8.xml
M fe/src/test/java/org/apache/impala/planner/CardinalityTest.java
M
java/calcite-planner/src/test/java/org/apache/impala/calcite/planner/TestCalciteStats.java
M testdata/workloads/functional-planner/queries/PlannerTest/pivot-clause.test
M testdata/workloads/functional-planner/queries/PlannerTest/unpivot-clause.test
M testdata/workloads/functional-query/queries/QueryTest/stats-extrapolation.test
M tests/query_test/test_datasketches.py
M tests/query_test/test_iceberg.py
M tests/query_test/test_insert_parquet.py
M tests/query_test/test_parquet_page_index.py
M tests/query_test/test_scanners.py
25 files changed, 494 insertions(+), 75 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/70/24570/13
--
To view, visit http://gerrit.cloudera.org:8080/24570
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ia95bcba978863ffb5d603fdf81bb2c75ba06f7da
Gerrit-Change-Number: 24570
Gerrit-PatchSet: 13
Gerrit-Owner: Daniel Vanko <[email protected]>
Gerrit-Reviewer: Arnab Karmakar <[email protected]>
Gerrit-Reviewer: Csaba Ringhofer <[email protected]>
Gerrit-Reviewer: Daniel Vanko <[email protected]>
Gerrit-Reviewer: Impala Public Jenkins <[email protected]>
Gerrit-Reviewer: Mihaly Szjatinya <[email protected]>
Gerrit-Reviewer: Zoltan Borok-Nagy <[email protected]>
