Hello Arnab Karmakar, Zoltan Borok-Nagy, Mihaly Szjatinya, Csaba Ringhofer,
Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24570
to look at the new patch set (#12).
Change subject: IMPALA-12675: Set PARQUET_ANNOTATE_STRINGS_UTF8 to true by
default and turn on validation
......................................................................
IMPALA-12675: Set PARQUET_ANNOTATE_STRINGS_UTF8 to true by default and
turn on validation
Now that Impala has a BINARY type for binary data and broader UTF-8
support, STRING columns written to Parquet are annotated with the
UTF-8 logical type by default (PARQUET_ANNOTATE_STRINGS_UTF8 now
defaults to true).
To keep the annotation truthful, STRING values are validated as UTF-8
on the Parquet write path (using the vendored simdutf library)
whenever the annotation is enabled. A value that is not valid UTF-8
aborts the write with PARQUET_INVALID_UTF8_STRING. Validation is an
inseparable consequence of the annotation: there is no option to keep
the annotation while writing unvalidated values, since that would
produce a file whose metadata lies about its contents. To store
non-UTF-8 bytes, set PARQUET_ANNOTATE_STRINGS_UTF8 to false to omit
the annotation, or use a BINARY column. CHAR/VARCHAR (always text) and
BINARY columns are not validated. Iceberg tables always annotate
STRING columns as UTF-8 (a spec requirement that is not configurable),
so their STRING values are always validated; use a BINARY column to
store non-UTF-8 bytes there.
Details:
* Vendor the simdutf 9.0.0 single-file amalgamation under
be/src/thirdparty/simdutf/, wrapped by IsValidUtf8() in
be/src/util/utf8-util.{h,cc} (with a unit test). The AVX-512 kernel
is disabled to match the treatment of the vendored roaring.c.
* Validate once per non-null value in
HdfsParquetTableWriter::AppendRow().
* Update the docs, including a new Impala 5.0.x entry in
impala_incompatible_changes.xml documenting the default flip as a
behavior change.
Testing:
* added new be test: Utf8UtilTest
* set PARQUET_ANNOTATE_STRINGS_UTF8=false in test_datasketches.py so it
can write the binary data into STRING columns
* add new tests to test_iceberg.py and test_scanners.py
* modified existing e2e testcases to match new estimations
Change-Id: Ia95bcba978863ffb5d603fdf81bb2c75ba06f7da
Assisted-by: Claude Opus 4.8 (Claude Code)
---
M LICENSE.txt
M be/src/exec/parquet/hdfs-parquet-table-writer.cc
M be/src/exec/parquet/parquet-metadata-utils.cc
A be/src/thirdparty/simdutf/LICENSE-APACHE
A be/src/thirdparty/simdutf/LICENSE-MIT
A be/src/thirdparty/simdutf/README.md
A be/src/thirdparty/simdutf/simdutf.cpp
A be/src/thirdparty/simdutf/simdutf.h
M be/src/util/CMakeLists.txt
A be/src/util/utf8-util-test.cc
A be/src/util/utf8-util.cc
A be/src/util/utf8-util.h
M bin/rat_exclude_files.txt
M bin/run_clang_tidy.sh
M common/thrift/Query.thrift
M common/thrift/generate_error_codes.py
M docs/topics/impala_incompatible_changes.xml
M docs/topics/impala_parquet_annotate_strings_utf8.xml
M fe/src/test/java/org/apache/impala/planner/CardinalityTest.java
M
java/calcite-planner/src/test/java/org/apache/impala/calcite/planner/TestCalciteStats.java
M testdata/workloads/functional-planner/queries/PlannerTest/pivot-clause.test
M testdata/workloads/functional-planner/queries/PlannerTest/unpivot-clause.test
M testdata/workloads/functional-query/queries/QueryTest/stats-extrapolation.test
M tests/query_test/test_datasketches.py
M tests/query_test/test_iceberg.py
M tests/query_test/test_insert_parquet.py
M tests/query_test/test_parquet_page_index.py
M tests/query_test/test_scanners.py
28 files changed, 82,847 insertions(+), 41 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/70/24570/12
--
To view, visit http://gerrit.cloudera.org:8080/24570
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ia95bcba978863ffb5d603fdf81bb2c75ba06f7da
Gerrit-Change-Number: 24570
Gerrit-PatchSet: 12
Gerrit-Owner: Daniel Vanko <[email protected]>
Gerrit-Reviewer: Arnab Karmakar <[email protected]>
Gerrit-Reviewer: Csaba Ringhofer <[email protected]>
Gerrit-Reviewer: Daniel Vanko <[email protected]>
Gerrit-Reviewer: Impala Public Jenkins <[email protected]>
Gerrit-Reviewer: Mihaly Szjatinya <[email protected]>
Gerrit-Reviewer: Zoltan Borok-Nagy <[email protected]>
