Impala Public Jenkins has submitted this change and it was merged. (
http://gerrit.cloudera.org:8080/24579 )
Change subject: IMPALA-15139: Support DELTA_LENGTH_BYTE_ARRAY Parquet encoding
......................................................................
IMPALA-15139: Support DELTA_LENGTH_BYTE_ARRAY Parquet encoding
Add read support for the DELTA_LENGTH_BYTE_ARRAY Parquet encoding
(encoding enum value 6) for BYTE_ARRAY columns
(STRING, VARCHAR, BINARY).
The new ParquetDeltaLengthByteArrayDecoder class decodes the lengths
section using the existing ParquetDeltaDecoder<int32_t> infrastructure,
then slices bytes from the page buffer into StringValues. Short strings
(up to StringValue::SMALL_LIMIT bytes) are smallified into the
StringValue's inline storage so the page buffer can be released early;
longer strings remain as zero-copy references into the page buffer.
Lengths are decoded lazily in fixed-size batches (1024 at a time) to
keep memory use O(1) regardless of page size. The data page header's
num_values is used as a trusted bound to reject corrupt in-page counts.
DELTA_BINARY_PACKED is also whitelisted in IsEncodingSupported() because
some writers list it in column chunk metadata when encoding BYTE_ARRAY
columns with DELTA_LENGTH_BYTE_ARRAY.
A missing miniblocks_in_block_ == 0 guard is added to
ParquetDeltaDecoder to avoid undefined behaviour on corrupt input.
The Trino minicluster session-property-config files that disabled
DELTA_LENGTH_BYTE_ARRAY as a workaround are removed, and a new interop
test verifies that Impala reads Trino-written Parquet tables correctly.
Testing:
parquet-delta-length-byte-array-decoder-test: unit tests for the new
decoder (malformed input, empty strings, batch decode, skip).
tests/query_test/test_scanners.py
::TestParquetV2::test_delta_length_byte_array_encoding:
end-to-end tests reading pre-generated Parquet files with
DELTA_LENGTH_BYTE_ARRAY-encoded columns, with and without codegen.
::TestParquetV2::test_delta_length_byte_array_smallify_boundary:
correctness at the Smallify boundary (len=11 vs len=12).
::TestParquetV2::test_delta_length_byte_array_large_table
(exhaustive): 2000-value page exercises FillLengthsBuffer() refill
end-to-end.
::TestParquetV2::test_delta_length_byte_array_multipage
(exhaustive): 2-page file (1200 rows per page) exercises cross-page
decoding and NewPage() state reset.
tests/custom_cluster/test_trino_interop.py
::test_impala_reads_trino_parquet: Trino writes a Parquet table
with VARCHAR columns (Trino 482 uses DELTA_LENGTH_BYTE_ARRAY by
default) and Impala reads it back.
be/src/benchmarks/parquet-delta-length-byte-array-benchmark.cc:
micro-benchmark
Regression:
tests/query_test/test_scanners.py::TestParquet
tests/query_test/test_scanners.py::TestParquetV2
tests/query_test/test_parquet_stats.py
Change-Id: I9e4395fa30cd7fb2c89ef5a2b16b043d4de60b52
Assisted-by: Claude Sonnet 4.6 (GitHub Copilot)
Reviewed-on: http://gerrit.cloudera.org:8080/24579
Reviewed-by: Impala Public Jenkins <[email protected]>
Tested-by: Impala Public Jenkins <[email protected]>
---
M be/src/benchmarks/CMakeLists.txt
A be/src/benchmarks/parquet-delta-length-byte-array-benchmark.cc
M be/src/exec/parquet/CMakeLists.txt
M be/src/exec/parquet/parquet-column-readers.cc
M be/src/exec/parquet/parquet-delta-decoder.cc
M be/src/exec/parquet/parquet-delta-decoder.h
A be/src/exec/parquet/parquet-delta-length-byte-array-decoder-test.cc
A be/src/exec/parquet/parquet-delta-length-byte-array-decoder.cc
A be/src/exec/parquet/parquet-delta-length-byte-array-decoder.h
M be/src/exec/parquet/parquet-metadata-utils.cc
M be/src/runtime/string-value.h
M common/thrift/generate_error_codes.py
A testdata/bin/minicluster_trino/.gitignore
M testdata/bin/minicluster_trino/Dockerfile
D testdata/bin/minicluster_trino/session-property-config.json
D testdata/bin/minicluster_trino/session-property-config.properties
A testdata/data/boundary_smallify_delta_length_byte_array.parquet
A testdata/data/large_delta_length_byte_array.parquet
A testdata/data/mixed_types_delta_length_byte_array.parquet
A testdata/data/multipage_delta_length_byte_array.parquet
A testdata/parquet_delta_length_byte_array_encoding/README
A testdata/parquet_delta_length_byte_array_encoding/parquet_files_generator.py
A
testdata/workloads/functional-query/queries/QueryTest/parquet-delta-length-byte-array-encoding.test
M tests/custom_cluster/test_trino_interop.py
M tests/query_test/test_scanners.py
25 files changed, 1,689 insertions(+), 53 deletions(-)
Approvals:
Impala Public Jenkins: Looks good to me, approved; Verified
--
To view, visit http://gerrit.cloudera.org:8080/24579
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I9e4395fa30cd7fb2c89ef5a2b16b043d4de60b52
Gerrit-Change-Number: 24579
Gerrit-PatchSet: 21
Gerrit-Owner: Mihaly Szjatinya <[email protected]>
Gerrit-Reviewer: Balazs Hevele <[email protected]>
Gerrit-Reviewer: Csaba Ringhofer <[email protected]>
Gerrit-Reviewer: Impala Public Jenkins <[email protected]>
Gerrit-Reviewer: Mihaly Szjatinya <[email protected]>
Gerrit-Reviewer: Zoltan Borok-Nagy <[email protected]>
