Mihaly Szjatinya has uploaded this change for review. (
http://gerrit.cloudera.org:8080/24579
Change subject: IMPALA-15139: Support DELTA_LENGTH_BYTE_ARRAY Parquet encoding
......................................................................
IMPALA-15139: Support DELTA_LENGTH_BYTE_ARRAY Parquet encoding
Add read support for the DELTA_LENGTH_BYTE_ARRAY Parquet encoding
(encoding enum value 6) for BYTE_ARRAY columns
(STRING, VARCHAR, BINARY).
The new ParquetDeltaLengthByteArrayDecoder class decodes the lengths
section using the existing ParquetDeltaDecoder<int32_t> infrastructure,
then slices bytes zero-copy from the page buffer into StringValues.
All string lengths are decoded eagerly on NewPage() so that subsequent
NextValue() / NextValues() / SkipValues() calls need only index into
the pre-decoded lengths vector.
Changes:
- be/src/exec/parquet/parquet-delta-length-byte-array-decoder.h/.cc:
New decoder class with NewPage, NextValue, NextValues, SkipValues,
and GetTotalValueCount.
- be/src/exec/parquet/parquet-metadata-utils.cc: Whitelist
DELTA_LENGTH_BYTE_ARRAY and DELTA_BINARY_PACKED in
IsEncodingSupported().
DELTA_BINARY_PACKED is listed in column chunk metadata by some writers
when encoding BYTE_ARRAY columns with DELTA_LENGTH_BYTE_ARRAY.
- be/src/exec/parquet/parquet-column-readers.h/.cc: Wire the new decoder
into InitDataDecoder (StringValue/BYTE_ARRAY specialization),
ReadValue, MaterializeValueBatch, SkipEncodedValuesInPage,
DecodeValue, and DecodeValues dispatch paths.
- be/src/exec/parquet/CMakeLists.txt: Register new source and
test target.
- testdata/parquet_delta_length_byte_array_encoding/: Pre-generated test
Parquet file and pyarrow generator script.
- tests/query_test/test_parquet_delta_length_byte_array_encoding.py:
E2E test suite.
Testing:
Unit tests (13/13 passed):
parquet-delta-length-byte-array-decoder-test
E2E tests (10/10 passed, with/without codegen):
tests/query_test/test_parquet_delta_length_byte_array_encoding.py
Parquet regression tests (315 passed, 14 expected skips):
tests/query_test/test_scanners.py::TestParquet
tests/query_test/test_parquet_stats.py
Change-Id: I9e4395fa30cd7fb2c89ef5a2b16b043d4de60b52
Assisted-by: Claude Sonnet 4.6 (GitHub Copilot)
---
M be/src/exec/parquet/CMakeLists.txt
M be/src/exec/parquet/parquet-column-readers.cc
A be/src/exec/parquet/parquet-delta-length-byte-array-decoder-test.cc
A be/src/exec/parquet/parquet-delta-length-byte-array-decoder.cc
A be/src/exec/parquet/parquet-delta-length-byte-array-decoder.h
M be/src/exec/parquet/parquet-metadata-utils.cc
A testdata/parquet_delta_length_byte_array_encoding/README
A
testdata/parquet_delta_length_byte_array_encoding/mixed_types_delta_length_byte_array.parquet
A testdata/parquet_delta_length_byte_array_encoding/parquet_files_generator.py
A tests/query_test/test_parquet_delta_length_byte_array_encoding.py
10 files changed, 789 insertions(+), 1 deletion(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/79/24579/1
--
To view, visit http://gerrit.cloudera.org:8080/24579
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I9e4395fa30cd7fb2c89ef5a2b16b043d4de60b52
Gerrit-Change-Number: 24579
Gerrit-PatchSet: 1
Gerrit-Owner: Mihaly Szjatinya <[email protected]>
