Zoltán Borók-Nagy created IMPALA-15427:
------------------------------------------
Summary: Batch-decode Parquet values of nullable columns with
scattered NULLs
Key: IMPALA-15427
URL: https://issues.apache.org/jira/browse/IMPALA-15427
Project: IMPALA
Issue Type: Improvement
Components: Backend
Reporter: Zoltán Borók-Nagy
ScalarColumnReader::ReadValueBatch()
(be/src/exec/parquet/parquet-column-readers.cc
around line 528) has two paths:
* When the definition levels are a repeated RLE run (all NULL or all non-NULL),
it uses
the batched MaterializeValueBatchRepeatedDefLevel() / DecodeValues() path.
* When they are a bit-packed literal run, which is typical when NULLs are
scattered, it
caches up to 1024 definition levels. MaterializeValueBatch() (around line 574)
then
decodes each non-NULL value on its own through ReadSlot() / GetNextValue().
Real data with a few percent NULLs spends most of its time on the second, one
value at
a time path. For example, most TPC-DS fact-table columns have about 4% NULLs.
This path
has no codegen.
h3. Proposed fix
Add a batched variant for the common case: not in a collection, no
file-position slot,
no page filtering, and no type conversion or validation. It would:
# count the non-NULL values in the cached definition-level window;
# decode them with one DecodeValues() call into a 1024-entry stack buffer;
# scatter them forward into the tuples, setting null bits for the NULL
positions.
Conversion and validation types (DATE, TIMESTAMP, decimal rescale),
collections, Iceberg
position deletes and page filtering stay on the current path.
h3. Measurements
Standalone replica built from copies of Impala's RLE and dictionary decoders,
reader
time:
||NULL ratio||Speedup||
|0%|1.00x (no change)|
|1%|1.22–1.94x|
|4%|1.29–1.56x (e.g. 2.80 → 1.90 ns/value, 12-bit dictionary)|
|30%|1.25–1.37x|
Two other variants were measured and rejected:
* spreading values backwards in place: 0.95–1.13x on 64/128-byte tuples;
* capping literal runs: 0.86–0.93x on wide tuples.
The remaining cost is mostly definition-level decoding, about 0.5 ns/value.
Validation needed: the Parquet BE/EE tests, and an A/B run on TPC-DS
store_sales.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
