Gabriel39 opened a new pull request, #65972:
URL: https://github.com/apache/doris/pull/65972
### What problem does this PR solve?
File Scanner V2 still executes several fixed-width Parquet decode and filter
stages with scalar row loops. On hot scans, byte-stream-split transpose,
DELTA_BINARY_PACKED reconstruction, numeric dictionary materialization,
nullable expansion, and raw predicate comparison can become CPU bottlenecks
after I/O is cached.
### What is changed?
- Add runtime-dispatched AVX2 kernels with scalar fallbacks for:
- BYTE_STREAM_SPLIT 4-byte and 8-byte transpose
- DELTA_BINARY_PACKED INT32/INT64 prefix reconstruction
- 4-byte and 8-byte numeric dictionary ID gather
- in-place nullable compact-value expansion
- raw INT32/INT64/FLOAT/DOUBLE predicate comparison
- Wire the kernels only into File Scanner V2:
- native byte-stream-split and delta decoders
- cache-resident direct dictionary materialization
- `ColumnChunkReader` nullable POD expansion
- Parquet raw fixed-width predicate evaluation
- Preserve the existing generic paths for unsupported widths, small batches,
non-AVX2 CPUs, and non-cache-resident dictionary strategies.
- Preserve Parquet wrapping arithmetic and Doris floating-point ordering
(`NaN == NaN`, and NaN sorts above finite values).
- Extend the Parquet benchmark matrix:
- decoder selectivity boundaries: 0%, 1%, 10%, 50%, 90%, 100%
- 80 isolated `ParquetKernel` scenarios
- applicable INT32/INT64/FLOAT/DOUBLE physical types
- nullable rates and clustered/alternating null placement
- 32, 4,096, and 262,144 entry dictionary working sets
File Scanner V1 is unchanged.
### Check list
- [x] `ParquetSimdKernelsTest`: 5/5 passed
- [x] `ParquetBenchmarkScenariosTest`: 6/6 passed
- [x] ASAN CMake objects and `Core`/`Exprs`/`Format`/`Util` static libraries
compiled
- [x] Real Release `benchmark_main.cpp` compiled with Doris project flags
and `-Werror`
- [x] Release kernel benchmark runner registered all 80 scenarios
- [x] Representative smoke cases for all five kernels passed their
correctness oracles
- [x] `git diff --check`
The local full benchmark build was not used for performance claims because
the host's broken `git-submodule` pyenv shim forced an unrelated 14k-target
OpenBLAS rebuild. Stable before/after performance comparison should use the
checked-in matrix on a controlled host.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
