Gabriel39 opened a new pull request, #65674:
URL: https://github.com/apache/doris/pull/65674
### What problem does this PR solve?
Issue Number: None
Related PR: None
FileScannerV2's Parquet path previously depended on Arrow metadata/value
adapters and built intermediate decoded objects before materializing Doris
columns. That design duplicated decoding and conversion, retained large
temporary buffers for nested and string columns, and made sparse predicate
reads expensive—especially when nullable values forced selected decoding into
many small read/skip operations.
This PR completes an independent native Parquet implementation under
`format_v2`. The V2 path now owns its metadata tree, page/level/value decoders,
predicate scheduling, and Doris-column materialization without calling the
legacy V1 reader or falling back to Arrow value materialization.
### What is changed?
#### Native metadata and reader ownership
- Replace the Parquet metadata adapter with a native metadata tree used
consistently by schema mapping, Row Group planning, statistics, dictionary
pruning, Bloom filters, page indexes, and scan readers.
- Preserve compatible LIST/MAP/STRUCT wrapper semantics while validating
malformed schema nodes, offsets, sizes, page headers, level counts, dictionary
IDs, and index metadata before use.
- Keep all new implementation in `format_v2`; FE and the legacy
`be/src/format` V1 reader remain unchanged.
#### Native page decoding and direct materialization
- Decode Page V1/V2 definition and repetition levels and native value
streams for PLAIN, BOOLEAN RLE, dictionary, DELTA_BINARY_PACKED,
DELTA_LENGTH_BYTE_ARRAY, DELTA_BYTE_ARRAY, and BYTE_STREAM_SPLIT encodings.
- Keep physical decoding in the native decoder and physical-to-logical
conversion in `DataTypeSerDe`, appending directly into final Doris columns.
- Reconstruct ARRAY/MAP/STRUCT columns from native levels and child readers;
levels-only/count scans do not materialize unused payload values.
- Remove obsolete Arrow RecordReader, Arrow builder/value-reader,
metadata-adapter, compatibility overload, and I/O shim paths from the V2
Parquet chain.
#### Sparse nullable and dictionary optimization
- Normalize sparse selections into selected non-null physical ranges, decode
them in batches, and restore nullable logical positions separately. Scalar
primitive, DECIMAL, DATE/DATETIME, string/binary, and dictionary paths no
longer fall back to per-run decode merely because NULLs are interleaved.
- Stream repeated and literal dictionary IDs directly into the destination
for dense/cache-resident dictionaries. For sparse reads whose typed dictionary
exceeds L2, retain only selected IDs and perform one compact gather.
- Materialize each typed dictionary once per generation, reuse it for
pruning/filtering/output, validate every ID, and roll back partial output on
late corruption or conversion failure.
#### Predicate and batch execution
- Evaluate eligible predicate-only PLAIN `INT`, `BIGINT`, `FLOAT`, and
`DOUBLE` literal comparisons directly on physical values, including nullable
remapping, without materializing or compacting a Doris predicate column.
- Retain predicate row mappings across single-column rounds and compact only
at required multi-column/delete/output boundaries.
- Base adaptive growth on physical rows consumed and allow fully filtered
batches to grow independently of final output Block size, preventing the
scanner from remaining at a tiny batch size after consecutive empty outputs.
- Reuse bounded decoder, level, selection, null-map, binary, and nested
scratch; release exceptional retained buffers after their parent shape is
committed.
#### I/O, cache, and observability
- Use the native footer/page cache and stable file identity rules, coalesce
safe remote ranges, and keep decoder/dictionary mutable state outside shared
caches.
- Add detailed counters for native decoding, sparse selections and NULL
fallback, predicate compaction, PLAIN direct predicates, dictionary filtering,
adaptive batches, cache activity, and retained scratch.
- Update the FileScannerV2 design and review documents to describe the
native metadata, decoding, selection, predicate, cache, fallback, and ownership
contracts.
### Correctness and fallback rules
- Unsupported encodings, logical conversions, projected direct-predicate
columns, mixed dictionary/plain chunks, nested/repeated optimization
boundaries, or incomplete indexes are rejected or conservatively handled before
consuming decoder cursors.
- Optimization fallbacks remain inside the V2 implementation and preserve
logical/value cursor alignment; the V2 scan path does not delegate to V1.
- Missing statistics, dictionaries, Bloom filters, page indexes, or stable
cache identities only disable the corresponding optimization and never remove
candidate rows.
### Release note
FileScannerV2 Parquet scans now use a native Doris metadata tree and native
page decoders, with batched nullable sparse decoding, cache-aware dictionary
materialization, and direct physical predicates for eligible PLAIN primitive
columns.
### Check List (For Author)
- Test: Unit Test
- `ninja -C be/ut_build_ASAN doris_be_test -j128`
- Relevant ASAN suites: 208 tests from 8 suites passed.
- CI-compatible `clang-format 16.0.6` verification passed for all files
reported by the formatter job.
- Behavior changed: Yes. FileScannerV2 uses the native V2 Parquet
metadata/decode/materialization chain and its selection-first predicate paths.
- Does this need documentation: Yes. Updated
`docs/file-scanner-v2-parquet-scan-design.md`,
`docs/file-scanner-v2-code-review-guide.md`, and
`docs/file-scanner-v2-design.md`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
