hubgeter opened a new pull request, #70284:
URL: https://github.com/apache/doris/pull/70284
### What problem does this PR solve?
Issue Number: N/A
Related PR: #65674
Problem Summary:
Repeated scans of hot Parquet files still decode page encodings and
materialize columns after page-cache hits. Add an optional cache of complete
decoded row-group output columns so subsequent scans can copy native values
without constructing the corresponding native reader or prefetching its column
chunks.
The cache defaults off (`parquet_decoded_column_cache_capacity_bytes=0`).
Its initial scope is immutable files and TINYINT, SMALLINT, INT, BIGINT, FLOAT
and DOUBLE, including nullable columns. Mutable files bypass it even with a
nonzero mtime because seconds-resolution timestamps cannot distinguish
same-size in-place overwrites. The key includes hashed source properties, file
identity, row group, physical leaf, field ID, output type and strict decode
mode. Timestamp, decimal, string/binary, nested and Variant columns use the
existing path.
Only naturally complete, unfiltered output scans fill entries. Partial
reads, selected reads, skips, early close and decode errors cannot publish
incomplete columns. Predicate columns remain outside admission because direct
predicate paths can produce dictionary IDs or placeholders. Complete hits still
support filtered output and physical row selection; table casts, defaults and
deletes run normally.
Reuse the existing LRU/CacheManager and tracked page allocation. A shared
reservation budget covers filling, resident and pinned entries, including
replaced entries, with entry and per-file admission limits. Add
hit/miss/write/bypass/byte counters and copy timers, plus focused cache tests
and a real Parquet scan test covering warm reuse, filtering and mutable-file
overwrite. Document scope and configuration in the existing scan design.
Validation status:
- `build-support/clang-format.sh` with clang-format 16 and
`build-support/check-format.sh`: passed.
- `build-support/check-build-hygiene.sh`: passed.
- New cache and test translation units checked with
`build-support/run-clang-tidy.sh` using individual commands derived from the
ASAN build database (PCH omitted; the local Clang resource directory supplied).
- Full ASAN BE build and focused BE unit-test execution are in progress; no
passing execution or performance result is claimed yet. This draft remains
pending that validation and CI.
### Release note
Add an opt-in, memory-bounded cache for complete decoded Parquet numeric
row-group columns from immutable files. It is disabled by default.
### Check List (For Author)
- Test
- [ ] Regression test
- [ ] Unit Test
- [x] Manual test (add detailed scripts or steps below)
- Formatting, header hygiene and static analysis as listed above;
runtime validation is pending.
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason
- Behavior changed:
- [ ] No.
- [x] Yes. Optional decoded-column reuse when the BE cache is enabled;
defaults remain unchanged.
- Does this need documentation?
- [ ] No.
- [x] Yes. Configuration and support boundaries are documented in
`docs/file-scanner-v2-parquet-scan-design.md`.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
