eldenmoon opened a new pull request, #69773:
URL: https://github.com/apache/doris/pull/69773
### What problem does this PR solve?
Issue Number: None
Problem Summary:
Selective scans returning a wide VARIANT eagerly expand all physical
subcolumns before storage pruning rejects a segment. When the working set
exceeds the per-segment reader cache, each query reconstructs thousands of
iterators for segments that return no rows.
Defer non-predicate VARIANT iterators until index, delete-bitmap and
row-range pruning leaves rows. Keep predicate/delete-condition and TopN inputs
eager, preserve virtual iterators populated by index evaluation, and initialize
surviving projections before materialization and cloud prefetch. Empty batches
skip the column read phase because their deferred iterators are absent.
Reproduced on master `a47d4613d2d`. Publication-time master `62e3d81a42b`
adds only an unrelated FE Iceberg change; the BE code is unchanged.
**RELEASE benchmark:** one local shared-nothing BE, 2095 physical
subcolumns, 435 segments, 873 input rows, two projected VARIANT columns and
three result rows. BE caches retain their defaults, including reader-cache
capacity 100. BE execution is limited to two CPUs using `num_cores=2` and
thread affinity. The fixture sets `variant_max_subcolumns_count=2108` and
disables table auto compaction to hold the layout fixed.
| Metric | Before | After |
|---|---:|---:|
| Warm client latency, median of three rounds | 5.851 s | 0.224 s |
| Scanner CPU for the representative warm query | 6.705 s | 118.330 ms |
| Column-iterator initialization, accumulated | 9.845 s | 60.854 ms |
| Hierarchical VARIANT iterators per query | 870 | 4 |
| Reader-cache misses during the query, process-wide | 920249 | 4342 |
Both representative profiles report 435 candidate segments and 870 rows
filtered by inverted indexes. Only the two surviving segments now construct the
two VARIANT projections. All four projection result hashes match before/after
and under ASAN.
Timings exclude the first query after restart; filesystem caches were not
cleared. CPU flame graphs were also collected. Their windows contain different
numbers of queries, so raw sample totals are not used as per-query speedup
evidence. Measurements use synthetic data; live cloud validation remains
pending.
### Release note
Reduce scan initialization overhead for selective queries returning wide
VARIANT objects by avoiding projection iterators for segments eliminated by
storage pruning.
### Check List (For Author)
- Test:
- [x] Unit Test: seven focused ASAN tests passed. Three initialization
expectations fail on unmodified master. Coverage includes surviving/default
NULL reads, delete/range-pruned EOF, eager condition/TopN inputs, retained
virtual-column results, and the existing nullable virtual-column test.
- [x] Regression test:
`variant_p0/test_variant_projection_deferred_init` generated with the runner
and verified on ASAN. Covers index hit/empty, VARIANT predicates, expressions,
subtrees/arrays, TopN, SQL NULL, vector filtering and MoW delete/replace.
- [x] Regression test: existing
`variant_p0/predefine/test_compaction_variant_predefine_with_sparse_limit`
passed on ASAN.
- [x] Manual test: large-fixture projection hashes match ASAN/RELEASE;
unmodified master RELEASE also passes the final SQL golden as a semantic
control.
- [x] Manual test: RELEASE A/B profiles and CPU samples, clang-format
16, build hygiene, changed-line product clang-tidy and full new-test clang-tidy
passed.
- Behavior changed:
- [x] Yes. Projection initialization is deferred; query results,
configuration defaults, persisted data and FE/BE protocols are unchanged.
- Does this need documentation?
- [x] No. Internal scan optimization.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
