Gabriel39 opened a new pull request, #69625:
URL: https://github.com/apache/doris/pull/69625
### What problem does this PR solve?
Lance profiles lack Session metadata-cache miss diagnostics, ordinary scalar
scans choose competing indexes by coverage and name rather than predicate
selectivity, and Foyer memory capacity is coupled to the data-cache read block
size.
This PR targets **branch-4.2** and adds:
- `LanceSessionIndexCacheMisses` and `LanceSessionMetadataCacheMisses` in
the reader profile. They report BE-wide Session deltas observed during each
reader's dataset-open windows, including dataset opening. Availability and
scope strings distinguish unavailable statistics from zero misses and
explicitly describe concurrent activity. Overlapping reader windows are not
query-exclusive counts and must not be summed as an exact query total.
- `enable_lance_scalar_index_auto_selection`, enabled by default. Ordinary
scans compare estimated driver candidates plus uncovered physical rows using
Nereids predicate estimates. Disabling it retains coverage-based selection.
Existing index eligibility, fragment ownership, fallback scans, and vector/FTS
filter semantics are preserved.
- `lance_data_cache_memory_capacity_bytes`, independent of read-block and
disk capacities. The default `0` retains the previous one-configured-block
memory tier, including nondefault block sizes. Explicit capacities are passed
to Foyer; negative values are rejected before unsigned conversion.
Usage and statistical limitations are documented in
`docs/lance-cache-and-index-selection.md`.
### Release note
Add Lance Session cache-miss profile diagnostics, selectivity-aware
scalar-index selection controlled by a session variable, and independently
configurable Foyer memory capacity.
### Validation
- 81 focused FE tests passed, including selection costs, partial coverage,
unknown/invalid estimates, cross-field residuals, variable forwarding,
scan-range serialization, and propagation of real column statistics through
physical-plan translation.
- Confirmed the original branch-4.2 scalar planner fails the new selection
checks.
- Two focused BE capacity tests passed in an ASAN-linked test harness.
- Changed BE translation units and test sources passed compiler syntax
checks with freshly generated protobuf headers.
- FE Checkstyle, clang-format 16 on all affected C++ files, and `git diff
--check` passed.
- The complete BE unit suite and full build still require CI; they were not
verified locally.
### Check List (For Author)
- Test
- [x] Unit Test
- [ ] Regression test
- [ ] Manual test
- Behavior changed:
- [x] Yes. Scalar-index cost selection is enabled by default and can be
disabled. Foyer's default memory behavior is preserved.
- Does this need documentation?
- [x] Yes. Usage notes are included in this PR.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
