bobhan1 opened a new pull request, #63056:
URL: https://github.com/apache/doris/pull/63056
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary:
This PR refactors segment data-file prefetch for file cache reads and adds a
cache-block-aware remote reader.
Previously, segment prefetch was managed outside the file reader and
depended on a shared underlying reader across columns. That made the prefetch
state harder to isolate, required upper layers to manually trigger prefetch,
and coupled rowid/page knowledge directly to the prefetcher. For cold
object-storage scans, this also limited the ability to issue parallel
cache-block-sized reads early enough to trade object-store IOPS for bandwidth.
This PR changes the design as follows:
1. Add `CacheBlockAwarePrefetchRemoteReader`.
- It inherits from `CachedRemoteFileReader`.
- It owns at most one read pattern, so progress is isolated per physical
column iterator.
- Normal foreground `read_at()` calls automatically advance the prefetch
window by file offset.
- `async_touch_initial_window()` lets callers touch the first window
before the first foreground read when the first ranges are guaranteed to be
read, such as predicate columns after segment index pruning.
- It uses file cache blocks as the prefetch granularity and submits
asynchronous local-cache touch tasks through
`CachedRemoteFileReader::async_touch_local_cache()`.
2. Split rowid-to-file-range conversion into `SegmentFileAccessRangeBuilder`.
- The builder converts selected rowids through the ordinal index into
data-page file ranges.
- It also supports full data-page range construction for full-segment
readers.
- The IO layer expands those file ranges to cache blocks, preserving
large pages and pages that cross cache-block boundaries.
3. Remove the old segment prefetcher path.
- When segment file-cache prefetch is enabled, segment data-file readers
are created as independent `CacheBlockAwarePrefetchRemoteReader` instances.
- Multiple columns no longer share the same underlying cache-aware
reader, so each physical iterator has an isolated prefetch cursor and no
external pattern id is needed.
- Predicate columns pre-touch the first prefetch window after their
pattern is installed because their candidate ranges must be read to evaluate
predicates.
- Non-predicate and common-expression columns keep the normal
`read_at()`-triggered behavior because their final rowids are produced batch by
batch after predicate evaluation.
4. Rename the lower-level cache warming API.
- `CachedRemoteFileReader::prefetch_range()` is renamed to
`async_touch_local_cache()` to describe the actual behavior more clearly:
asynchronously download the requested range into the local file cache.
5. Add focused BE unit tests.
- Cover rowid-to-file-range building.
- Cover cache-block plan construction and cursor advancement.
- Cover large pages and pages crossing cache-block boundaries.
- Cover `read_at()`-triggered prefetch.
- Cover independent per-reader patterns for different column iterators.
- Cover the initial-window touch path used by predicate columns.
The optimization intentionally spends more object-storage IOPS to expose
more parallelism and improve aggregate bandwidth for cold segment reads. Query
results are unchanged.
### Release note
None
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [x] Unit Test
- `build-support/clang-format.sh`
- `git diff --check`
- `./run-be-ut.sh --run
--filter=CacheBlockAwarePrefetchRemoteReaderTest.*:BlockFileCacheTest.usage_example_read_at_automatically_prefetches_single_pattern:BlockFileCacheTest.cached_remote_file_reader_async_touch_local_cache_downloads_range:BlockFileCacheTest.cache_block_aware_prefetch_remote_reader*
-j100`
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [x] No. Query results are unchanged. The only behavioral difference is
file-cache prefetch scheduling when segment file-cache prefetch is explicitly
enabled.
- [ ] Yes. <!-- Explain the behavior change -->
- Does this need documentation?
- [x] No.
- [ ] Yes. <!-- Add document PR link here. eg:
https://github.com/apache/doris-website/pull/1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR should
merge into -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
