bobhan1 opened a new pull request, #69982:
URL: https://github.com/apache/doris/pull/69982
### What problem does this PR solve?
Issue Number: None
Related PRs: #67292, #69116
Backport the merged PageIO read-ahead, I/O coalescing, and partial File
Cache block hole-fill pipeline to `branch-4.2`, together with its file-level
cache metadata restoration prerequisite.
The two commits are applied in dependency order:
1. #69116 — `e9b337360c87917a57c49ca5bbe696f0de526669`: restore all
persisted cache blocks for a file during lazy metadata loading. This provides
the file-level restoration API used by the read-ahead writeback coverage check.
2. #67292 — `9e61225e3ee8aa3235a4f7ab9647c3c53a304047`: PageIO read-ahead,
I/O coalescing, and partial-block hole filling.
The feature remains disabled by default. Eligible cloud scalar scans and
point queries use range read-ahead only when both
`enable_query_read_ahead=true` and `enable_async_file_cache_write=true`.
Consumed ranges can populate File Cache through complete-block writeback or
bounded background hole filling. Complex and Variant columns retain their
existing readers.
Adaptations for 4.2:
- Use 4.2's `Schema`, sparse column IDs, materialization-stage lists, and
`StorageReadOptions` statistics pointers. Add coverage for unselected columns
between requested IDs.
- Keep the existing point-query default-value handling for columns not yet
present in the BE tablet schema. Use 4.2's `seek_and_read_by_rowid(const
TabletSchema&, ...)` API and load the row-store Segment for the enabled path;
runtime switch changes reuse that Segment for synchronous fallback.
- Provide the row-store test fixture locally because 4.2 does not contain
the upstream `MowTransformTestBase` fixture. Adapt column writes to 4.2's
raw-data API and add a schema-taking `MockSegment` constructor for the new
tests.
- Keep the target branch's absence of `build-support/check-header-deps.py`.
### Release note
Add opt-in PageIO range read-ahead, I/O coalescing, and partial File Cache
block hole filling for cloud queries. The feature is disabled by default. Also
restore all persisted cache blocks of a file during BE startup metadata loading.
### Check List (For Author)
- Test:
- [x] Unit Test
- ASAN BE unit-test build and focused tests passed with PCH enabled and
4.2 prebuilt third-party dependencies: **643 passed, 3 existing tests skipped,
0 failures, 35 suites**. XML verification confirms all **206 newly added test
definitions** passed, including both startup metadata-restoration parameterized
suites.
- Changed-line clang-format 16 and `git diff --check` passed.
- No cluster regression or performance benchmark has been run for this
backport.
- Behavior changed:
- [x] Yes. Eligible cloud queries use the new read path when explicitly
enabled; the feature is disabled by default.
- Does this need documentation?
- [x] Yes. Public configuration documentation remains pending as in
#67292.
<details>
<summary>Focused test command</summary>
```bash
ENABLE_PCH=ON ./run-be-ut.sh --run \
--filter='*FileRangePlannerTest.*:*HoleFillPlannerTest.*:*PartialBlockWritebackManagerTest.*:*PartialBlockWritebackOptionsTest.*:*RangeCacheWritebackTest.*:*RangeWritebackDispatcherTest.*:*FileRangeCoalescerTest.*:*FileRangeReadSchedulerTest.*:*ReadAheadMetricsTest.*:*ReadIOTraceTest.*:*ReadIOTraceWriterTest.*:*S3FileReaderTest.*:*QueryContextReadAheadTest.*:*ColumnReadAheadTest.*:*SegmentReadAheadTest.*:*SegmentIteratorReadAheadTest.*:*ColumnReaderTest.*:*AsyncCacheWriteManagerTest.*:*AsyncCachedRemoteFileReaderTest.*:*AsyncCacheWriteConfigTest.*:*PointQueryReadAheadTest.*:*RowCacheTest.*:*LookupConnectionCacheTest.*:*BlockFileCacheTest.*:*FileCacheLazyRestoreTest.*:*PeerFileCacheLazyRestoreTest.*:*CacheBlockMetaStoreTest.*:*CachedRemoteFileReaderPeerTest.*:*CrossCGWinnerRaceTest.*:*S3WriteBufferConfigTest.*:*CountEmitShortcut.*:*CountOnIndexFastpath.*'
-j100
```
The skipped pre-existing cases are `BlockFileCacheTest.DISABLE_test_load`,
`BlockFileCacheTest.DISABLE_test_upgrade_cache_dir_version`, and
`BlockFileCacheTest.DISABLE_cached_remote_file_reader_direct_read_and_evict_cache`.
</details>
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
