bobhan1 opened a new pull request, #66209:
URL: https://github.com/apache/doris/pull/66209
### What problem does this PR solve?
Issue Number: None
Related PR: #65058
Problem Summary:
This backports #65058 to `branch-4.0`.
Remote scans can read a large amount of data from object storage. Before
this change, a query that had already exceeded its useful cache-write budget
could still keep writing later remote cache misses into file cache, increasing
cache churn and IO overhead.
This PR adds the session variable `file_cache_query_limit_bytes` to control
remote-scan read-through file-cache writes per query on each BE:
- `< 0`: disabled. This is the default.
- `= 0`: do not write remote-scan cache misses into local file cache from
the start of the query.
- `> 0`: allow remote-scan cache writes until the next cache block would
exceed the byte threshold. Once the threshold is exceeded, later cache misses
continue reading from remote storage but skip additional local file-cache
writes.
The limiter is query-wide within a BE and is propagated through scan,
segment loading, preload, segment footer, column-reader metadata, and
inverted-index metadata paths. The BE dynamic config
`file_cache_query_limit_segment_meta`, default `false`, optionally includes
segment footer and segment metadata cache writes in the same limit. Profile
counters expose the threshold, whether remote-only-on-miss was triggered, and
specialized cache-write bytes and timers.
The backport maps master storage-path and API changes to the corresponding
`branch-4.0` `olap`/`vec` code paths. Master-only modules that do not exist on
`branch-4.0` are not introduced.
### Release note
Add `file_cache_query_limit_bytes` for query-level remote-scan file-cache
write limiting, and add optional segment metadata accounting with
`file_cache_query_limit_segment_meta`.
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [x] Regression test
- [x] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
Validation:
- `git diff --check`
- `./build.sh --be --fe --cloud -j100`
- `./run-fe-ut.sh --run
org.apache.doris.qe.SessionVariablesTest,org.apache.doris.qe.ShortCircuitQueryContextTest`
(8 tests passed)
- Related BE ASAN unit tests for file-cache profiles and limiter behavior,
cached remote reads, inverted-index IO context, scanner, primary-key index,
segment footer, external column metadata, and column-reader cache (51 tests
across 8 suites passed)
- `env -u HTTP_PROXY -u HTTPS_PROXY -u http_proxy -u https_proxy -u
ALL_PROXY -u all_proxy ./run-regression-test.sh --run -d
cloud_p0/cache/remote_scan_no_write_file_cache -g docker -runMode=cloud
-dockerSuiteParallel 1` (2 suites passed)
- Behavior changed:
- [ ] No.
- [x] Yes. Remote scans can stop writing local file cache after
`file_cache_query_limit_bytes` is reached while continuing to read misses from
remote storage. Optional segment metadata accounting can include segment footer
and metadata cache writes in the same query-wide threshold.
- Does this need documentation?
- [ ] No.
- [x] Yes. The new user-visible knobs should be documented in a
follow-up docs PR.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR should
merge into -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
