bobhan1 opened a new pull request, #65058:
URL: https://github.com/apache/doris/pull/65058
### What problem does this PR solve?
Issue Number: None
Related PR: selectdb/selectdb-core#9151, selectdb/selectdb-core#9531,
selectdb/selectdb-core#9450
Problem Summary:
This PR upstreams a stacked set of cloud file-cache write-control changes.
It adds index-only file-cache write support and lazy-materialization write
control, then adds query-wide remote-scan cache write limiting through
`remote_scan_no_write_file_cache_threshold_bytes`. When a remote scan exceeds
the configured threshold, cache misses continue reading remotely without
writing more data into file cache.
The limiter is propagated through query context, cached remote reads,
inverted-index reads, file-cache stats/profile reporting, and cloud regression
coverage.
During local validation, the remote-scan Docker suite initially failed
because `clear&sync=true` can leave direct cache-file readers holding recently
cleared blocks, making a "cold" read hit stale local cache. The suite now
disables `enable_read_cache_file_directly` so it tests the remote-scan limiter
on the normal file-cache path deterministically, and it asserts the clear API
status.
### Release note
Add file-cache write controls for index-only writes and remote scans,
including `remote_scan_no_write_file_cache_threshold_bytes`.
### Check List (For Author)
- Test:
- `build-support/clang-format.sh`
- `git diff --check`
- `./run-be-ut.sh --run
--filter=BlockFileCacheTest.file_cache_profile_remote_only_on_miss_state_counters:BlockFileCacheTest.remote_scan_cache_write_limiter_strict_budget:BlockFileCacheTest.remote_scan_cache_write_limiter_threshold_zero_and_negative:BlockFileCacheTest.remote_scan_cache_write_limiter_concurrent_budget:BlockFileCacheTest.get_or_set_remote_scan_cache_write_limiter_admission:BlockFileCacheTest.get_or_set_remote_scan_cache_write_limiter_is_query_wide_for_index:BlockFileCacheTest.cached_remote_file_reader_policy_remote_only_with_scan_limiter:BlockFileCacheTest.cached_remote_file_reader_remote_scan_cache_write_limiter:DorisFSDirectoryTest.FSIndexInputSetIoContextPropagatesQueryLimiter
-j100`
- `./run-fe-ut.sh --run org.apache.doris.qe.SessionVariablesTest`
- `./build.sh --be --fe --cloud -j100`
- `env -u HTTP_PROXY -u HTTPS_PROXY -u http_proxy -u https_proxy -u
ALL_PROXY -u all_proxy ./run-regression-test.sh --run -f
regression-test/suites/cloud_p0/cache/remote_scan_no_write_file_cache/test_remote_scan_no_write_file_cache_threshold.groovy
-g docker -runMode=cloud -dockerSuiteParallel 1`
- Behavior changed: Yes. Remote scans can stop writing cache after the
configured per-query threshold while continuing to read remote data.
- Does this need documentation: Yes. The new session variable should be
documented in a follow-up docs PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
