bobhan1 opened a new pull request, #65658:
URL: https://github.com/apache/doris/pull/65658
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary:
On a file-cache miss, `CachedRemoteFileReader` currently performs two
different kinds of work on the query thread:
1. read bytes from remote storage to satisfy the query;
2. append and finalize those bytes into the local file cache for future
queries.
The first operation is on the critical path. The second is best-effort cache
population, but local filesystem latency and backpressure are still charged to
foreground scan latency because the two operations are coupled.
This PR implements phase 1 of asynchronous file-cache writes. For ordinary
reads, the query thread returns after the caller's buffer is complete and hands
eligible cache-miss blocks to per-disk background workers. The feature is
opt-in and disabled by default. Explicit cache-population operations, including
warm-up and prefetch/download paths, retain synchronous completion semantics.
Simply moving `append()` to another thread is unsafe. The design also has to
preserve cache-block ownership, avoid duplicate queued writes, survive
clear/remove races, bound background work, and keep existing warm-up and
dry-run behavior. This PR introduces dedicated components for those
responsibilities instead of transferring existing `FileBlock` objects across
threads.
#### Goals
- Remove best-effort local cache persistence from the ordinary remote-read
critical path.
- Reuse remote bytes that have already been fetched but are still waiting
for local persistence.
- Avoid duplicate background writes for the same cache block.
- Keep queued work bounded and make overload fall back to a cache miss
rather than block a query.
- Prevent stale tasks from recreating cache data after clear/remove
operations.
- Preserve the existing synchronous behavior for explicit cache-population
callers.
- Provide query-profile and per-cache-disk observability for the new path.
#### Non-goals of this phase
- This is not remote-read singleflight. Two cold readers that overlap before
either publishes its completed remote buffer can still issue separate remote
reads.
- This does not change local file-cache durability guarantees; persistence
remains best effort.
- This does not make warm-up or prefetch completion asynchronous.
- This does not enable the feature by default.
### Architecture
The query thread owns read planning and result assembly. Background workers
own cache-cell acquisition and persistence. The two sides exchange immutable,
reference-counted buffers through an inflight index and a bounded per-disk
service.
```mermaid
flowchart LR
Caller[Scan / query caller]
Reader[CachedRemoteFileReader]
Remote[Remote FileReader]
subgraph CacheDisk[One file-cache disk]
Probe[BlockFileCache read-only probe]
Index[InflightWriteBufferIndex]
Service[AsyncCacheWriteService]
Queue[Bounded MPMC queue]
Workers[Resizable write workers]
Storage[BlockFileCache and local storage]
end
Metrics[Runtime profile and per-disk bvars]
Caller -->|read request| Reader
Reader -->|lookup completed remote buffers| Index
Reader -->|observe cache state without ownership| Probe
Reader -->|one remaining middle-range read| Remote
Remote -->|aligned bytes| Reader
Reader -->|complete caller buffer| Caller
Reader -->|publish true MISS buffers| Index
Reader -->|non-blocking submit| Service
Service --> Queue
Queue --> Workers
Workers -->|revalidate, acquire, append, finalize| Storage
Workers -->|conditional cleanup| Index
Storage -.->|clear/remove advances write epoch| Service
Reader -.-> Metrics
Index -.-> Metrics
Service -.-> Metrics
```
#### Component responsibilities
| Component | Responsibility | Important guarantee |
| --- | --- | --- |
| `CachedRemoteFileReader` | Resolves synchronous versus asynchronous mode
for every read, combines inflight and local-cache coverage, performs the
required remote read, fills the caller buffer, and submits only true misses. |
The caller never observes an incomplete buffer. Cache-write submission is not
allowed to turn a successful remote read into a query failure. |
| `BlockFileCache::probe` | Reports downloaded, downloading, empty,
deleting, and missing ranges without creating cache cells or taking downloader
ownership. | Planning does not mutate cache state or transfer thread-affine
`FileBlock` downloader ownership. LRU touch happens only after a block is
actually consumed. |
| `InflightWriteBufferIndex` | Stores shard-protected references to
completed remote buffers that are pending persistence, keyed by cache key and
block offset and tagged with a write epoch. | Insert-if-absent selects one
background-write owner. Conditional removal prevents an old completion callback
from removing a newer entry. |
| `AsyncCacheWriteService` | Owns the per-disk bounded MPMC queue, pending
count, tracked-buffer accounting, resizable workers, task-age checks, shutdown,
and service-level metrics. | Submission is non-blocking. Backpressure rejects
work and rolls back its inflight entry instead of blocking the query thread. |
| Async write worker | Rechecks task age and epoch, calls the normal cache
admission/cell-creation path, skips blocks already downloaded or owned by
another downloader, and appends/finalizes only eligible empty blocks. |
Background work follows current cache state; it does not blindly overwrite or
resurrect blocks. |
| Cache write epoch | Represents the generation of work accepted by a cache
instance. Clear/remove paths advance it. | A task accepted under an older
generation is dropped before or during persistence. |
| Profile and bvar integration | Records query-local
submission/rejection/inflight reuse and per-disk queue, memory, latency, skip,
failure, and stale-task information. | Operators can distinguish remote-read
latency from asynchronous cache-write health and pressure. |
### Important flows
#### 1. Ordinary read with mixed coverage
Coverage is classified in this order:
1. completed remote buffers in `InflightWriteBufferIndex`;
2. existing cache blocks through the read-only probe;
3. remaining blocks as misses.
The reader consumes the maximum covered prefix and suffix. If an uncovered
region remains, everything between those boundaries is fetched with one aligned
remote read. This deliberately avoids fragmented remote requests and repeated
waits/local reads inside the middle span, even if that means rereading a cached
block located between two misses.
```mermaid
sequenceDiagram
participant C as Caller
participant R as CachedRemoteFileReader
participant I as Inflight index
participant F as BlockFileCache
participant O as Remote storage
participant S as Async write service
participant W as Worker
C->>R: read_at(offset, size)
R->>I: lookup aligned blocks for current epoch
I-->>R: reusable pending buffers
R->>F: read-only probe for uncovered runs
F-->>R: downloaded / downloading / miss coverage
R->>R: materialize covered prefix and suffix
alt all requested bytes are covered
R-->>C: return completed caller buffer
else a middle range remains
R->>O: one aligned middle-range read
O-->>R: remote bytes
R->>R: copy requested overlap into caller buffer
loop each true MISS block in the remote span
R->>I: insert completed block buffer if absent
alt another owner already published it
I-->>R: existing entry; skip duplicate write
else this reader owns persistence
R->>S: try_submit without waiting
alt queue accepts task
S-->>R: accepted
else queue or shutdown rejects task
S-->>R: rejected
R->>I: conditional rollback
end
end
end
R-->>C: return completed caller buffer
S->>W: dequeue in background
W->>F: revalidate epoch and current block state
W->>F: append and finalize eligible empty blocks
W->>I: conditional completion cleanup
end
```
#### 2. Later reader reuses a pending buffer
After the first remote read publishes a buffer and before its worker
finishes persistence, a later reader can copy the requested bytes directly from
the inflight entry. It does not wait for the local file to be finalized and
does not submit another write task for that block.
This index deduplicates pending cache writes and bridges the interval
between remote-read completion and local persistence. It intentionally does not
claim to deduplicate remote requests that began before publication.
#### 3. Backpressure and asynchronous failure
The per-disk pending-task limit bounds queued work. Buffer allocation, queue
submission, or shutdown can reject a task. In that case the reader
conditionally removes its own inflight entry and records the rejection, but
still returns the already-read data to the caller.
Likewise, an append/finalize failure is logged and counted by the worker; it
does not retroactively fail a query whose remote read succeeded. A future read
simply observes another cache miss and may retry population.
#### 4. Clear/remove and stale-task handling
Cache invalidation advances the service's write epoch. Both inflight
lookup/replacement and worker processing compare their entry or task against
the current epoch. Workers check the epoch before acquiring cache blocks and
again while processing them. This closes the window where old queued work could
recreate blocks after a clear/remove operation.
Completion cleanup uses pointer-conditional removal. Therefore, even if a
newer generation has published an entry for the same block offset, an older
task cannot erase it.
#### 5. Mode selection and compatibility
`CacheWriteMode` is resolved for every read so an online configuration
switch affects existing readers. Ordinary reads may select the asynchronous
path when the feature is enabled. Dry-run, warm-up, peer-cache, segment-index
population, downloader, and explicit prefetch paths select or override
synchronous mode because their completion semantics depend on the cache being
populated before returning.
The default remains synchronous because `enable_async_file_cache_write`
defaults to `false`.
#### 6. Worker lifecycle
Each `BlockFileCache` creates its index and service during initialization.
Worker count can be resized online. Shutdown first stops new submitters, waits
for already registered submitters, drains accepted tasks, joins workers, and
only then destroys state referenced by task-finalization callbacks.
### Configuration
| Configuration | Default | Purpose |
| --- | ---: | --- |
| `enable_async_file_cache_write` | `false` | Enables the ordinary
asynchronous write path. |
| `async_file_cache_write_workers_per_disk` | `2` | Number of persistence
workers for each cache disk; supports online resize. |
| `async_file_cache_write_max_pending_tasks_per_disk` | `256` | Bounds
accepted and queued tasks per disk. |
| `async_file_cache_write_batch_size` | `16` | Maximum tasks a worker
handles per dequeue loop. |
| `async_file_cache_write_watchdog_warn_secs` | `30` | Warns when a task
waited too long. |
| `async_file_cache_write_watchdog_drop_secs` | `120` | Drops excessively
old best-effort work. |
| `enable_inflight_write_buffer_index` | `true` | Enables pending-buffer
reuse and duplicate-write suppression. |
| `inflight_write_buffer_index_shard_count` | `64` | Controls index lock
sharding at cache initialization. |
### Observability
Query runtime profiles expose asynchronous submissions, rejections,
buffer-allocation failures, stale-epoch drops, and inflight hits/misses.
Per-cache-disk bvars additionally expose queue depth, tracked buffer bytes,
write latency, task-age timeouts, append failures, state-based skips, index
size/memory, conditional-removal results, and backpressure rollbacks.
### Tests
- BE ASAN targeted unit tests: **26/26 passed** across four suites.
- read-only probe behavior;
- inflight insert/lookup/epoch replacement and concurrent publication;
- queue limits, shutdown, worker resize, watchdog, epoch invalidation,
multi-cell writes, and partial overlap;
- mixed cache coverage, one-middle-read behavior, pending-buffer reuse,
backpressure rollback, and per-read mode resolution;
- runtime-profile delta aggregation.
- Docker cloud regression: **1/1 suite passed**.
- a cold query submits asynchronous cache writes;
- after the queue drains, a second query observes probe hits;
- switching the feature off restores synchronous behavior without
asynchronous submissions.
- Build and style:
- `./build.sh --be --fe --cloud -j100`
- `./build.sh --be -j100`
- `build-support/check-format.sh`
- clang-tidy was intentionally not run for this change.
### Release note
Add an opt-in asynchronous file-cache write path controlled by
`enable_async_file_cache_write`. It is disabled by default.
### Check List (For Author)
- Test
- [x] Regression test
- [x] Unit Test
- [x] Manual test (Docker cloud regression and runtime configuration
switch)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason
- Behavior changed:
- [ ] No.
- [x] Yes. When explicitly enabled, ordinary file-cache misses persist
eligible cache blocks through background workers. Default behavior and explicit
cache-population completion remain synchronous.
- Does this need documentation?
- [x] No. The feature is experimental and disabled by default.
- [ ] Yes.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
