liaoxin01 opened a new pull request, #68991:
URL: https://github.com/apache/doris/pull/68991
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary:
The file cache stored whether a block is TTL protected as the block's cache
type
(`FileCacheType::TTL`), persisted in rocksdb and in the LRU dumps, and
`BlockFileCacheTtlMgr` tried to keep that stored state in step with each
tablet's
`file_cache_ttl_seconds`. The stored state went stale in every way it could:
- After a BE restart the TTL manager only knew the tablets registered since
startup. Blocks of
a tablet whose TTL was added by ALTER were never moved to the TTL queue,
and blocks of a tablet
whose TTL was removed stayed there for good.
- Moving blocks into the TTL queue was edge-triggered, so a TTL extended
after it had expired
never brought the blocks back.
- The TTL queue's LRU dump was restored with a made-up 3h deadline, and
could leave the same block
in two queues.
- Writers that captured the deadline before an ALTER (queued warm-up tasks,
for one) kept writing
blocks with the old deadline.
- One type cannot say both INDEX and TTL: an index block of a TTL tablet
lost its class in the
TTL queue and came back as NORMAL.
- Evict-in-advance also reserved space for the TTL queue, so whenever the
cache crossed the
watermark it LRU-evicted unexpired TTL blocks, even with the TTL queue
within its quota.
- The peer fill server cached the blocks it filled under tablet 0.
This PR stops storing protection and computes it instead, from what does not
change on a block
(its tablet and its class) and the tablet's current TTL:
1. **Protection is computed.** A BE-wide `TabletCachePolicyRegistry` holds
each tablet's
`(ttl_seconds, ctime)`. A block is protected while `ctime + ttl_seconds`
is in the future.
A tablet the registry does not know yet is judged by the deadline its
blocks were written with,
which is how every block was judged before.
2. **Where the policies come from.** `CloudTablet::sync_meta()` and tablet
load update the
registry on every refresh. It compares with what it holds and only takes
its write lock on a
change. Tablets that have blocks cached but are not loaded are fetched
from the meta service
by a background thread of the registry, at most
`file_cache_tablet_policy_refresh_batch` per
second. A failed fetch ends the round and puts that tablet off for a
minute. Each cache path
keeps a copy in its rocksdb, so a restart knows the policies before it
fetches them again.
3. **Class and queue are separate.** A block persists its class (INDEX,
NORMAL or DISPOSABLE);
TTL is only a queue. Which queue a block belongs in is computed when it
is written, when
eviction meets it (a protected block found in a class queue is skipped
and moved to the TTL
queue), and when it is loaded at restart. A background sweep moves blocks
whose protection
changed. It only runs when the registry reports that some tablet gained
or lost protection,
or that a deadline passed, plus a daily backstop.
4. **No more TTL dump.** The TTL queue is no longer dumped, and a dump left
by an older version
is removed at startup. `BlockFileCacheTtlMgr` and its three configs are
removed.
5. **Eviction.**
- Evict-in-advance only frees unprotected data; protected blocks are
evicted only when a write
needs their space.
- An eviction scan gives up on a queue after
`file_cache_evict_max_skipped_blocks_per_scan`
blocks it cannot evict.
- Protected bytes that still have to be evicted are counted in
`file_cache_ttl_protected_evicted_bytes`.
6. **Tablet ids.** Constructing a `CacheContext` from an `IOContext` now
requires the tablet id.
The peer fill server uses the tablet id the client sends.
7. **Diagnostics.**
- `information_schema.file_cache_info.TYPE` is computed the same way.
- New columns `PROTECTED` and `PROTECTED_UNTIL`; columns are only added.
New metrics:
- `file_cache_ttl_protected_evicted_bytes`
- `file_cache_evict_skipped_blocks`
- `file_cache_evict_scan_truncated`
- `file_cache_ttl_relocated_blocks`
- `file_cache_ttl_legacy_v2_protected_bytes`
- `file_cache_tablet_policy_num`
- `file_cache_tablet_policy_fetch_num` / `_failed_num`
Upgrade and downgrade:
- **Block rows.** rocksdb rows still record the deadline next to the class,
so an older BE reads
the blocks back and protects them as before. Rows an older BE wrote with
type TTL load as
NORMAL.
- **Policy copies.** They live in rocksdb's default column family under
their own key prefix.
An older BE opens that column family but never reads it.
- **Disk layout.** Unchanged; the version file stays the same.
- **Configs.** Removed configs left in be.conf are ignored with a warning.
### Release note
The file cache now decides TTL protection from each tablet's current
`file_cache_ttl_seconds`.
ALTER and TTL expiry take effect on cached data, including across BE
restarts, and index blocks
of TTL tables keep their class.
`information_schema.file_cache_info` has two new columns, `PROTECTED` and
`PROTECTED_UNTIL`.
BE configs:
- Removed: `file_cache_background_ttl_gc_interval_ms`,
`file_cache_background_ttl_info_update_interval_ms`,
`file_cache_background_tablet_id_flush_interval_ms`.
- Added: `file_cache_tablet_policy_refresh_interval_s`,
`file_cache_tablet_policy_refresh_batch`,
`file_cache_queue_sweep_interval_ms`,
`file_cache_queue_sweep_batch_blocks`,
`file_cache_queue_sweep_backstop_interval_s`,
`file_cache_ttl_relocate_batch_blocks`,
`file_cache_evict_max_skipped_blocks_per_scan`.
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [x] Regression test: TTL suites under `cloud_p0/cache/ttl` updated for
the removed
configs and metric. `ex06_ttl_restart_consistency` also covers a TTL
added right before a
restart, and checks the bytes in the TTL queue and the new columns.
Not run locally.
- [x] Unit Test: new `tablet_cache_policy_registry_test` and
`block_file_cache_test_ttl_policy`, plus updated file cache tests. All
file cache and cloud
tablet BE unit test suites pass locally: 427 passed, 3 skipped
(already disabled).
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [ ] No.
- [x] Yes.
- Protection follows the tablet's current TTL instead of what the
block stored.
- Evict-in-advance no longer evicts protected blocks.
- `information_schema.file_cache_info.TYPE` is computed.
- Disposable blocks are no longer offered for warm up while
protected.
- A disposable read of a TTL tablet no longer moves its blocks to
the hot end of the TTL
queue.
- Does this need documentation?
- [ ] No.
- [x] Yes. The new columns and configs need documenting; doc PR to
follow.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR should
merge into -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
