924060929 opened a new pull request, #66633:
URL: https://github.com/apache/doris/pull/66633
### What problem does this PR solve?
Issue Number: None
Related PR: #66407
Problem Summary:
FE naming caches and connector metadata caches currently use duplicate cache
wrappers and independently maintained invalidation dependencies. Each connector
must remember every sibling cache affected by catalog, database, table, or
partition invalidation. Concurrent loads, refresh, eviction callbacks, and
catalog close also need consistent publication and cleanup semantics across
these implementations.
This PR makes `fe-connector-cache` the shared metadata-cache runtime while
preserving one cache owner per catalog. It introduces declarative cache
definitions and hierarchical scopes, then migrates ADBC, Hive, HMS, Hudi,
Iceberg, MaxCompute, and Paimon caches to the shared framework. FE keeps a thin
adapter for Doris-specific naming, object, and `IdNameIndex` publication
semantics.
The shared runtime provides:
- catalog/database/table/partition scoped invalidation without
connector-maintained sibling-cache lists;
- state-identity publication fencing so loads started before invalidation
cannot publish stale values afterward;
- exact-key and bulk-load invalidation fencing;
- load deduplication, refresh, eviction cleanup, metrics, and catalog
lifecycle management;
- expected-value physical cleanup so delayed removal callbacks cannot remove
newer publications;
- per-catalog isolation without registry-wide publication locks on normal
cache operations.
The change also removes duplicate `CacheSpec`, `CacheFactory`, and legacy
connector-cache entry implementations. FE-specific cache publication retains
the lock order `names -> object -> IdNameIndex`, and validates identity before
mutating cache state.
Performance measurements on the final commit show approximately 85.8 ns/op
for the public `CatalogMetaCache -> MetaCache` hit path. Eight-thread
publication reached 2.85M ops/s for one cache and 2.51M ops/s for separate
caches in one registry. Invalidating and physically cleaning 120,000 entries
took approximately 32 ms on the local benchmark.
### Release note
None
### Check List (For Author)
- Test
- [ ] Regression test
- [x] Unit Test
- `fe-connector-cache`: 100 tests passed
- FE targeted metadata-cache tests: 93 tests passed
- Iceberg cache tests: 66 tests passed
- Full `./build.sh --fe` build passed, including Checkstyle
- [x] Manual test
- Re-profiled the final commit with CPU, allocation, and lock
async-profiler runs
- Ran hit, cold-load, eight-thread publication, logical
invalidation, and 120,000-entry cleanup benchmarks
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason
- Behavior changed:
- [x] No. This is an internal cache-framework refactor preserving
external metadata semantics.
- [ ] Yes.
- Does this need documentation?
- [x] No.
- [ ] Yes.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
