deardeng opened a new pull request, #66984:
URL: https://github.com/apache/doris/pull/66984
`CloudReplica.primaryClusterToBackend` (`be`, previously `bes`) is persisted
per replica, but nothing ever removes an entry: `CloudSystemInfoService
.dropCluster()` only touches the system-info maps, `CloudTabletRebalancer
.loopCloudReplica()` only walks the compute groups that currently exist, and
`unprotectUpdateCloudReplica()` only puts. So every created/dropped compute
group leaves one key behind on every replica, forever.
On an instance that had repeatedly added and removed compute groups this
grew to 392 route keys per replica against 3 live compute groups: 45.5M entries
over 167k replicas, 1.78 GB of the 2.07 GB image (85.9%) and ~8.6 GB of live
heap. Metadata checkpoint keeps the online catalog and the checkpoint catalog
resident at the same time, which no longer fit in the heap, so checkpoint OOMed
on every leader in turn and the image never advanced.
Drop an entry once its backend id no longer resolves. Such an entry is never
routed to: `getBackendIdImpl()` already resolves the id, gets null and falls
back to `hashReplicaToBe()`, so it only costs memory and image bytes. The
predicate is derived from the backend set, which is itself replicated through
the edit log, so every FE reaches the same result without a new edit log op.
A dead primary is kept while its compute group still has a live secondary.
With `enable_immediate_be_assign=false` -- the default -- primary pointing at
an unavailable BE while the secondary holds the rehashed live one is the normal
failover state, and `FrontendServiceImpl.getTabletReplicaInfos()` reaches
secondaries through `getPrimaryComputeGroupIds()`, so dropping the primary key
would hide a live secondary BE from the lazy fetch peer cache candidates. The
secondary map is cleaned first, so a dead pair is still fully removed.
Two call sites:
- `gsonPostProcess()`, outside the `bes` migration branch, so both formats
are covered. The backends module is loaded before db/recycleBin, and the
checkpoint thread resolves `Env.getCurrentEnv()` to its own Env, so the backend
set read there belongs to the image being loaded. This shrinks the load peak
and makes existing oversized images converge.
- The existing per-replica callback in `completeRouteInfo()`, so a running
leader converges too instead of waiting for a restart. It is pinned to one
compute group per round because `loopCloudReplica()` invokes the callback once
per (replica, compute group) while holding `table.readLock()`.
`secondaryClusterToBackends` is not persisted, but leaks the same way in
heap, so it is cleaned by the same predicate.
Guarded by `enable_cloud_replica_stale_route_clean` (default true).
### What problem does this PR solve?
Issue Number: close #xxx
Related PR: #xxx
Problem Summary:
### Release note
None
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [x] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [x] No.
- [ ] Yes. <!-- Explain the behavior change -->
- Does this need documentation?
- [x] No.
- [ ] Yes. <!-- Add document PR link here. eg:
https://github.com/apache/doris-website/pull/1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR should
merge into -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
