jihuayu opened a new issue, #398:
URL: https://github.com/apache/kvrocks-controller/issues/398
## Background
The controller's current health decision path is still based on a single
active leader's local memory. Standby controller instances do not contribute
probe observations, and failure state is lost when the leader process restarts
or leadership changes.
## Current behavior
- Only the elected leader starts cluster checkers. When a controller loses
leadership, it closes all local `ClusterChecker` instances.
- `controller/controller.go`: `resume`, `suspend`, `syncLoop`
- Probe failure counts are stored only in `ClusterChecker.failureCounts
map[string]int64`.
- `controller/cluster.go`: `failureCounts`, `increaseFailureCount`,
`resetFailureCount`
- When the local failure count reaches `maxFailureCount`, the leader
directly calls `PromoteNewMaster` and writes the updated cluster topology.
- `controller/cluster.go`: `increaseFailureCount` ->
`cluster.PromoteNewMaster` -> `clusterStore.UpdateCluster`
- The store interface has no persisted probe report, health decision, CAS,
fencing token, or per-decision lease abstraction.
- `store/store.go`: `Store`, `UpdateCluster`, `SetCluster`
## Why this is risky
- A controller restart or leader change clears all accumulated failure
observations.
- A network issue visible only from the current leader can become an
automatic failover decision.
- There is no durable audit trail showing which controller instances
observed the failure and when.
- There is no quorum-style distinction between a local subjective failure
observation and an objective cluster decision.
- The final topology update is not tied to an expected cluster version plus
a decision lease/fencing token.
## Expected direction
Introduce a persisted, multi-instance health decision layer before automatic
failover:
1. Add durable `HealthReport` records keyed by namespace, cluster, shard,
node, and controller instance ID.
2. Include at least: `observed_at`, `expires_at` or TTL, probe result, error
class, observed cluster version/epoch, role, sequence, and controller ID.
3. Add a `HealthDecision` or `FailoverDecision` object that represents the
computed objective state, for example `Healthy`, `Suspect`, `ObjectivelyDown`,
`FailoverPending`, `FailoverCommitted`, `Blocked`.
4. Require quorum or configurable agreement before automatic master failover.
5. Commit failover only with a store-level CAS/transaction or lease/fencing
token against the expected cluster version.
6. Keep the behavior idempotent across controller restart and leader
transfer.
## Acceptance criteria
- Failure observations survive controller leader changes until their TTL
expires.
- Multiple controller instances can report independent health observations
while only one instance commits the final decision.
- Automatic master failover requires a persisted objective decision, not
only one leader's in-memory counter.
- Stale leaders or stale decisions cannot overwrite a newer cluster topology.
- Tests cover leader restart, leader handoff, asymmetric probe failure,
stale decision rejection, and successful quorum-backed failover.
## Related follow-up areas
- Add health/failover Prometheus metrics.
- Expose health decision state in API/UI.
- Define conflict rules with manual failover, node offline/online, and slot
migration.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
