Abhishek Chennaka has uploaded this change for review. (
http://gerrit.cloudera.org:8080/24978
Change subject: WIP [tools] add 'kudu cluster check_location_maintenance'
......................................................................
WIP [tools] add 'kudu cluster check_location_maintenance'
Taking a whole location (rack, availability zone) offline is routine, but
nothing in the tooling answers the question an operator has to answer
first: would the cluster survive it? ksck reports current health and the
rebalancer enforces placement going forward, but neither one models the
hypothetical loss of every tablet server in a location.
Adds an action that does:
kudu cluster check_location_maintenance <master_addresses> [--location=L]
Three invariants must hold for a location to be safe to take offline:
1. The cluster has at least 3 locations. With fewer, some tablet has a
majority of its replicas in one location by construction.
2. Every table has replication_factor >= 3, since an RF<3 table cannot
survive the loss of a single replica.
3. For every tablet, the healthy voter replicas that would remain online
still form a majority of the table's replication factor.
The first two are cluster-wide, so they are evaluated once and reported at
the top. The third is per-location: with --location only that location is
checked, otherwise every location is checked and a verdict table is
printed. An unknown --location is an error naming the locations the cluster
does have, so a typo cannot read as a pass, and the action exits non-zero
if any location is unsafe so that it can gate a script.
Invariant 3 counts only voters, since non-voters do not vote in a Raft
election, and among those only the ones on healthy tablet servers, since a
voter that cannot be reached cannot be relied on to form the remaining
quorum. This is the same rule the rebalancer's placement policy enforces,
but the check does not reuse DetectPlacementPolicyViolations(), which skips
tablets that are not HEALTHY or RECOVERING. A tablet that is already
under-replicated is precisely the one most at risk from losing a location.
The check is a snapshot. It says the location is safe to lose now, not that
it will still be safe once the location has drained, and it does not look
at master replica placement.
TODO:
Create a Jira for the issue and add corresponding tests.
Change-Id: Ibd4cbaa8dab835d4d27270c06b4b0028aab8285b
Co-Authored-By: Claude Code <[email protected]>
---
M src/kudu/tools/CMakeLists.txt
A src/kudu/tools/location_maintenance.cc
A src/kudu/tools/location_maintenance.h
M src/kudu/tools/tool_action_cluster.cc
4 files changed, 386 insertions(+), 0 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/78/24978/1
--
To view, visit http://gerrit.cloudera.org:8080/24978
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: Ibd4cbaa8dab835d4d27270c06b4b0028aab8285b
Gerrit-Change-Number: 24978
Gerrit-PatchSet: 1
Gerrit-Owner: Abhishek Chennaka <[email protected]>
