mnpoonia opened a new pull request, #8594:
URL: https://github.com/apache/hbase/pull/8594

   ## Summary
   
   `RegionInTransitionTracker` and `RegionStates` are two separate in-memory 
maps on the master that are not kept in sync with each other automatically. 
Several code paths deleted a region from `RegionStates` directly without 
notifying the tracker, so a deleted region could be left behind in the tracker 
forever — surfacing as a permanently stuck "STUCK Region-In-Transition" warning 
that only clears on a full master restart (`loadMeta()` rebuilds the tracker 
from `hbase:meta` from scratch; a GC'd region isn't in meta, so no stale entry 
survives that path).
   
   This was the root cause of a 9.8h stuck RIT incident: `GCRegionProcedure` 
deleted a no-longer-needed region from `RegionStates` but never told the 
tracker, leaving a stale entry that nothing but a restart could clear.
   
   **Fix:** add `AssignmentManager#deleteRegion(RegionInfo)` and its plural 
form `#deleteRegions(List<RegionInfo>)` as the single sync point that updates 
both maps together, and route every region-deletion call site through it 
instead of calling `RegionStates#deleteRegion(s)` directly.
   
   Two call sites are confirmed live bugs of the same shape and severity as the 
`GCRegionProcedure` case, since the region being deleted can still be in a 
non-terminal state in the tracker at the time of deletion:
   - `GCRegionProcedure` (the original incident)
   - `AssignmentManager#deleteTable`
   
   Three additional call sites with the same code pattern were audited and 
fixed defensively, even though the region being deleted there is already 
untracked (either already terminal/OFFLINE on a disabled table, or a 
non-default replica, which the tracker never tracks in the first place) — 
hardening against future regressions rather than fixes for currently-reachable 
bugs:
   - `AssignmentManagerUtil#removeNonDefaultReplicas` (split/merge)
   - `EnableTableProcedure` (replica-count-decrease cleanup)
   - `RestoreSnapshotProcedure#deleteRegionsFromInMemoryStates`
   
   ## Test plan
   
   - [x] `mvn -pl hbase-server -am compile` / `test-compile` clean
   - [x] New regression tests: 
`TestAssignmentManager#testDeleteRegionRemovesStaleRegionInTransitionEntry` and 
`#testDeleteRegionsRemovesStaleRegionInTransitionEntries`, both pass
   - [x] `TestAssignmentManager` (15 tests), `TestRegionInTransitionTracker` (4 
tests) — pass, no regressions
   - [x] `TestMergeTableRegionsProcedure` (8), `TestSplitTableRegionProcedure` 
(12), `TestEnableTableProcedure` (4), `TestRestoreSnapshotProcedure` (5) — 
pass, no regressions


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to