[
https://issues.apache.org/jira/browse/HBASE-30389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
huginn updated HBASE-30389:
---------------------------
Description:
After an active master failover, AssignmentManager can retain a stale region
server location for a CLOSED or otherwise non-online region, even though the
region is not assigned to that server.
When the active master rebuilds assignment state from hbase:meta, stale
location data can be loaded into RegionStateNode. Because only OPEN, OPENING,
CLOSING, SPLITTING, and MERGING states are added to a ServerStateNode, the
current location and per-server assignment state can become inconsistent.
The fix is to normalize the loaded region location to null for states without
current-host semantics while preserving lastHost for locality and recovery.
This keeps locations for online and transitioning states unchanged and adds
regression coverage for CLOSED, OPEN, implicit OFFLINE, and split-recovery
state loading.
was:
h2. What happens
After an active master failover, a region in a non-online state can retain an
old region server in the new master's in-memory region location.
h2. When it happens
This can occur when the active master rebuilds assignment state from hbase:meta
and the row still contains stale location columns for a region whose persisted
state is not one of OPEN, OPENING, CLOSING, SPLITTING, or MERGING.
h2. Impact
The new master can expose stale region ownership that is inconsistent with its
per-server assignment state, which can affect assignment-related decisions and
status reporting.
h2. Root cause
AssignmentManager.RegionMetaLoadingVisitor.visitRegionState unconditionally
copies the location read from hbase:meta into RegionStateNode, but only
current-host states are added to a ServerStateNode. For CLOSED and other
non-online states, a stale location can therefore remain in memory even though
the region is not assigned to that server.
See [AssignmentManager.java on community
master|https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/AssignmentManager.java#L1865-L1889].
h2. Proposed fix
Normalize the loaded location to null for states that do not have current-host
semantics, while preserving lastHost for locality and recovery. Keep locations
for online and transitioning states unchanged, and add regression coverage for
master failover and state loading.
h2. Reproduction / Testing
Testing evidence will be added by the reporter.
> Normalize region locations when loading state after master failover
> -------------------------------------------------------------------
>
> Key: HBASE-30389
> URL: https://issues.apache.org/jira/browse/HBASE-30389
> Project: HBase
> Issue Type: Bug
> Components: master
> Affects Versions: 2.4.11
> Reporter: huginn
> Assignee: huginn
> Priority: Major
> Labels: pull-request-available
>
> After an active master failover, AssignmentManager can retain a stale region
> server location for a CLOSED or otherwise non-online region, even though the
> region is not assigned to that server.
> When the active master rebuilds assignment state from hbase:meta, stale
> location data can be loaded into RegionStateNode. Because only OPEN, OPENING,
> CLOSING, SPLITTING, and MERGING states are added to a ServerStateNode, the
> current location and per-server assignment state can become inconsistent.
> The fix is to normalize the loaded region location to null for states without
> current-host semantics while preserving lastHost for locality and recovery.
> This keeps locations for online and transitioning states unchanged and adds
> regression coverage for CLOSED, OPEN, implicit OFFLINE, and split-recovery
> state loading.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
