[
https://issues.apache.org/jira/browse/HBASE-30350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18110132#comment-18110132
]
Umesh Kumar Kumawat commented on HBASE-30350:
---------------------------------------------
[https://github.com/apache/hbase/pull/8594#issuecomment-5488088420]
{quote}We only delete an region once it is closed. I can think of 3 scenario of
region close- 1. Table disabled 2. Merge parents 3. Split parent . In all 3
cases, regions should be out of Region-In-trnasition tracker when region was
getting closed so we should never need to handle it while deleteing.
I think bug here, that we need to find out, is why it was not removed from
tracker while closing. I am opposed to the change in tracker when actually
there is no state change.
{quote}
> Stale entry left in RegionInTransitionTracker after GCRegionProcedure purges
> region metadata
> --------------------------------------------------------------------------------------------
>
> Key: HBASE-30350
> URL: https://issues.apache.org/jira/browse/HBASE-30350
> Project: HBase
> Issue Type: Bug
> Components: master, Region Assignment
> Affects Versions: 3.0.0, 4.0.0-alpha-1, 2.5.16, 2.6.7
> Reporter: Aman Poonia
> Assignee: Aman Poonia
> Priority: Major
> Labels: pull-request-available
>
> GCRegionProcedure (state {{GC_REGION_PURGE_METADATA}}) removes a GC'd
> split/merged parent region from {{AssignmentManager#regionStates}} but never
> removes the corresponding entry from the separate
> {{RegionInTransitionTracker}} map. These two maps are not kept in sync
> automatically.
> The analogous cleanup path, {{AssignmentManager#markRegionAsMerged}},
> correctly calls both {{regionStates.deleteRegion(ri)}} and
> {{regionInTransitionTracker.handleRegionDelete(ri)}} for each merge
> parent. {{GCRegionProcedure}} only calls the former.
> As a result, once a region is GC'd via {{CatalogJanitor}} ->
> {{GCRegionProcedure}}, its ghost entry remains in
> {{RegionInTransitionTracker}} indefinitely. The periodic "STUCK
> Region-In-Transition"
> chore ({{ProcExecTimeout}} thread) keeps re-reporting this already-deleted
> region as stuck on every run, producing misleading WARN logs and inflating
> RIT-age metrics/alerts. The stale entry is only
> cleared when the active master fails over, since
> {{AssignmentManager#joinCluster}} -> {{loadMeta}} rebuilds all in-memory
> assignment state fresh from {{hbase:meta}}.
> *Observed in production:* a region was GC'd (fully removed from
> {{hbase:meta}} and HDFS) but continued to be logged as "STUCK
> Region-In-Transition" for ~9 hours until the next master failover
> silently cleared it, giving a false impression of an actual 9-hour stuck
> RIT / unavailability.
> *Fix:* add a shared {{AssignmentManager#deleteRegion(RegionInfo)}} method
> that removes the region from both {{regionStates}} and
> {{regionInTransitionTracker}} together, and have both
> {{markRegionAsMerged}} and {{GCRegionProcedure}} call it instead of
> touching {{regionStates}} directly.
> Confirmed present on {{apache/master}} as of commit fb4286d3046.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
