Nirdosh Kumar Yadav created HBASE-30334:
-------------------------------------------
Summary: MergeTableRegionsProcedure.rollbackState() is a no-op for
MERGE_TABLE_REGIONS_CHECK_CLOSED_REGIONS; when the gate fails the parents stay
CLOSED, producing an indefinite stuck RIT
Key: HBASE-30334
URL: https://issues.apache.org/jira/browse/HBASE-30334
Project: HBase
Issue Type: Bug
Components: master
Affects Versions: 2.5.10, 2.6.1
Reporter: Nirdosh Kumar Yadav
Assignee: Nirdosh Kumar Yadav
h4. {color:#172b4d}Problem Statement{color}
{color:#172b4d}When MergeTableRegionsProcedure fails at the
MERGE_TABLE_REGIONS_CHECK_CLOSED_REGIONS state — e.g. because
AssignmentManagerUtil.checkClosedRegion() throws "Recovered.edits are found in
Region: … abort split/merge to prevent data loss" — the procedure framework
unwinds through rollbackState(). The MERGE_TABLE_REGIONS_CHECK_CLOSED_REGIONS
case is a bare break;, so the parents that were CLOSED during the prior forward
step MERGE_TABLE_REGIONS_CLOSE_REGIONS are never re-opened. Both parent regions
remain OFFLINE and the RIT can persist indefinitely until an operator
intervenes.{color}
h4. Root Cause
MergeTableRegionsProcedure.rollbackState() at
MergeTableRegionsProcedure.java:287-288:
case MERGE_TABLE_REGIONS_CHECK_CLOSED_REGIONS: break; // no-op — parents stay
OFFLINE
The symmetric case in SplitTableRegionProcedure.rollbackState() at
SplitTableRegionProcedure.java:386-388 handles this correctly:
case SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS: openParentRegion(env); // calls
reopenRegionsForRollback break;
openParentRegion() (SplitTableRegionProcedure.java:644-648) delegates to the
shared helper AssignmentManagerUtil.reopenRegionsForRollback()
(AssignmentManagerUtil.java:283-290). The merge path is missing this call.
h4. Key Observations
* MergeTableRegionsProcedure pid=46990253 tripped checkClosedRegion at
18:32:33.293 UTC on parent 112d9f08181e80aaae01e68c9229c3cb.
* Rollback ran but did not re-open either parent (112d9f08… and 9e93ca76…),
leaving both OFFLINE.
* The parents held an xlock preventing any fresh ASSIGN procedures from making
progress.
* Stuck RIT lasted 48 minutes (18:32:33 → 19:21:29) on the OpenTSDB critical
tier; only operator intervention (fresh ASSIGN procedures pid=47001645/47001646
after HMaster failover) cleared it.
h4. Proposed Solution
{code:java}
/hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/MergeTableRegionsProcedure.java
@@ line 287 case MERGE_TABLE_REGIONS_CHECK_CLOSED_REGIONS:
openParentRegions(env);
break; {code}
The helper AssignmentManagerUtil.reopenRegionsForRollback() is already
exercised by SplitTableRegionProcedure under production load, so risk is
minimal.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
