[
https://issues.apache.org/jira/browse/HBASE-30334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nirdosh Kumar Yadav updated HBASE-30334:
----------------------------------------
Description:
h4. {color:#172b4d}Problem Statement{color}
{color:#172b4d}When MergeTableRegionsProcedure fails at the
MERGE_TABLE_REGIONS_CHECK_CLOSED_REGIONS state — e.g. because
AssignmentManagerUtil.checkClosedRegion() throws "Recovered.edits are found in
Region: … abort split/merge to prevent data loss" — the procedure framework
unwinds through rollbackState(). The MERGE_TABLE_REGIONS_CHECK_CLOSED_REGIONS
case is a bare break;, so the parents that were CLOSED during the prior forward
step MERGE_TABLE_REGIONS_CLOSE_REGIONS are never re-opened. Both parent regions
remain OFFLINE and the RIT can persist indefinitely until an operator
intervenes.{color}
{color:#172b4d}*What happened:*{color} * {color:#172b4d}Region {{112d9f08}} was
gracefully moved from *rs-132 → rs-81* at {{{}16:36:34 UTC{}}}.{color}
* {color:#172b4d}The close on rs-132 had already flushed the region's edits,
and rs-81 opened the region with {{{}openSeqNum=4997750282{}}}, establishing
that the edits up to that point were durable.{color}
* {color:#172b4d}~25 seconds later, rs-132 was declared dead as part of the
broader graceful RS drain.{color}
* {color:#172b4d}The WAL split worker (rs-34) subsequently created a
{{recovered.edits}} file for this region containing edit
{{{}seqId=4997750280{}}}.{color}
* {color:#172b4d}This edit was {*}already durable in the HFiles{*}. The
recovered edit file was effectively stale/orphaned, but its existence was not
recognized as harmless.{color}
{color:#172b4d}*Where it got stuck:*{color} * {color:#172b4d}At {{{}18:32:33
UTC{}}}, {{MergeTableRegionsProcedure (pid=46990253)}} unassigned the region
successfully.{color}
* {color:#172b4d}During {{{}MERGE_TABLE_REGIONS_CHECK_CLOSED_REGIONS{}}}, the
procedure saw the {{recovered.edits}} file and failed the check.{color}
* {color:#172b4d}The merge procedure retained the region lock and did not
recover automatically, leaving the region in {*}CLOSED/RIT for ~48m
56s{*}.{color}
* {color:#172b4d}The RIT only cleared after the HMaster failover caused the
procedure to replay and fresh top-level ASSIGN procedures reopened the
regions.{color}
h4. *Root Cause*
MergeTableRegionsProcedure.rollbackState() at
MergeTableRegionsProcedure.java:287-288:
case MERGE_TABLE_REGIONS_CHECK_CLOSED_REGIONS: break; // no-op — parents stay
OFFLINE
The symmetric case in SplitTableRegionProcedure.rollbackState() at
SplitTableRegionProcedure.java:386-388 handles this correctly:
case SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS: openParentRegion(env); // calls
reopenRegionsForRollback break;
openParentRegion() (SplitTableRegionProcedure.java:644-648) delegates to the
shared helper AssignmentManagerUtil.reopenRegionsForRollback()
(AssignmentManagerUtil.java:283-290). The merge path is missing this call.
h4. Key Observations
* MergeTableRegionsProcedure pid=46990253 tripped checkClosedRegion at
18:32:33.293 UTC on parent 112d9f08181e80aaae01e68c9229c3cb.
* Rollback ran but did not re-open either parent (112d9f08… and 9e93ca76…),
leaving both OFFLINE.
* The parents held an xlock preventing any fresh ASSIGN procedures from making
progress.
* Stuck RIT lasted 48 minutes (18:32:33 → 19:21:29) on the OpenTSDB critical
tier; only operator intervention (fresh ASSIGN procedures pid=47001645/47001646
after HMaster failover) cleared it.
h4. Proposed Solution
{code:java}
/hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/MergeTableRegionsProcedure.java
@@ line 287 case MERGE_TABLE_REGIONS_CHECK_CLOSED_REGIONS:
openParentRegions(env);
break; {code}
The helper AssignmentManagerUtil.reopenRegionsForRollback() is already
exercised by SplitTableRegionProcedure under production load, so risk is
minimal.
was:
h4. {color:#172b4d}Problem Statement{color}
{color:#172b4d}When MergeTableRegionsProcedure fails at the
MERGE_TABLE_REGIONS_CHECK_CLOSED_REGIONS state — e.g. because
AssignmentManagerUtil.checkClosedRegion() throws "Recovered.edits are found in
Region: … abort split/merge to prevent data loss" — the procedure framework
unwinds through rollbackState(). The MERGE_TABLE_REGIONS_CHECK_CLOSED_REGIONS
case is a bare break;, so the parents that were CLOSED during the prior forward
step MERGE_TABLE_REGIONS_CLOSE_REGIONS are never re-opened. Both parent regions
remain OFFLINE and the RIT can persist indefinitely until an operator
intervenes.{color}
h4. Root Cause
MergeTableRegionsProcedure.rollbackState() at
MergeTableRegionsProcedure.java:287-288:
case MERGE_TABLE_REGIONS_CHECK_CLOSED_REGIONS: break; // no-op — parents stay
OFFLINE
The symmetric case in SplitTableRegionProcedure.rollbackState() at
SplitTableRegionProcedure.java:386-388 handles this correctly:
case SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS: openParentRegion(env); // calls
reopenRegionsForRollback break;
openParentRegion() (SplitTableRegionProcedure.java:644-648) delegates to the
shared helper AssignmentManagerUtil.reopenRegionsForRollback()
(AssignmentManagerUtil.java:283-290). The merge path is missing this call.
h4. Key Observations
* MergeTableRegionsProcedure pid=46990253 tripped checkClosedRegion at
18:32:33.293 UTC on parent 112d9f08181e80aaae01e68c9229c3cb.
* Rollback ran but did not re-open either parent (112d9f08… and 9e93ca76…),
leaving both OFFLINE.
* The parents held an xlock preventing any fresh ASSIGN procedures from making
progress.
* Stuck RIT lasted 48 minutes (18:32:33 → 19:21:29) on the OpenTSDB critical
tier; only operator intervention (fresh ASSIGN procedures pid=47001645/47001646
after HMaster failover) cleared it.
h4. Proposed Solution
{code:java}
/hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/MergeTableRegionsProcedure.java
@@ line 287 case MERGE_TABLE_REGIONS_CHECK_CLOSED_REGIONS:
openParentRegions(env);
break; {code}
The helper AssignmentManagerUtil.reopenRegionsForRollback() is already
exercised by SplitTableRegionProcedure under production load, so risk is
minimal.
> MergeTableRegionsProcedure.rollbackState() is a no-op for
> MERGE_TABLE_REGIONS_CHECK_CLOSED_REGIONS; when the gate fails the parents
> stay CLOSED, producing an indefinite stuck RIT
> ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-30334
> URL: https://issues.apache.org/jira/browse/HBASE-30334
> Project: HBase
> Issue Type: Bug
> Components: master
> Affects Versions: 2.6.1, 2.5.10
> Reporter: Nirdosh Kumar Yadav
> Assignee: Nirdosh Kumar Yadav
> Priority: Minor
>
> h4. {color:#172b4d}Problem Statement{color}
> {color:#172b4d}When MergeTableRegionsProcedure fails at the
> MERGE_TABLE_REGIONS_CHECK_CLOSED_REGIONS state — e.g. because
> AssignmentManagerUtil.checkClosedRegion() throws "Recovered.edits are found
> in Region: … abort split/merge to prevent data loss" — the procedure
> framework unwinds through rollbackState(). The
> MERGE_TABLE_REGIONS_CHECK_CLOSED_REGIONS case is a bare break;, so the
> parents that were CLOSED during the prior forward step
> MERGE_TABLE_REGIONS_CLOSE_REGIONS are never re-opened. Both parent regions
> remain OFFLINE and the RIT can persist indefinitely until an operator
> intervenes.{color}
>
> {color:#172b4d}*What happened:*{color} * {color:#172b4d}Region {{112d9f08}}
> was gracefully moved from *rs-132 → rs-81* at {{{}16:36:34 UTC{}}}.{color}
> * {color:#172b4d}The close on rs-132 had already flushed the region's edits,
> and rs-81 opened the region with {{{}openSeqNum=4997750282{}}}, establishing
> that the edits up to that point were durable.{color}
> * {color:#172b4d}~25 seconds later, rs-132 was declared dead as part of the
> broader graceful RS drain.{color}
> * {color:#172b4d}The WAL split worker (rs-34) subsequently created a
> {{recovered.edits}} file for this region containing edit
> {{{}seqId=4997750280{}}}.{color}
> * {color:#172b4d}This edit was {*}already durable in the HFiles{*}. The
> recovered edit file was effectively stale/orphaned, but its existence was not
> recognized as harmless.{color}
> {color:#172b4d}*Where it got stuck:*{color} * {color:#172b4d}At {{{}18:32:33
> UTC{}}}, {{MergeTableRegionsProcedure (pid=46990253)}} unassigned the region
> successfully.{color}
> * {color:#172b4d}During {{{}MERGE_TABLE_REGIONS_CHECK_CLOSED_REGIONS{}}},
> the procedure saw the {{recovered.edits}} file and failed the check.{color}
> * {color:#172b4d}The merge procedure retained the region lock and did not
> recover automatically, leaving the region in {*}CLOSED/RIT for ~48m
> 56s{*}.{color}
> * {color:#172b4d}The RIT only cleared after the HMaster failover caused the
> procedure to replay and fresh top-level ASSIGN procedures reopened the
> regions.{color}
> h4. *Root Cause*
> MergeTableRegionsProcedure.rollbackState() at
> MergeTableRegionsProcedure.java:287-288:
>
> case MERGE_TABLE_REGIONS_CHECK_CLOSED_REGIONS: break; // no-op — parents stay
> OFFLINE
> The symmetric case in SplitTableRegionProcedure.rollbackState() at
> SplitTableRegionProcedure.java:386-388 handles this correctly:
>
> case SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS: openParentRegion(env); //
> calls reopenRegionsForRollback break;
> openParentRegion() (SplitTableRegionProcedure.java:644-648) delegates to the
> shared helper AssignmentManagerUtil.reopenRegionsForRollback()
> (AssignmentManagerUtil.java:283-290). The merge path is missing this call.
> h4. Key Observations
> * MergeTableRegionsProcedure pid=46990253 tripped checkClosedRegion at
> 18:32:33.293 UTC on parent 112d9f08181e80aaae01e68c9229c3cb.
> * Rollback ran but did not re-open either parent (112d9f08… and 9e93ca76…),
> leaving both OFFLINE.
> * The parents held an xlock preventing any fresh ASSIGN procedures from
> making progress.
> * Stuck RIT lasted 48 minutes (18:32:33 → 19:21:29) on the OpenTSDB critical
> tier; only operator intervention (fresh ASSIGN procedures
> pid=47001645/47001646 after HMaster failover) cleared it.
> h4. Proposed Solution
> {code:java}
> /hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/MergeTableRegionsProcedure.java
>
> @@ line 287 case MERGE_TABLE_REGIONS_CHECK_CLOSED_REGIONS:
> openParentRegions(env);
> break; {code}
>
> The helper AssignmentManagerUtil.reopenRegionsForRollback() is already
> exercised by SplitTableRegionProcedure under production load, so risk is
> minimal.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
