[
https://issues.apache.org/jira/browse/HBASE-22143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16812706#comment-16812706
]
stack commented on HBASE-22143:
-------------------------------
Dealing with these in the past, IIRC, I'd file an assign for each region stuck
in OPENING. For assigns, expected states can be
{code}
State.OPEN, // State may already be OPEN if we died after receiving the OPEN
from regionserver
// but before complete finish of AssignProcedure. HBASE-20100.
State.OFFLINE, State.CLOSED, // disable/offline
State.SPLITTING, State.SPLIT, // ServerCrashProcedure
State.OPENING, State.FAILED_OPEN, // already in-progress (retrying)
{code}
After assign completed, I'd then file the unassign.
Thats what this paragraph in the 'fixing' section on the hbck2 page was trying
to say:
"Also, if a table is DISABLED, you cannot assign a Region. In the Master logs,
you will see that the Master will report that the assign has been skipped
because the table is DISABLED. You may want to assign a Region because it is
currently in the OPENING state and you want it in the CLOSED state so it agrees
with the table's DISABLED state. In this situation, you may have to temporarily
set the table status to ENABLED, just so you can do the assign, and then set it
back again after the unassign."
https://github.com/apache/hbase-operator-tools/tree/master/hbase-hbck2
That said, this facility looks like a nice addition. Was it useful in a
particular predicament [~wchevreuil]? IIRC, locks were a problem. There might
not be an AP anymore but the lock might be in place. Would have to break those
before being able to do anything.
> HBCK2 setRegionState command
> ----------------------------
>
> Key: HBASE-22143
> URL: https://issues.apache.org/jira/browse/HBASE-22143
> Project: HBase
> Issue Type: New Feature
> Components: hbase-operator-tools, hbck2
> Reporter: Wellington Chevreuil
> Assignee: Wellington Chevreuil
> Priority: Minor
> Fix For: hbck2-1.0.0
>
> Attachments: HBASE-22143.master.0001.patch,
> HBASE-22143.master.0002.patch, HBASE-22143.master.0003.patch
>
>
> Among some of the current AMv2 issues, we faced situation where some regions
> had state as OPENING in meta, with an RS startcode that was not valid
> anymore. There was no AP running, the region stays permanently being logged
> as IN-Transition on master logs, yet no procedure is really trying to bring
> it online. Current hbck2 unassigns/assigns commands didn't work either, as
> per the exception shown, it expects regions to be in state SPLITTING, SPLIT,
> MERGING, OPEN, or CLOSING:
> {noformat}
> WARN org.apache.hadoop.hbase.master.assignment.RegionTransitionProcedure:
> Failed transition, suspend 1secs pid=7093,
> state=RUNNABLE:REGION_TRANSITION_DISPATCH, locked=true; UnassignProcedure
> table=rc_accounts, region=db85127b77fa56f7ad44e2c988e53925,
> server=server1.example.com,16020,1552682193324; rit=OPENING,
> location=server1.example.com,16020,1552682193324; waiting on rectified
> condition fixed by other Procedure or operator intervention
> org.apache.hadoop.hbase.exceptions.UnexpectedStateException: Expected
> [SPLITTING, SPLIT, MERGING, OPEN, CLOSING] so could move to CLOSING but
> current state=OPENING
> at
> org.apache.hadoop.hbase.master.assignment.RegionStates$RegionStateNode.transitionState(RegionStates.java:166)
> at
> org.apache.hadoop.hbase.master.assignment.AssignmentManager.markRegionAsClosing(AssignmentManager.java:1479)
> at
> org.apache.hadoop.hbase.master.assignment.UnassignProcedure.updateTransition(UnassignProcedure.java:212)
> at
> org.apache.hadoop.hbase.master.assignment.RegionTransitionProcedure.execute(RegionTransitionProcedure.java:369)
> at
> org.apache.hadoop.hbase.master.assignment.RegionTransitionProcedure.execute(RegionTransitionProcedure.java:97)
> at org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:957)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1835)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1595){noformat}
> In this specific case, since we know the region is not actually being
> operated by any proc and is not really open anywhere, it's ok to manually set
> it's state to one of those assigns/unassigns can operate on, so this jira
> proposes a new hbck2 command that allows for arbitrarily set a region to a
> given state.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
