[
https://issues.apache.org/jira/browse/HBASE-21083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16704195#comment-16704195
]
Xu Cang edited comment on HBASE-21083 at 11/30/18 3:32 AM:
-----------------------------------------------------------
[~allan163] thanks for response
The reason I am asking is, seems for hbase branch-1 there is no reliable way to
unblock or bypass stuck Procedures. And this feature is something potentially
can be applied to branch-1 to alleviate engineering burden such as manually
operating on WAL files.
I briefly skimmed your patch and I see most of the code change is related to
ProcedureV2, not AMv2. So since branch-1 has ProcedureV2, I wanted to ask how
hard to port this high-level logic to branch-1.
was (Author: xucang):
[~allan163] thanks for response
The reason I am asking is, seems for hbase branch-1 there is no reliable way to
unblock or bypass stuck Procedures. And this feature is something potentially
can be applied to branch-1 to alleviate engineering burden such as manually
operating on WAL files.
I briefly skimmed your patch and I see most of the code change is related to
ProcedureV2, not AMv2. So since branch-1 has ProcedureV2, so I was asking how
hard to port this high-level logic to branch-1.
> Introduce a mechanism to bypass the execution of a stuck procedure
> ------------------------------------------------------------------
>
> Key: HBASE-21083
> URL: https://issues.apache.org/jira/browse/HBASE-21083
> Project: HBase
> Issue Type: Sub-task
> Components: amv2
> Affects Versions: 2.1.0, 2.0.1
> Reporter: Allan Yang
> Assignee: Allan Yang
> Priority: Major
> Fix For: 3.0.0, 2.1.1, 2.0.2
>
> Attachments: HBASE-21083.branch-2.0.001.patch,
> HBASE-21083.branch-2.0.002.patch, HBASE-21083.branch-2.0.003.patch,
> HBASE-21083.branch-2.0.003.patch, HBASE-21083.branch-2.0.003.patch,
> HBASE-21083.branch-2.1.001.patch
>
>
> Offline discussed with [~stack] and [~Apache9]. We all agreed that we need to
> introduce a mechanism to 'force complete' a stuck procedure, so the AMv2 can
> continue running.
> we still have some unrevealed bugs hiding in our AMv2 and procedureV2 system,
> we need something to interfere with stuck procedures before HBCK2 can work.
> This is very crucial for a production ready system.
> For now, we have little ways to interfere with running procedures. Aborting
> them is not a good choice, since some procedures are not abort-able. And some
> procedure may have overridden the abort() method, which will ignore the abort
> request.
> So, here, I will introduce a mechanism to bypass the execution of a stuck
> procedure.
> Basically, I added a field called 'bypass' to Procedure class. If we set this
> field to true, all the logic in execute/rollback will be skipped, letting
> this procedure and its ancestors complete normally and releasing the lock
> resources at last.
> Notice that bypassing a procedure may leave the cluster in a middle state,
> e.g. the region not assigned, or some hdfs files left behind.
> The Operators need know the side effect of bypassing and recover the
> inconsistent state of the cluster themselves, like issuing new procedures to
> assign the regions.
> A patch will be uploaded and review board will be open. For now, only APIs in
> ProcedureExecutor are provided. If anything is fine, I will add it to master
> service and add a shell command to bypass a procedure. Or, maybe we can use
> dynamically compiled JSPs to execute those APIs as mentioned in HBASE-20679.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
