[
https://issues.apache.org/jira/browse/HBASE-21788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16762967#comment-16762967
]
Bahram Chehrazy commented on HBASE-21788:
-----------------------------------------
I rebooted the master and it got stuck in the same state again. Maybe the
server "<server1>,16020,1549450371876" no longer owns meta. But the stale state
in the ZK is lying to the master.
2019-02-06 17:50:25,785 INFO [master/**************:16000:becomeActiveMaster]
assignment.AssignmentManager: Attach pid=32700, ppid=32695,
state=WAITING:*REGION_STATE_TRANSITION_CONFIRM_OPENED*, hasLock=false;
TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN to
rit=*OPENING*, location=*<server1>,16020,1549450371876*, table=hbase:meta,
region=1588230740 to restore RIT
2019-02-06 17:50:25,867 INFO
[master/****************:16000:becomeActiveMaster] master.ServerManager:
Registering regionserver=*<server1>,16020,1549450371876*
2019-02-06 17:50:26,168 INFO [master/**************:16000:becomeActiveMaster]
master.HMaster: hbase:meta
{1588230740 state=OPENING, ts=1549451497525,
server=*<server1>,16020,1549450371876*}
2019-02-06 17:50:30,760 WARN [master/************:16000:becomeActiveMaster]
master.HMaster: hbase:meta,,1.1588230740 is NOT online; state=\{1588230740
state=OPENING, ts=1549451497525, server=*<server1>,16020,1549450371876*}
; ServerCrashProcedures=true. Master startup cannot progress, in
holding-pattern until region onlined.
> OpenRegionProcedure (after recovery?) is unreliable and needs to be improved
> ----------------------------------------------------------------------------
>
> Key: HBASE-21788
> URL: https://issues.apache.org/jira/browse/HBASE-21788
> Project: HBase
> Issue Type: Bug
> Affects Versions: 3.0.0
> Reporter: Sergey Shelukhin
> Priority: Critical
>
> Not much for this one yet.
> I repeatedly see the cases when the region is stuck in OPENING, and after
> master restart RIT is recovered, and stays WAITING; its OpenRegionProcedure
> (also recovered) is stuck in Runnable and never does anything for hours. I
> cannot find logs on the target server indicating that it ever tried to do
> anything after master restart.
> This procedure needs at the very least logging of what it's trying to do, and
> maybe a timeout so it unconditionally fails after a configurable period (1
> hour?).
> I may also investigate why it doesn't do anything and file a separate bug. I
> wonder if it's somehow related to the region status check, but this is just a
> hunch.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
