[
https://issues.apache.org/jira/browse/HBASE-21843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16759961#comment-16759961
]
Wellington Chevreuil commented on HBASE-21843:
----------------------------------------------
Main issue is how the RS instance ended up with no WAL dir. I've seen this
happen when my VMs cluster was running got paused/resumed. Cluster was running,
region R1 was assigned in meta to say, RS1-TS1. Several host crashes/restarts
later, cluster came back on, now say RS1-TS5, meta still has region R1 assigned
to RS1-T1 in meta, but there's no RS1-T1 WAL dir. In this case, R1 is not added
to the list of offline regions during meta load (because it's state is open),
and there's never a SCP triggered for RS1-T1 (because
RegionServerTracker/ServerManaer don't find any RS1-T1 anywhere), so R1 will be
forever open on RS1-T1 in meta, never actually getting assigned to a live RS.
> AM misses region assignment in catastrophic scenarios where RS assigned to
> the region in Meta does not have a WAL dir.
> ----------------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-21843
> URL: https://issues.apache.org/jira/browse/HBASE-21843
> Project: HBase
> Issue Type: Bug
> Components: amv2
> Affects Versions: 3.0.0, 2.1.0, 2.2.0
> Reporter: Wellington Chevreuil
> Assignee: Wellington Chevreuil
> Priority: Major
> Attachments: HBASE-21843.master.001.patch
>
>
> A bit unusual, but managed to face this twice lately on both distributed and
> local standalone mode, on VMs. Somehow, after some VM pause/resume, got into
> a situation where regions on meta were assigned to a give RS startcode that
> had no corresponding WAL dir.
> That caused those regions to never get assigned, because the given RS
> startcode is not found anywhere by RegionServerTracker/ServerManager, so no
> SCP is created to this RS startcode, leaving the region "open" on a dead
> server forever, in META.
> Could get this sorted by adding extra check on loadMeta, checking if the RS
> assigned to the region in meta is not online and doesn't have a WAL dir, then
> mark this region as offline.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
