[
https://issues.apache.org/jira/browse/HBASE-28192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17784677#comment-17784677
]
Viraj Jasani commented on HBASE-28192:
--------------------------------------
{quote}Actually, the most dangerous thing is always that, people think they can
fix something without knowing the root cause and then they just make thing
worse...
{quote}
I agree, but this case is quite particular. I am not suggesting we schedule
recovery for any inconsistent state of meta, I just meant to say that if meta
is already online as per AssignmentManager but the server it is online on is
not even live, we already have a problem that we will likely not recover unless
that dead server SCP is being processed. The only way out for this case is for
operator to schedule recovery of the old server, the more it takes for operator
to understand what the current state of the cluster is, higher are the chances
of client requests failures in that duration and higher num of stuck procedures
will be accumulated.
If meta state is not online, we don't need any change in the current logic.
{quote}So here, meta is already online on server3-1,61020,1699456864765, but
after server1 becomes active, the loaded meta location is
server3-1,61020,1698687384632, which is a dead server?
{quote}
Correct.
{quote}And this happens on a rolling upgrading from 2.4 to 2.5? What is the
version for server1 and server4? Server4 is 2.4.x and server and server1 is
2.5.x?
{quote}
Yes, so far we observed this only during 2.4 to 2.5 upgrade. Let me get back
with the version details of masters (server4 and server1) in sometime.
> Master should recover if meta region state is inconsistent
> ----------------------------------------------------------
>
> Key: HBASE-28192
> URL: https://issues.apache.org/jira/browse/HBASE-28192
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 2.4.17, 2.5.6
> Reporter: Viraj Jasani
> Assignee: Viraj Jasani
> Priority: Major
> Fix For: 2.6.0, 2.4.18, 3.0.0-beta-1, 2.5.7
>
>
> During active master initialization, before we set master as active (i.e.
> {_}setInitialized(true){_}), we need both meta and namespace regions online.
> If the region state of meta or namespace is inconsistent, active master can
> get stuck in the initialization step:
> {code:java}
> private boolean isRegionOnline(RegionInfo ri) {
> RetryCounter rc = null;
> while (!isStopped()) {
> ...
> ...
> ...
> // Check once-a-minute.
> if (rc == null) {
> rc = new RetryCounterFactory(Integer.MAX_VALUE, 1000, 60_000).create();
> }
> Threads.sleep(rc.getBackoffTimeAndIncrementAttempts());
> }
> return false;
> }
> {code}
> In one of the recent outage, we observed that meta was online on a server,
> which was correctly reflected in meta znode, but the server starttime was
> different. This means that as per the latest transition record, meta was
> marked online on old server (same server with old start time). This kept
> active master initialization waiting forever and some SCPs got stuck in
> initial stage where they need to access meta table before getting candidate
> for region moves.
> The only way out of this outage is for operator to schedule recoveries using
> hbck for old server, which triggers SCP for old server address of meta. Since
> many SCPs were stuck, the processing of new SCP too was taking some time and
> manual restart of active master triggered failover, and new master was able
> to complete SCP for old meta server, correcting the meta assignment details,
> which eventually marked master as active and only after this, we were able to
> see real large num of RITs that were hidden so far.
> We need to let master recover from this state to avoid manual intervention.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
