[
https://issues.apache.org/jira/browse/HBASE-26371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HBASE-26371 started by Viraj Jasani.
--------------------------------------------
> Prioritize meta region move over other region moves in region_mover
> -------------------------------------------------------------------
>
> Key: HBASE-26371
> URL: https://issues.apache.org/jira/browse/HBASE-26371
> Project: HBase
> Issue Type: Bug
> Affects Versions: 1.6.0
> Reporter: Viraj Jasani
> Assignee: Viraj Jasani
> Priority: Major
> Fix For: 2.5.0, 3.0.0-alpha-2, 1.7.2, 2.4.8, 2.3.8
>
>
> We have seen few issues in production when meta region movement took some
> time from one server to another and in the meanwhile some other system
> table's regions were also moved (that were hosted on the same server)
> simultaneously but when non-meta system regions came online on other servers,
> the new servers could not make info:sn update to meta table for updated
> destination of system regions (e.g namespace region) and at the same time,
> active master was also bounced and the new active master that comes online
> usually reads namespace region's location from meta table and considers it as
> final, hence even if for instance, namespace region is already online (but on
> different host), the inconsistent info:sn value would prevent master from
> getting initialized because it keeps waiting for namespace region's
> availability on old regionserver. In this case, we need to make special
> arrangement to bring namespace region online on the old server only.
> {code:java}
> 2021-10-12 20:00:00,630 INFO [1f507eff84ef336f1250]
> regionserver.HRegionServer - Post open deploy tasks for
> hbase:namespace,1626899414773.52693312958f1f507eff84ef336f1250.
> 2021-10-12 20:04:18,622 INFO [1f507eff84ef336f1250] hbase.MetaTableAccessor -
> Updated row hbase:namespace,1626899414773.52693312958f1f507eff84ef336f1250.
> with server=server-0,60020,1633467603387
> 2021-10-12 20:04:18,622 INFO [1f507eff84ef336f1250] client.AsyncProcess -
> #27, waiting for some tasks to finish. Expected max=0, tasksInProgress=4
> hasError=false, tableName=hbase:meta
> 2021-10-12 20:04:18,622 INFO [1f507eff84ef336f1250] client.AsyncProcess -
> Left over 4 task(s) are processed on server(s): []
> 2021-10-12 20:04:18,622 DEBUG [1f507eff84ef336f1250]
> regionserver.HRegionServer - Finished post open deploy task for
> hbase:namespace,1626899414773.52693312958f1f507eff84ef336f1250.
> {code}
> Similar to namespace, even other user or system table regions that are hosted
> on the same server as meta have also encountered inconsistent state updates
> specifically when meta region moves around and active master is also
> restarted around the same time. And once active master comes online, we have
> to fix such inconsistencies with hbck.
> On the other hand, there have been some enhancement around not requiring meta
> region's colocation with active master as part of ZK-less region assignment,
> e.g HBASE-11610
> We have not yet enabled ZK-less region assignment entirely, only migration
> config is enabled i.e. hbase.assignment.usezk.migrating. With this, we expect
> active master to perform an additional write to meta table for the updated
> region state (in addition to updating RIT map in the memory of RegionStates).
> We have seen some hanging state here as well if meta region is going through
> some transition (not available) and other non-meta regions are also moved by
> the region mover simultaneously, and active master cannot complete meta
> update, which further delays intermediate state transition based ZK watcher
> updates.
> {code:java}
> client.AsyncProcess - #3, waiting for 1 actions to finish on table:
> hbase:meta
> {code}
> If we take a step back, and think about these issues, all issues are
> associated with graceful start/stop of regionservers. Region mover will try
> to move all regions of the given server in parallel using user configurable
> thread pool and hence it gives no preference to meta.
> On the other hand, after trying to reproduce this inconsistent region state
> behaviour with non-graceful start/stop, I have realized that we don't face
> such issues because ServerCrashProcedure (SCP) always prioritize meta
> region's availability over any other regions if the server being processed by
> the SCP was hosting the meta region. This is exactly what region_mover should
> also provide. Given that every non-meta region's location is stored in meta
> table, meta region must always be moved first and only after it comes online,
> can other regions be allowed to be moved in parallel using the configured
> thread pool.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
