[ 
https://issues.apache.org/jira/browse/HBASE-30547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ASF GitHub Bot updated HBASE-30547:
-----------------------------------
    Labels: pull-request-available  (was: )

> Master active time can go backwards after a failover to a master whose clock 
> is behind
> --------------------------------------------------------------------------------------
>
>                 Key: HBASE-30547
>                 URL: https://issues.apache.org/jira/browse/HBASE-30547
>             Project: HBase
>          Issue Type: Bug
>          Components: master, Region Assignment
>    Affects Versions: 4.0.0-alpha-1, 3.0.0-beta-2, 2.6.1, 2.5.11
>            Reporter: Boyi Dai
>            Priority: Major
>              Labels: pull-request-available
>
> Since HBASE-28690, remote procedures send the active time of the master that 
> started them, and {{MasterRpcServices.throwOnOldMaster}} rejects a report 
> whose initiating active time is greater than the receiving master's own, with 
> {{MasterNotRunningException("Another master is active")}}. The active time is 
> taken from the wall clock in {{HMaster.finishActiveMasterInitialization}}. If 
> the next active master's clock is behind the previous one's by more than the 
> time between the two masters becoming active, its active time is lower than 
> the previous master's, and it rejects reports for procedures the previous 
> master started.
> For a region open this does not recover on its own:
> # The active master M0 dispatches an {{OpenRegionProcedure}} with its active 
> time T0, and the region server opens the region and reports OPENED with T0.
> # M0 loses leadership before handling the report, and M1 becomes active with 
> an active time T1 < T0 because its clock is behind.
> # {{HRegionServer.reportRegionStateTransition}} builds the request once and 
> retries it with T0, and M1 rejects every attempt. 
> {{MasterNotRunningException}} is not one of the exceptions it backs off on, 
> so it retries immediately.
> # M1 dispatches the open again with the same procedure id, and the region 
> server ignores it with "already submitted, just ignore it", since the first 
> handler is still reporting.
> The region stays in OPENING for as long as M1 is active. With the clocks 
> skewed this way, the same comparison also lets a master that is no longer 
> active accept reports for procedures started by the new master (T1 < T0), 
> which is what HBASE-28690 meant to prevent. Region close and the other remote 
> procedures carry the same active time, so their reports are rejected the same 
> way.
> This was found through code analysis, and I reproduced the region open case 
> with a test that holds the OPENED report on the active master, fails over to 
> a master whose clock is behind, and releases the report. The new master 
> logged {{Report for procId: 9 and initiatingMasterAT 1794786260006 received 
> on master with activeTime 1794786200028}} about 50,000 to 110,000 times in 
> the 60 seconds the test waits, since the region server retries without 
> backing off, and the procedure never finished.
> *Expected:* the active time of a new active master is greater than that of 
> every earlier one, whatever the clocks say.
> *Proposed fix:* store the active time in the master local region. A master 
> becoming active opens the master local region, which fences off the previous 
> master, then takes {{max(now, last stored + 1)}} as its active time and 
> stores it, before any remote procedure is dispatched. A warning is logged 
> when the stored time wins, since it means the masters' clocks disagree. The 
> fix only covers failovers between masters that have it; branch-2.5 has 
> HBASE-28690 but no master state store, so a backport there would need another 
> place to keep the value.
> I have a patch with tests and will open a PR.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

Reply via email to