Boyi Dai created HBASE-30547:
--------------------------------

             Summary: Master active time can go backwards after a failover to a 
master whose clock is behind
                 Key: HBASE-30547
                 URL: https://issues.apache.org/jira/browse/HBASE-30547
             Project: HBase
          Issue Type: Bug
          Components: master, Region Assignment
    Affects Versions: 2.5.11, 2.6.1, 3.0.0-beta-2, 4.0.0-alpha-1
            Reporter: Boyi Dai


Since HBASE-28690, remote procedures send the active time of the master that 
started them, and {{MasterRpcServices.throwOnOldMaster}} rejects a report whose 
initiating active time is greater than the receiving master's own, with 
{{MasterNotRunningException("Another master is active")}}. The active time is 
taken from the wall clock in {{HMaster.finishActiveMasterInitialization}}. If 
the next active master's clock is behind the previous one's by more than the 
time between the two masters becoming active, its active time is lower than the 
previous master's, and it rejects reports for procedures the previous master 
started.

For a region open this does not recover on its own:
# The active master M0 dispatches an {{OpenRegionProcedure}} with its active 
time T0, and the region server opens the region and reports OPENED with T0.
# M0 loses leadership before handling the report, and M1 becomes active with an 
active time T1 < T0 because its clock is behind.
# {{HRegionServer.reportRegionStateTransition}} builds the request once and 
retries it with T0, and M1 rejects every attempt. {{MasterNotRunningException}} 
is not one of the exceptions it backs off on, so it retries immediately.
# M1 dispatches the open again with the same procedure id, and the region 
server ignores it with "already submitted, just ignore it", since the first 
handler is still reporting.

The region stays in OPENING for as long as M1 is active. With the clocks skewed 
this way, the same comparison also lets a master that is no longer active 
accept reports for procedures started by the new master (T1 < T0), which is 
what HBASE-28690 meant to prevent. Region close and the other remote procedures 
carry the same active time, so their reports are rejected the same way.

This was found through code analysis, and I reproduced the region open case 
with a test that holds the OPENED report on the active master, fails over to a 
master whose clock is behind, and releases the report. The new master logged 
{{Report for procId: 9 and initiatingMasterAT 1794786260006 received on master 
with activeTime 1794786200028}} about 50,000 to 110,000 times in the 60 seconds 
the test waits, since the region server retries without backing off, and the 
procedure never finished.

*Expected:* the active time of a new active master is greater than that of 
every earlier one, whatever the clocks say.

*Proposed fix:* store the active time in the master local region. A master 
becoming active opens the master local region, which fences off the previous 
master, then takes {{max(now, last stored + 1)}} as its active time and stores 
it, before any remote procedure is dispatched. A warning is logged when the 
stored time wins, since it means the masters' clocks disagree. The fix only 
covers failovers between masters that have it; branch-2.5 has HBASE-28690 but 
no master state store, so a backport there would need another place to keep the 
value.

I have a patch with tests and will open a PR.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

Reply via email to