Boyi Dai created HBASE-30547:
--------------------------------
Summary: Master active time can go backwards after a failover to a
master whose clock is behind
Key: HBASE-30547
URL: https://issues.apache.org/jira/browse/HBASE-30547
Project: HBase
Issue Type: Bug
Components: master, Region Assignment
Affects Versions: 2.5.11, 2.6.1, 3.0.0-beta-2, 4.0.0-alpha-1
Reporter: Boyi Dai
Since HBASE-28690, remote procedures send the active time of the master that
started them, and {{MasterRpcServices.throwOnOldMaster}} rejects a report whose
initiating active time is greater than the receiving master's own, with
{{MasterNotRunningException("Another master is active")}}. The active time is
taken from the wall clock in {{HMaster.finishActiveMasterInitialization}}. If
the next active master's clock is behind the previous one's by more than the
time between the two masters becoming active, its active time is lower than the
previous master's, and it rejects reports for procedures the previous master
started.
For a region open this does not recover on its own:
# The active master M0 dispatches an {{OpenRegionProcedure}} with its active
time T0, and the region server opens the region and reports OPENED with T0.
# M0 loses leadership before handling the report, and M1 becomes active with an
active time T1 < T0 because its clock is behind.
# {{HRegionServer.reportRegionStateTransition}} builds the request once and
retries it with T0, and M1 rejects every attempt. {{MasterNotRunningException}}
is not one of the exceptions it backs off on, so it retries immediately.
# M1 dispatches the open again with the same procedure id, and the region
server ignores it with "already submitted, just ignore it", since the first
handler is still reporting.
The region stays in OPENING for as long as M1 is active. With the clocks skewed
this way, the same comparison also lets a master that is no longer active
accept reports for procedures started by the new master (T1 < T0), which is
what HBASE-28690 meant to prevent. Region close and the other remote procedures
carry the same active time, so their reports are rejected the same way.
This was found through code analysis, and I reproduced the region open case
with a test that holds the OPENED report on the active master, fails over to a
master whose clock is behind, and releases the report. The new master logged
{{Report for procId: 9 and initiatingMasterAT 1794786260006 received on master
with activeTime 1794786200028}} about 50,000 to 110,000 times in the 60 seconds
the test waits, since the region server retries without backing off, and the
procedure never finished.
*Expected:* the active time of a new active master is greater than that of
every earlier one, whatever the clocks say.
*Proposed fix:* store the active time in the master local region. A master
becoming active opens the master local region, which fences off the previous
master, then takes {{max(now, last stored + 1)}} as its active time and stores
it, before any remote procedure is dispatched. A warning is logged when the
stored time wins, since it means the masters' clocks disagree. The fix only
covers failovers between masters that have it; branch-2.5 has HBASE-28690 but
no master state store, so a backport there would need another place to keep the
value.
I have a patch with tests and will open a PR.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)