SpookyJumpyBeans opened a new pull request, #8898:
URL: https://github.com/apache/hbase/pull/8898

   https://issues.apache.org/jira/browse/HBASE-30547
   
   Since HBASE-28690 a master rejects a report whose initiating master active 
time is greater than its own. The active time comes from the wall clock, so 
after a failover to a master whose clock is behind, the new master can have a 
lower active time than the old one and rejects every report for a procedure the 
old master started. For a region open, the region server keeps retrying OPENED 
with the old active time and ignores the new master's re-dispatch of the same 
procedure, so the region stays in OPENING while the new master is active.
   
   ### Changes
   
   - New `MasterActiveTimeStore`, a `MasterStateStore` holding the last active 
master's active time in the master local region.
   - `HMaster.finishActiveMasterInitialization` still reads the clock as 
before, then, right after opening the master local region, sets the active time 
to `max(now, last stored + 1)` and stores it:
   
   ```java
   masterRegion = MasterRegionFactory.create(this);
   rsListStorage = new MasterRegionServerList(masterRegion, this);
   // The active time is sent with remote procedures and must be greater than 
that of every earlier
   // active master, see MasterActiveTimeStore. Opening the master local region 
has fenced off the
   // earlier active master, and no remote procedure has been sent yet.
   this.masterActiveTime = new 
MasterActiveTimeStore(masterRegion).advance(masterActiveTime);
   ```
   
   Opening the master local region renames the previous master's WAL directory 
and recovers the lease on its WAL files, so the previous master can no longer 
write to it. The procedure executor is created after this point, so no remote 
procedure carries the earlier value, and reports are only checked once 
`serviceStarted` is set, also later. A warning is logged when the stored time 
is used, since it means the masters' clocks disagree.
   - The "completed initialization" log line measures from the clock read 
instead of the active time, which can now be ahead of the clock.
   - `masterActiveTime` is now `volatile`, since RPC handlers read it and it is 
now written twice during initialization.
   - The clock read is moved into a protected `getCurrentTimeForActiveTime()`, 
so a test can simulate a master whose clock is behind.
   
   ### Limits
   
   - If a past active master's clock was far ahead, later active times follow 
the stored value (plus one) until the clocks catch up, so the active time shown 
in the UI and the `masterActiveTime` metric can be ahead of the wall clock in 
that time. The comparisons stay consistent; without this change such a master 
would have caused this bug for the masters after it.
   - Only failovers between masters with this change are covered. The first 
master started on this version has no stored value and uses its clock. An older 
master in a rolling upgrade does not store its active time.
   - branch-2.5 has HBASE-28690 but no master state store, so a backport there 
would need another place for the value. branch-2.6, branch-2, branch-3 have 
`MasterStateStore`.
   
   Other approaches I looked at: the master start code comes from the same 
clock; the zxid of the active master znode is monotonic but on a different 
scale, which would break comparisons with older masters during an upgrade; and 
the initiating active time is not persisted with the procedures, so it can't be 
recovered from the procedure store.
   
   ### Tests
   
   - `TestOpenRegionReportAfterFailoverToMasterWithClockBehind`: holds the 
OPENED report on the active master, fails over to a master whose clock is 
behind (through an override of `getCurrentTimeForActiveTime()`), and releases 
the report. Without this change the new master keeps rejecting the report 
("initiatingMasterAT 1794786260006 received on master with activeTime 
1794786200028", thousands of times per run) and the open never finishes. With 
this change the open finishes on the new master.
   - `TestMasterActiveTimeStore`: unit tests for `advance`, including a 
malformed stored value.
   
   I ran these on Linux (WSL) with JDK 21: the new test passed 5 of 5 runs with 
this change and fails without it. These also pass with it: 
`TestMasterActiveTimeStore`, `TestOpenRegionProcedureHang`, 
`TestReportRegionStateTransitionRetry`, `TestServerRemoteProcedure`, 
`TestMasterFailover`, `TestMasterMetrics`, `TestMasterMetricsWrapper`, 
`TestMasterNoCluster`, `TestMasterRegionInitialize`, 
`TestSnapshotCleanupStateStore`.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to