[
https://issues.apache.org/jira/browse/PHOENIX-7978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Himanshu Gwalani updated PHOENIX-7978:
--------------------------------------
Description:
The replay/forward round-eligibility gate is evaluated on the wall clock, but
PHOENIX-7813 aligned the scheduler wake — fired on the monotonic clock
(System.nanoTime) — to that boundary with zero margin. Small
nanoTime-vs-wall-clock drift can tip a wake just below the boundary, so the
round is not yet eligible and the region server loses a full (~60s) cycle. Most
damaging during planned failover.
Fix (shared base class ReplicationLogDiscovery, inherited by replay +
forwarder):
1. Epsilon margin so the aligned wake lands just after the boundary rather
than exactly on it
(config phoenix.replication.discovery.aligned.delay.epsilon.millis,
default 500ms).
2. Re-anchor to the wall-clock grid every cycle (self-rescheduling one-shot
instead of
scheduleAtFixedRate) so alignment error cannot accumulate.
was:
*Issue Summary*
A _planned_ failover under sustained write load converged *correctly* — no
deadlock, no dual-active, zero RPO, data byte-for-byte identical — but took
{*}~131 s{*}, over the documented *120 s* SLA ceiling. The excess is *not* a
drain-volume problem and *not* the old deadlock. It is a {*}scheduling
artifact{*}: the standby’s replay poller runs on a rigid fixed-rate grid, and
on this run it woke *~20 ms before* the final round became eligible for replay,
so it saw “nothing to do,” went back to sleep, and did not look again for a
{*}full ~60 s cycle{*}. A 20 ms miss cost ~59 s of wall-clock.
*Fix*
**
Two scheduling styles would behave very differently here:
* *Fixed-delay* waits N seconds {_}after each run finishes{_}. A run that
found nothing could be followed by another attempt one interval later — but
more importantly, the phase drifts, so you would not deterministically re-miss
the same eligibility instant.
* *Fixed-rate* (what is used) fires on the immovable grid regardless. When a
tick returns empty there is no short retry — the code just waits for the next
grid slot, ~60 s away.
The poller has no notion of “the thing I’m waiting for will be ready in 20 ms,
let me look again soon.” It looked, saw not-ready, and its next opportunity was
a minute out. That ~59 s of dead time is a pure scheduling artifact: nothing
was draining, nothing was stuck, no data was at risk. The round sat
*eligible-and-unprocessed* from 06:25:09 to 06:26:08 solely because the two
clocks were 20 ms out of phas.
> Replication replay/forward can skip a round when the aligned poll wake fires
> just before the wall-clock eligibility boundary
> ----------------------------------------------------------------------------------------------------------------------------
>
> Key: PHOENIX-7978
> URL: https://issues.apache.org/jira/browse/PHOENIX-7978
> Project: Phoenix
> Issue Type: Sub-task
> Reporter: Himanshu Gwalani
> Assignee: Himanshu Gwalani
> Priority: Major
>
> The replay/forward round-eligibility gate is evaluated on the wall clock, but
> PHOENIX-7813 aligned the scheduler wake — fired on the monotonic clock
> (System.nanoTime) — to that boundary with zero margin. Small
> nanoTime-vs-wall-clock drift can tip a wake just below the boundary, so the
> round is not yet eligible and the region server loses a full (~60s) cycle.
> Most damaging during planned failover.
>
> Fix (shared base class ReplicationLogDiscovery, inherited by replay +
> forwarder):
> 1. Epsilon margin so the aligned wake lands just after the boundary rather
> than exactly on it
> (config phoenix.replication.discovery.aligned.delay.epsilon.millis,
> default 500ms).
> 2. Re-anchor to the wall-clock grid every cycle (self-rescheduling one-shot
> instead of
> scheduleAtFixedRate) so alignment error cannot accumulate.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
