[
https://issues.apache.org/jira/browse/PHOENIX-7987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Himanshu Gwalani updated PHOENIX-7987:
--------------------------------------
Description:
h2. What
Adds a {{force-promote}} subcommand that promotes a survivor standby to
serving-active by driving it through the validated {{STANDBY_TO_ACTIVE}}
transition, instead of forcing straight to {{ACTIVE_IN_SYNC}} via raw
{{{}update -F{}}}.
h2. Why
When the active cluster's ZK is unreachable, cooperative failover can't
complete and the only surface today is {{{}update -s ACTIVE_IN_SYNC -F{}}}.
That path bypasses transition validation, skips the replay drain, and strands
the promoted group's replay in {{DEGRADED}} forever (frozen consistency point →
cluster-wide compaction retention hold; clears only on RS restart — S20e).
{{force-promote}} lands on {{{}STANDBY_TO_ACTIVE{}}}, which reuses the
PHOENIX-7920 replay machinery to reset {{DEGRADED}} → {{SYNCED_RECOVERY}} →
{{SYNC}} and auto-promote to {{{}ACTIVE_IN_SYNC{}}}, preserving the drain
guarantee.
h2. How
* *{{HAGroupStoreManager.setHAGroupStatusToStandbyToActive()}}* — Validates
persisted state $\in$ {{{}{STANDBY, DEGRADED_STANDBY}{}}}, then calls
{{setHAGroupStatusIfNeeded(STANDBY_TO_ACTIVE)}} (validated CAS path, works
peer-blind).
* *{{force-promote}} command* — Requires an explicit ack flag + prints the
fencing prerequisite (old active must be fenced), then polls for convergence to
{{{}ACTIVE_IN_SYNC{}}}; on timeout, prints guidance + {{abort-failover}}
fallback.
* *New poll predicate ({{{}local == ACTIVE_IN_SYNC{}}})* — The existing
{{isStableFailoverPair}} checks {{local == STANDBY}} and can't be reused for a
promotion with a down peer.
All changes are in {{{}phoenix-core-client{}}}; the replay side is unchanged
(reuses PHOENIX-7920). Reversible via the existing {{{}abort-failover{}}}.
was:
The replay service can get stuck in an infinite loop if there is a persistent
issue while processing older files in the in-progress directory.
{code:java}
files = replicationLogTracker.getOlderInProgressFiles(oldestTimestampToProcess);
while (!files.isEmpty()) {
processOneRandomFile(files);
files =
replicationLogTracker.getOlderInProgressFiles(oldestTimestampToProcess);
} {code}
> Add force-promote command to PhoenixHAAdminTool
> -----------------------------------------------
>
> Key: PHOENIX-7987
> URL: https://issues.apache.org/jira/browse/PHOENIX-7987
> Project: Phoenix
> Issue Type: Sub-task
> Reporter: Himanshu Gwalani
> Assignee: Himanshu Gwalani
> Priority: Major
>
> h2. What
> Adds a {{force-promote}} subcommand that promotes a survivor standby to
> serving-active by driving it through the validated {{STANDBY_TO_ACTIVE}}
> transition, instead of forcing straight to {{ACTIVE_IN_SYNC}} via raw
> {{{}update -F{}}}.
> h2. Why
> When the active cluster's ZK is unreachable, cooperative failover can't
> complete and the only surface today is {{{}update -s ACTIVE_IN_SYNC -F{}}}.
> That path bypasses transition validation, skips the replay drain, and strands
> the promoted group's replay in {{DEGRADED}} forever (frozen consistency point
> → cluster-wide compaction retention hold; clears only on RS restart — S20e).
> {{force-promote}} lands on {{{}STANDBY_TO_ACTIVE{}}}, which reuses the
> PHOENIX-7920 replay machinery to reset {{DEGRADED}} → {{SYNCED_RECOVERY}} →
> {{SYNC}} and auto-promote to {{{}ACTIVE_IN_SYNC{}}}, preserving the drain
> guarantee.
> h2. How
> * *{{HAGroupStoreManager.setHAGroupStatusToStandbyToActive()}}* — Validates
> persisted state $\in$ {{{}{STANDBY, DEGRADED_STANDBY}{}}}, then calls
> {{setHAGroupStatusIfNeeded(STANDBY_TO_ACTIVE)}} (validated CAS path, works
> peer-blind).
> * *{{force-promote}} command* — Requires an explicit ack flag + prints the
> fencing prerequisite (old active must be fenced), then polls for convergence
> to {{{}ACTIVE_IN_SYNC{}}}; on timeout, prints guidance + {{abort-failover}}
> fallback.
> * *New poll predicate ({{{}local == ACTIVE_IN_SYNC{}}})* — The existing
> {{isStableFailoverPair}} checks {{local == STANDBY}} and can't be reused for
> a promotion with a down peer.
> All changes are in {{{}phoenix-core-client{}}}; the replay side is unchanged
> (reuses PHOENIX-7920). Reversible via the existing {{{}abort-failover{}}}.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
