Lokesh Khurana created PHOENIX-8027:
---------------------------------------
Summary: Allow consistent failover to be initiated only from
ACTIVE_IN_SYNC
Key: PHOENIX-8027
URL: https://issues.apache.org/jira/browse/PHOENIX-8027
Project: Phoenix
Issue Type: Sub-task
Reporter: Lokesh Khurana
Assignee: Lokesh Khurana
Today initiate-failover is accepted from ACTIVE_NOT_IN_SYNC (store-and-forward)
and moves the
group to ACTIVE_NOT_IN_SYNC_TO_STANDBY while the forwarder drains the local
backlog to the
standby. In that window a single peer HDFS write failure is unrecoverable:
1. SyncAndForwardModeImpl.onFailure calls transitionToStoreAndForward, which
asks the HAGroupStore to move the group to ACTIVE_NOT_IN_SYNC.
2. ACTIVE_NOT_IN_SYNC is not an allowed transition from
ACTIVE_NOT_IN_SYNC_TO_STANDBY, so the update throws
InvalidClusterRoleTransitionException.
3. The mode never changes, the local store-and-forward write is never
attempted, and the writer goes fatal ("Both SYNC and SAF replication writes
failed").
4. ReplicationLogGroup.abort() aborts the RegionServer. The RSs that open its
regions replay the recovered edits in IndexRegionObserver.preWALRestore and
abort the same way.
abort-failover does not help: it only applies to a standby in
STANDBY_TO_ACTIVE, and the
standby never reaches that state.
Testing found (FAILOVER policy, active cluster with 6 RegionServers, ~3 rows/s
client load):
- Repro: set the active to ACTIVE_NOT_IN_SYNC, then to
ACTIVE_NOT_IN_SYNC_TO_STANDBY, then put a 1-byte space quota on the peer's
/phoenixHA/<group>/in.
- The first RS aborted 0.13 s after the failed peer write. The other 5 aborted
6 s later on
RS_OPEN_REGION threads while opening its regions.
- 18 RS restarts in about 2 min. The group's writes stopped for 92 s. Every
region of every table on those RSs was reassigned, not only the HA group's. No
data was lost.
- With a healthy peer, the same steps drained in about 100 s and completed the
failover.
Proposal: allow initiate-failover only from ACTIVE_IN_SYNC, and reject
ACTIVE_NOT_IN_SYNC with an error telling the operator to wait for
ACTIVE_IN_SYNC, or to fix the peer first if the group is stuck. No capability
is lost: the ANIS path already had to drain to ACTIVE_IN_SYNC_TO_STANDBY before
cutover, so the operator now just waits for the drain before starting the
failover. Keep the ACTIVE_NOT_IN_SYNC_TO_STANDBY state and its transitions so
groups already in that state, and mixed-version clusters, keep working.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
