[
https://issues.apache.org/jira/browse/PHOENIX-8037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lokesh Khurana resolved PHOENIX-8037.
-------------------------------------
Resolution: Fixed
> PhoenixHAAdminTool: print failover time markers
> -----------------------------------------------
>
> Key: PHOENIX-8037
> URL: https://issues.apache.org/jira/browse/PHOENIX-8037
> Project: Phoenix
> Issue Type: Sub-task
> Reporter: Lokesh Khurana
> Assignee: Lokesh Khurana
> Priority: Major
>
> h3. Change
> {{PhoenixHAAdminTool initiate-failover}} and {{abort-failover}} print one
> timestamped marker line at the start and one at the end of the command. Each
> line goes to stdout and to the log at INFO:
> {noformat}
> PHOENIX_HA_FAILOVER_MARKER marker=<NAME> group=<HA group> ts=<epoch ms>
> ts_utc=<yyyy-MM-ddTHH:mm:ss.SSSZ>
> {noformat}
> Markers:
> {{{}- FAILOVER_START{}}}: after the arguments are parsed, before the tool
> reads the HA group record.
> {{{}- FAILOVER_COMPLETE{}}}: the cluster pair reached a stable state (local
> {{{}STANDBY{}}}, peer active) before the timeout.
> {{{}- FAILOVER_TIMEOUT{}}}: the cluster pair did not reach a stable state
> before the timeout, or the poll failed.
> {{{}- FAILOVER_FAILED{}}}: the command stopped early with an error. For
> example, the HA group is not found, the state is not {{{}ACTIVE_IN_SYNC{}}},
> or the state transition is rejected.
> {{{}- ABORT_START{}}}, {{{}ABORT_COMPLETE{}}}, {{ABORT_TIMEOUT}} and
> {{{}ABORT_FAILED{}}}: the same events for {{{}abort-failover{}}}.
> Each start marker has exactly one end marker.
> h3. Motivation
> A RegionServer can stop after it writes a mutation to the WAL and before it
> sends the mutation to the standby cluster. The client gets an error for that
> write. The ServerCrashProcedure then replays the WAL on the active cluster.
> {{IndexRegionObserver#preWALRestore}} sends a replayed edit to the standby
> only while the local cluster role is active. Testing found this sequence:
> - A RegionServer stops in this window.
> - A failover starts before the WAL replay.
> - The replay applies the edit on the old active cluster only, so the two
> clusters have different data.
> - A failover back to the old active cluster shows a write that the client
> thinks failed.
> A check for pending ServerCrashProcedures before the failover does not
> prevent this:
> - The replayed edits are still sent while the role isĀ
> {{{}ACTIVE_TO_STANDBY{}}}. They stop only when the cluster moves to
> {{{}STANDBY{}}}, after the peer reaches {{{}ACTIVE_IN_SYNC{}}}. This occurs
> after the check, and the tool does not control it.
> - The check cannot see a RegionServer that has stopped but whose ZooKeeper
> session has not expired yet. This can take 40 s to 90 s, as given by the
> ZooKeeper server settings.
> - The check cannot see a region that the ServerCrashProcedure gives to a
> region procedure that is already attached.
> - Every RegionServer exit gets a ServerCrashProcedure, also a clean stop.
> Thus the check would also fire during a rolling restart.
> h3. Operator steps after a failover
> - Wait for the old active cluster to quiesce, with no pending
> ServerCrashProcedures and no regions in transition.
> - Find and repair the divergence with PhoenixSyncTableTool.
> The markers give the exact time of the failover. Use them to find the related
> logs and to select the time range for PhoenixSyncTableTool. Replayed cells
> keep their original write timestamps, which are before
> {{{}FAILOVER_START{}}}. Thus {{-{-}from-time{-}}} must start before
> {{{}FAILOVER_START{}}}. Set {{-to-time}} explicitly, because the default is
> the current time minus 1 hour.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)