[ 
https://issues.apache.org/jira/browse/PHOENIX-8037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lokesh Khurana resolved PHOENIX-8037.
-------------------------------------
    Resolution: Fixed

> PhoenixHAAdminTool: print failover time markers
> -----------------------------------------------
>
>                 Key: PHOENIX-8037
>                 URL: https://issues.apache.org/jira/browse/PHOENIX-8037
>             Project: Phoenix
>          Issue Type: Sub-task
>            Reporter: Lokesh Khurana
>            Assignee: Lokesh Khurana
>            Priority: Major
>
> h3. Change
> {{PhoenixHAAdminTool initiate-failover}} and {{abort-failover}} print one 
> timestamped marker line at the start and one at the end of the command. Each 
> line goes to stdout and to the log at INFO:
> {noformat}
> PHOENIX_HA_FAILOVER_MARKER marker=<NAME> group=<HA group> ts=<epoch ms> 
> ts_utc=<yyyy-MM-ddTHH:mm:ss.SSSZ>
> {noformat}
> Markers:
> {{{}- FAILOVER_START{}}}: after the arguments are parsed, before the tool 
> reads the HA group record.
> {{{}- FAILOVER_COMPLETE{}}}: the cluster pair reached a stable state (local 
> {{{}STANDBY{}}}, peer active) before the timeout.
> {{{}- FAILOVER_TIMEOUT{}}}: the cluster pair did not reach a stable state 
> before the timeout, or the poll failed.
> {{{}- FAILOVER_FAILED{}}}: the command stopped early with an error. For 
> example, the HA group is not found, the state is not {{{}ACTIVE_IN_SYNC{}}}, 
> or the state transition is rejected.
> {{{}- ABORT_START{}}}, {{{}ABORT_COMPLETE{}}}, {{ABORT_TIMEOUT}} and 
> {{{}ABORT_FAILED{}}}: the same events for {{{}abort-failover{}}}.
> Each start marker has exactly one end marker.
> h3. Motivation
> A RegionServer can stop after it writes a mutation to the WAL and before it 
> sends the mutation to the standby cluster. The client gets an error for that 
> write. The ServerCrashProcedure then replays the WAL on the active cluster. 
> {{IndexRegionObserver#preWALRestore}} sends a replayed edit to the standby 
> only while the local cluster role is active. Testing found this sequence:
> - A RegionServer stops in this window.
> - A failover starts before the WAL replay.
> - The replay applies the edit on the old active cluster only, so the two 
> clusters have different data.
> - A failover back to the old active cluster shows a write that the client 
> thinks failed.
> A check for pending ServerCrashProcedures before the failover does not 
> prevent this:
> - The replayed edits are still sent while the role isĀ 
> {{{}ACTIVE_TO_STANDBY{}}}. They stop only when the cluster moves to 
> {{{}STANDBY{}}}, after the peer reaches {{{}ACTIVE_IN_SYNC{}}}. This occurs 
> after the check, and the tool does not control it.
> - The check cannot see a RegionServer that has stopped but whose ZooKeeper 
> session has not expired yet. This can take 40 s to 90 s, as given by the 
> ZooKeeper server settings.
> - The check cannot see a region that the ServerCrashProcedure gives to a 
> region procedure that is already attached.
> - Every RegionServer exit gets a ServerCrashProcedure, also a clean stop. 
> Thus the check would also fire during a rolling restart.
> h3. Operator steps after a failover
> - Wait for the old active cluster to quiesce, with no pending 
> ServerCrashProcedures and no regions in transition.
> - Find and repair the divergence with PhoenixSyncTableTool.
> The markers give the exact time of the failover. Use them to find the related 
> logs and to select the time range for PhoenixSyncTableTool. Replayed cells 
> keep their original write timestamps, which are before 
> {{{}FAILOVER_START{}}}. Thus {{-{-}from-time{-}}} must start before 
> {{{}FAILOVER_START{}}}. Set {{-to-time}} explicitly, because the default is 
> the current time minus 1 hour.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

Reply via email to