[
https://issues.apache.org/jira/browse/PHOENIX-8037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lokesh Khurana updated PHOENIX-8037:
------------------------------------
Description:
h3. Change
{{PhoenixHAAdminTool initiate-failover}} and {{abort-failover}} print one
timestamped marker line at the start and one at the end of the command. Each
line goes to stdout and to the log at INFO:
{noformat}
PHOENIX_HA_FAILOVER_MARKER marker=<NAME> group=<HA group> ts=<epoch ms>
ts_utc=<yyyy-MM-ddTHH:mm:ss.SSSZ>
{noformat}
Markers:
{{{}- FAILOVER_START{}}}: after the arguments are parsed, before the tool reads
the HA group record.
{{{}- FAILOVER_COMPLETE{}}}: the cluster pair reached a stable state (local
{{{}STANDBY{}}}, peer active) before the timeout.
{{{}- FAILOVER_TIMEOUT{}}}: the cluster pair did not reach a stable state
before the timeout, or the poll failed.
{{{}- FAILOVER_FAILED{}}}: the command stopped early with an error. For
example, the HA group is not found, the state is not {{{}ACTIVE_IN_SYNC{}}}, or
the state transition is rejected.
{{{}- ABORT_START{}}}, {{{}ABORT_COMPLETE{}}}, {{ABORT_TIMEOUT}} and
{{{}ABORT_FAILED{}}}: the same events for {{{}abort-failover{}}}.
Each start marker has exactly one end marker.
h3. Motivation
A RegionServer can stop after it writes a mutation to the WAL and before it
sends the mutation to the standby cluster. The client gets an error for that
write. The ServerCrashProcedure then replays the WAL on the active cluster.
{{IndexRegionObserver#preWALRestore}} sends a replayed edit to the standby only
while the local cluster role is active. Testing found this sequence:
- A RegionServer stops in this window.
- A failover starts before the WAL replay.
- The replay applies the edit on the old active cluster only, so the two
clusters have different data.
- A failover back to the old active cluster shows a write that the client
thinks failed.
A check for pending ServerCrashProcedures before the failover does not prevent
this:
- The replayed edits are still sent while the role is
{{{}ACTIVE_TO_STANDBY{}}}. They stop only when the cluster moves to
{{{}STANDBY{}}}, after the peer reaches {{{}ACTIVE_IN_SYNC{}}}. This occurs
after the check, and the tool does not control it.
- The check cannot see a RegionServer that has stopped but whose ZooKeeper
session has not expired yet. This can take 40 s to 90 s, as given by the
ZooKeeper server settings.
- The check cannot see a region that the ServerCrashProcedure gives to a region
procedure that is already attached.
- Every RegionServer exit gets a ServerCrashProcedure, also a clean stop. Thus
the check would also fire during a rolling restart.
h3. Operator steps after a failover
- Wait for the old active cluster to quiesce, with no pending
ServerCrashProcedures and no regions in transition.
- Find and repair the divergence with PhoenixSyncTableTool.
The markers give the exact time of the failover. Use them to find the related
logs and to select the time range for PhoenixSyncTableTool. Replayed cells keep
their original write timestamps, which are before {{{}FAILOVER_START{}}}. Thus
{{-{-}from-time{-}}} must start before {{{}FAILOVER_START{}}}. Set {{-to-time}}
explicitly, because the default is the current time minus 1 hour.
was:
h3. Problem
A RegionServer (RS) on the active cluster can stop after it writes a mutation
to the HBase WAL and before it sends the mutation to the standby cluster. Thus
the mutation does not get to the standby cluster. The client receives an error
and thinks that the write failed.
The HMaster then starts a ServerCrashProcedure (SCP) for the RS. The SCP
replays the WAL into the regions on the active cluster. If the cluster is still
active at the time of the replay, {{IndexRegionObserver#preWALRestore}} sends
the mutation to the standby cluster. If a failover occurs before the replay,
the mutation does not get to the standby cluster.
If a failover starts while the SCP is pending, this sequence can occur:
# The RS on cluster A (ACTIVE) writes mutation M to the WAL. The RS stops
before it sends M to cluster B (STANDBY).
# The client receives an error for M.
# The operator starts a failover. Cluster A becomes STANDBY. Cluster B becomes
ACTIVE.
# The SCP on cluster A replays M into the local region. M does not get to
cluster B.
# Cluster A has M. Cluster B does not have M. The data on the two clusters is
not the same.
# The operator does a failover back to cluster A. Readers see M on cluster A,
but the client thinks that M failed.
At this time, {{initiate-failover}} checks only the HA group state
({{ACTIVE_IN_SYNC}}). It does not check for a pending SCP. Also,
{{initiate-failover}} has no option to bypass a check. Only {{-g}}, {{-t}} and
{{-h}} are available. The {{-F/--force}} option is available only on the
{{update}} command.
h3. Proposed change
# In {{initiate-failover}}, before the tool starts the failover, connect to the
local (active) cluster with the HBase Admin API.
# Find each {{ServerCrashProcedure}} that is not complete (for example, with
{{Admin#getProcedures()}}).
# If one or more SCPs are pending, do not start the failover. Show the
procedure IDs and the server names. Return {{RET_VALIDATION_ERROR}}.
# Add an override option to {{initiate-failover}}. Use the {{-F/--force}} name,
for consistency with the {{update}} command. With this option, the tool does
not stop the failover for a pending SCP. Use this option for an ungraceful
failover in an emergency.
# When the operator uses the override option and an SCP is pending, show a
warning. The warning tells the operator that some writes can be on the old
active cluster only.
# Update the help text for {{initiate-failover}}.
h3. Out of scope
* The repair of the writes that are on the old active cluster only.
PhoenixSyncTable (PHOENIX-7750) will do this repair in a different JIRA.
* An RS stop that the HMaster has not found yet. The check cannot see this RS
stop.
* An RS stop that occurs after the check is complete.
h3. Tests
* Unit test: the tool parses the override option. The check gives the correct
result with a mock Admin.
* IT: when an SCP is pending on the active cluster, {{initiate-failover}}
returns {{RET_VALIDATION_ERROR}}. The HA group stays in {{ACTIVE_IN_SYNC}}.
* IT: when an SCP is pending and the operator uses the override option, the
failover starts and the tool shows the warning.
* IT: when no SCP is pending, the failover starts as before.
Summary: PhoenixHAAdminTool: print failover time markers (was:
PhoenixHAAdminTool: Do not start a failover when a ServerCrashProcedure is
pending on the active cluster)
> PhoenixHAAdminTool: print failover time markers
> -----------------------------------------------
>
> Key: PHOENIX-8037
> URL: https://issues.apache.org/jira/browse/PHOENIX-8037
> Project: Phoenix
> Issue Type: Sub-task
> Reporter: Lokesh Khurana
> Assignee: Lokesh Khurana
> Priority: Major
>
> h3. Change
> {{PhoenixHAAdminTool initiate-failover}} and {{abort-failover}} print one
> timestamped marker line at the start and one at the end of the command. Each
> line goes to stdout and to the log at INFO:
> {noformat}
> PHOENIX_HA_FAILOVER_MARKER marker=<NAME> group=<HA group> ts=<epoch ms>
> ts_utc=<yyyy-MM-ddTHH:mm:ss.SSSZ>
> {noformat}
> Markers:
> {{{}- FAILOVER_START{}}}: after the arguments are parsed, before the tool
> reads the HA group record.
> {{{}- FAILOVER_COMPLETE{}}}: the cluster pair reached a stable state (local
> {{{}STANDBY{}}}, peer active) before the timeout.
> {{{}- FAILOVER_TIMEOUT{}}}: the cluster pair did not reach a stable state
> before the timeout, or the poll failed.
> {{{}- FAILOVER_FAILED{}}}: the command stopped early with an error. For
> example, the HA group is not found, the state is not {{{}ACTIVE_IN_SYNC{}}},
> or the state transition is rejected.
> {{{}- ABORT_START{}}}, {{{}ABORT_COMPLETE{}}}, {{ABORT_TIMEOUT}} and
> {{{}ABORT_FAILED{}}}: the same events for {{{}abort-failover{}}}.
> Each start marker has exactly one end marker.
> h3. Motivation
> A RegionServer can stop after it writes a mutation to the WAL and before it
> sends the mutation to the standby cluster. The client gets an error for that
> write. The ServerCrashProcedure then replays the WAL on the active cluster.
> {{IndexRegionObserver#preWALRestore}} sends a replayed edit to the standby
> only while the local cluster role is active. Testing found this sequence:
> - A RegionServer stops in this window.
> - A failover starts before the WAL replay.
> - The replay applies the edit on the old active cluster only, so the two
> clusters have different data.
> - A failover back to the old active cluster shows a write that the client
> thinks failed.
> A check for pending ServerCrashProcedures before the failover does not
> prevent this:
> - The replayed edits are still sent while the role is
> {{{}ACTIVE_TO_STANDBY{}}}. They stop only when the cluster moves to
> {{{}STANDBY{}}}, after the peer reaches {{{}ACTIVE_IN_SYNC{}}}. This occurs
> after the check, and the tool does not control it.
> - The check cannot see a RegionServer that has stopped but whose ZooKeeper
> session has not expired yet. This can take 40 s to 90 s, as given by the
> ZooKeeper server settings.
> - The check cannot see a region that the ServerCrashProcedure gives to a
> region procedure that is already attached.
> - Every RegionServer exit gets a ServerCrashProcedure, also a clean stop.
> Thus the check would also fire during a rolling restart.
> h3. Operator steps after a failover
> - Wait for the old active cluster to quiesce, with no pending
> ServerCrashProcedures and no regions in transition.
> - Find and repair the divergence with PhoenixSyncTableTool.
> The markers give the exact time of the failover. Use them to find the related
> logs and to select the time range for PhoenixSyncTableTool. Replayed cells
> keep their original write timestamps, which are before
> {{{}FAILOVER_START{}}}. Thus {{-{-}from-time{-}}} must start before
> {{{}FAILOVER_START{}}}. Set {{-to-time}} explicitly, because the default is
> the current time minus 1 hour.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
