Lokesh Khurana created PHOENIX-8037:
---------------------------------------
Summary: PhoenixHAAdminTool: Do not start a failover when a
ServerCrashProcedure is pending on the active cluster
Key: PHOENIX-8037
URL: https://issues.apache.org/jira/browse/PHOENIX-8037
Project: Phoenix
Issue Type: Sub-task
Reporter: Lokesh Khurana
Assignee: Lokesh Khurana
h3. Problem
A RegionServer (RS) on the active cluster can stop after it writes a mutation
to the HBase WAL and before it sends the mutation to the standby cluster. Thus
the mutation does not get to the standby cluster. The client receives an error
and thinks that the write failed.
The HMaster then starts a ServerCrashProcedure (SCP) for the RS. The SCP
replays the WAL into the regions on the active cluster. If the cluster is still
active at the time of the replay, {{IndexRegionObserver#preWALRestore}} sends
the mutation to the standby cluster. If a failover occurs before the replay,
the mutation does not get to the standby cluster.
If a failover starts while the SCP is pending, this sequence can occur:
# The RS on cluster A (ACTIVE) writes mutation M to the WAL. The RS stops
before it sends M to cluster B (STANDBY).
# The client receives an error for M.
# The operator starts a failover. Cluster A becomes STANDBY. Cluster B becomes
ACTIVE.
# The SCP on cluster A replays M into the local region. M does not get to
cluster B.
# Cluster A has M. Cluster B does not have M. The data on the two clusters is
not the same.
# The operator does a failover back to cluster A. Readers see M on cluster A,
but the client thinks that M failed.
At this time, {{initiate-failover}} checks only the HA group state
({{ACTIVE_IN_SYNC}}). It does not check for a pending SCP. Also,
{{initiate-failover}} has no option to bypass a check. Only {{-g}}, {{-t}} and
{{-h}} are available. The {{-F/--force}} option is available only on the
{{update}} command.
h3. Proposed change
# In {{initiate-failover}}, before the tool starts the failover, connect to the
local (active) cluster with the HBase Admin API.
# Find each {{ServerCrashProcedure}} that is not complete (for example, with
{{Admin#getProcedures()}}).
# If one or more SCPs are pending, do not start the failover. Show the
procedure IDs and the server names. Return {{RET_VALIDATION_ERROR}}.
# Add an override option to {{initiate-failover}}. Use the {{-F/--force}} name,
for consistency with the {{update}} command. With this option, the tool does
not stop the failover for a pending SCP. Use this option for an ungraceful
failover in an emergency.
# When the operator uses the override option and an SCP is pending, show a
warning. The warning tells the operator that some writes can be on the old
active cluster only.
# Update the help text for {{initiate-failover}}.
h3. Out of scope
* The repair of the writes that are on the old active cluster only.
PhoenixSyncTable (PHOENIX-7750) will do this repair in a different JIRA.
* An RS stop that the HMaster has not found yet. The check cannot see this RS
stop.
* An RS stop that occurs after the check is complete.
h3. Tests
* Unit test: the tool parses the override option. The check gives the correct
result with a mock Admin.
* IT: when an SCP is pending on the active cluster, {{initiate-failover}}
returns {{RET_VALIDATION_ERROR}}. The HA group stays in {{ACTIVE_IN_SYNC}}.
* IT: when an SCP is pending and the operator uses the override option, the
failover starts and the tool shows the warning.
* IT: when no SCP is pending, the failover starts as before.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
