Petr Fedchenkov created IMPALA-15597:
----------------------------------------
Summary: Restart or failover of a statestored can switch the
active catalogd
Key: IMPALA-15597
URL: https://issues.apache.org/jira/browse/IMPALA-15597
Project: IMPALA
Issue Type: Bug
Components: Catalog, Backend
Affects Versions: Impala 4.5.0
Reporter: Petr Fedchenkov
With statestored HA and catalogd HA, each statestored designates the active
catalogd on its own, from the catalogd registrations it receives
({{StatestoreCatalogdMgr::RegisterCatalogd()}}). The statestored HA handshake
and heartbeat carry no catalogd information, and {{TCatalogRegistration}} does
not say which catalogd is active now. So a statestored that restarts designates
the active catalogd from scratch: the first catalogd to register, or the one
with the higher priority if both register within
{{--catalogd_ha_preemption_wait_period_ms}}. If that statestored is or becomes
the active one, it sends {{UpdateCatalogd}} with its own choice, and the active
catalogd switches although it was healthy. The new active catalogd may then
reset its metadata ({{--catalogd_ha_reset_metadata_on_failover}}), and DDLs in
flight meet a catalogd failover. The code is the same on master.
{{--use_subscriber_id_as_catalogd_priority}} does not help: it makes both
statestoreds pick the same catalogd, which switches back whenever the other one
was active. On Kubernetes, deleting the active statestored pod switched the
active catalogd 1 time in 4 by default and 2 times in 4 with that flag.
Reproduced on a minicluster: make {{catalogds[1]}} active by killing
{{catalogds[0]}} and restarting it, restart the standby statestored, then kill
the active statestored. With {{--use_subscriber_id_as_catalogd_priority=true}}
the restarted statestored takes over with {{catalogds[0]}} as active.
Two ways to fix it:
# The catalogd tells a new statestored instance that it is active. The active
catalogd registers with {{force_catalogd_active=true}} when it registers with a
statestored instance it has not registered with before (a new statestore ID in
the {{GetProtocolVersion}} response). It does not do so if it follows no
statestored as active or the one it follows is unreachable, since it may then
have been demoted without knowing it. This needs no Thrift change, and the
registration paths a new statestored instance takes already honour the field.
It also covers a single statestored that restarts.
# The statestoreds share the designation. The active statestored sends the
active catalogd and its version in the HA heartbeat or handshake, and the
standby takes it over instead of designating its own. This keeps the decision
in the statestore, but it needs a Thrift change and does not cover a single
statestored, or both statestoreds, restarting.
A patch for option 1 with a custom cluster test,
{{test_active_catalogd_kept_after_statestored_restart}}, which fails without
the patch and passes with it, will be posted shortly. Views on the approach are
welcome.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
