Petr Fedchenkov created IMPALA-15596:
----------------------------------------
Summary: Unregistered statestore stub takes the RPCs of the active
statestored and swaps the statestored roles
Key: IMPALA-15596
URL: https://issues.apache.org/jira/browse/IMPALA-15596
Project: IMPALA
Issue Type: Bug
Components: Backend
Affects Versions: Impala 4.5.0
Reporter: Petr Fedchenkov
With statestored HA, a {{StatestoreStub}} that has not registered with its
statestored has a zero statestore ID, and
{{StatestoreStub::IsMatchingStatestoreId()}} treats a zero ID as matching any
statestored. This is meant for the window when an RPC from the statestored
arrives before the registration response. The code is the same on master.
If a subscriber starts while one statestored, say {{statestoreds[0]}}, is down,
its registration with {{statestoreds[0]}} fails and the stub stays
unregistered. The subscriber registers with {{statestoreds[1]}}. From then on
the heartbeats, topic updates and {{UpdateStatestoredRole}} RPCs of
{{statestoreds[1]}} can be matched to the unregistered stub of
{{statestoreds[0]}}:
* the registered stub of {{statestoreds[1]}} misses its heartbeats, enters
recovery mode after every {{--statestore_subscriber_timeout_seconds}} and
re-registers;
* each re-registration and role update swaps the active and standby
statestoreds of the subscriber, so it flips between following the active and
the standby statestored.
While the subscriber follows the standby statestored, it applies no topic or
catalogd updates, because the standby sends none. In one case on Kubernetes the
role swap waited behind the recovery loop of the first stub and landed after it
registered with the restarted standby: the subscriber then followed the standby
for good, with no membership updates and no catalogd failover notifications,
while it stayed Ready, until it was restarted.
Reproduced on a minicluster: kill {{statestoreds[0]}}, restart one impalad, and
its log shows "Receive {{UpdateStatestoredRole}} message from standby
statestored" 3 times in 10 s with
{{--statestore_subscriber_timeout_seconds=2}}. On Kubernetes we saw the same
with the default timeout: the subscriber flipped every 30 s for as long as
{{statestoreds[0]}} refused connections.
Proposed fix: apply the zero-ID wildcard only while a {{RegisterSubscriber}}
RPC of the stub is in flight, and let an exact ID match on the other stub win
over the wildcard. Patch with a custom cluster test to follow.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
