[
https://issues.apache.org/jira/browse/IMPALA-12525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17784267#comment-17784267
]
ASF subversion and git services commented on IMPALA-12525:
----------------------------------------------------------
Commit 44c85e85a51bad4faca95f8771b2c2c5a686ca90 in impala's branch
refs/heads/master from wzhou-code
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=44c85e85a ]
IMPALA-12525: Fix flaky test test_statestored_manual_failover
In test_statestored_manual_failover, statestore service failover is not
triggered sometimes when the network of active statestored is disabled
after manually forced failover.
During test, the network of active statestored could be disabled before
all subscribers re-registered with restarted statestored. This caused
some subscribers to not receive the notification of active statestored
change so that they could not correctly report connection states for
the requests from standby statestored.
This patch made following changes:
1) Updated the test case test_statestored_manual_failover to disable
the network of active statestored after all subscribers re-registering
with the restarted statestored.
2) Defined a new mutex active_lock_ in class StatestoreStub to protect
is_active_ since the mutex lock_ could be held for long time if the
subscriber lose the connection with statestored and enter recovery
mode.
3) Found one case that was not handled on Statestore subscribers. The
subscribers could be started before both statestore instances are
ready to accept registration requests. This caused impalad hit DCHECK.
Changed code to handle this case in this patch.
Added test cases to inject a real delay in statestored startup and
verify impalads and catalogd are able to tolerate this delay.
4) Updated address of active catalogd in the metrics of statestored
after statestore service failover.
5) Another test test_statestored_auto_failover_with_disabling_network
failed occasionally due to delay of HA Handshake RPC between two
statestore instances. The issue is tracked with IMPALA-12550. The last
two lines of the test are commented out temporarily.
Testing:
- Repeatedly ran test_statestored_manual_failover on Jenkins for
hundreds of times.
- Repeatedly ran test_statestored_manual_failover on local machine for
thousand times without failure.
- Passed core tests
Change-Id: If03bf09d22a2875d2c1eec8a4f62eeefc5d855dc
Reviewed-on: http://gerrit.cloudera.org:8080/20657
Reviewed-by: Riza Suminto <[email protected]>
Reviewed-by: Michael Smith <[email protected]>
Tested-by: Impala Public Jenkins <[email protected]>
> statestore.active-status did not reach value True in 120s
> ---------------------------------------------------------
>
> Key: IMPALA-12525
> URL: https://issues.apache.org/jira/browse/IMPALA-12525
> Project: IMPALA
> Issue Type: Bug
> Reporter: Fang-Yu Rao
> Assignee: Wenzhe Zhou
> Priority: Major
> Labels: broken-build, flaky-test
>
> We found that it's possible that
> [statestore.active-status|https://github.com/apache/impala/blob/master/tests/custom_cluster/test_statestored_ha.py#L452]
> could not reach value True in 120s.
> *+Error Message+*
> {code:java}
> AssertionError: Metric statestore.active-status did not reach value True in
> 120s. Dumping debug webpages in JSON format... Dumped memz JSON to
> $IMPALA_HOME/logs/metric_timeout_diags_20231026_01:53:53/json/memz.json
> Dumped metrics JSON to
> $IMPALA_HOME/logs/metric_timeout_diags_20231026_01:53:53/json/metrics.json
> Dumped queries JSON to
> $IMPALA_HOME/logs/metric_timeout_diags_20231026_01:53:53/json/queries.json
> Dumped sessions JSON to
> $IMPALA_HOME/logs/metric_timeout_diags_20231026_01:53:53/json/sessions.json
> Dumped threadz JSON to
> $IMPALA_HOME/logs/metric_timeout_diags_20231026_01:53:53/json/threadz.json
> Dumped rpcz JSON to
> $IMPALA_HOME/logs/metric_timeout_diags_20231026_01:53:53/json/rpcz.json
> Dumping minidumps for impalads/catalogds... Dumped minidump for Impalad PID
> 32539 Dumped minidump for Impalad PID 32543 Dumped minidump for Impalad PID
> 32550 Dumped minidump for Catalogd PID 32460
> {code}
> *+Stacktrace+*
> {code:java}
> custom_cluster/test_statestored_ha.py:500: in test_statestored_manual_failover
> self.__test_statestored_manual_failover(second_failover=True)
> custom_cluster/test_statestored_ha.py:452: in
> __test_statestored_manual_failover
> "statestore.active-status", expected_value=True, timeout=120)
> common/impala_service.py:144: in wait_for_metric_value
> self.__metric_timeout_assert(metric_name, expected_value, timeout)
> common/impala_service.py:213: in __metric_timeout_assert
> assert 0, assert_string
> E AssertionError: Metric statestore.active-status did not reach value True
> in 120s.
> E Dumping debug webpages in JSON format...
> E Dumped memz JSON to
> $IMPALA_HOME/logs/metric_timeout_diags_20231026_01:53:53/json/memz.json
> E Dumped metrics JSON to
> $IMPALA_HOME/logs/metric_timeout_diags_20231026_01:53:53/json/metrics.json
> E Dumped queries JSON to
> $IMPALA_HOME/logs/metric_timeout_diags_20231026_01:53:53/json/queries.json
> E Dumped sessions JSON to
> $IMPALA_HOME/logs/metric_timeout_diags_20231026_01:53:53/json/sessions.json
> E Dumped threadz JSON to
> $IMPALA_HOME/logs/metric_timeout_diags_20231026_01:53:53/json/threadz.json
> E Dumped rpcz JSON to
> $IMPALA_HOME/logs/metric_timeout_diags_20231026_01:53:53/json/rpcz.json
> E Dumping minidumps for impalads/catalogds...
> E Dumped minidump for Impalad PID 32539
> E Dumped minidump for Impalad PID 32543
> E Dumped minidump for Impalad PID 32550
> E Dumped minidump for Catalogd PID 32460
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
