Stephan Erb created AURORA-1972: ----------------------------------- Summary: Flapping Excutor HealthChecker Test Key: AURORA-1972 URL: https://issues.apache.org/jira/browse/AURORA-1972 Project: Aurora Issue Type: Bug Components: Executor Reporter: Stephan Erb
We currently have a set of flapping HealthChecking test that prevent our builds from passing (e.g. see https://reviews.apache.org/r/65565/). {code} ==================== FAILURES ==================== TestThreadedHealthCheckerWithDefaults.test_run_unhealthy_after_callback self = <apache.aurora.executor.common.test_health_checker.TestThreadedHealthCheckerWithDefaults testMethod=test_run_unhealthy_after_callback> mock_sleep = <MagicMock name='sleep' spec='builtin_function_or_method' id='4598488912'> @mock.patch('apache.aurora.executor.common.health_checker.time.sleep', spec=time.sleep) def test_run_unhealthy_after_callback(self, mock_sleep): mock_sleep.return_value = None health_status = [(True, None), (True, None), (False, 'failure-4'), (False, 'failure-5')] self.health.side_effect = lambda: health_status.pop(0) mock_is_set = mock.Mock(spec=threading._Event.is_set) liveness = [False, False, False, False, True] mock_is_set.side_effect = lambda: liveness.pop(0) self.health_checker.threaded_health_checker.dead.is_set = mock_is_set self.health_checker.threaded_health_checker.run() > assert mock_sleep.call_count == 4 E AssertionError: assert 9403 == 4 E + where 9403 = <MagicMock name='sleep' spec='builtin_function_or_method' id='4598488912'>.call_count .pants.d/pyprep/sources/365105c9a0472d6a1d7576426d316fe2aa7dcc77/apache/aurora/executor/common/test_health_checker.py:1292: AssertionError {code} Please notice the huge difference between actual and expected calls. This works as expected: {code} ./pants --cache-ignore --no-test-pytest-fast test.pytest src/test/python:: {code} This triggers the problem with traces as the one posted above: {code} ./pants --cache-ignore --no-test-pytest-fast test.pytest src/test/python:: {code} The flapping seems to be dependent on how pants executes the tests. This seems to have a side effect on how {{time.sleep}} mocking is performed. The relevant [test class|https://github.com/apache/aurora/blob/c85bffdd6f68312261697eee868d57069adda434/src/test/python/apache/aurora/executor/common/test_health_checker.py#L1033] and the [failing tests|https://github.com/apache/aurora/blob/c85bffdd6f68312261697eee868d57069adda434/src/test/python/apache/aurora/executor/common/test_health_checker.py#L1265-L1280] in particular are somewhat low-quality as they make heavy use of mocking. How do we want to proceed here? -- This message was sent by Atlassian JIRA (v7.6.3#76005)