This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
The following commit(s) were added to refs/heads/v1-10-test by this push:
new 51fd6d7 Removes stable tests from quarantine (#10768)
51fd6d7 is described below
commit 51fd6d7c3248ae085ae4a3b2e25458dc0398d40a
Author: Jarek Potiuk <[email protected]>
AuthorDate: Tue Sep 8 07:36:12 2020 +0200
Removes stable tests from quarantine (#10768)
We've observed the tests for last couple of weeks and it seems
most of the tests marked with "quarantine" marker are succeeding
in a stable way (https://github.com/apache/airflow/issues/10118)
The removed tests have success ratio of > 95% (20 runs without
problems) and this has been verified a week ago as well,
so it seems they are rather stable.
There are literally few that are either failing or causing
the Quarantined builds to hang. I manually reviewed the
master tests that failed for last few weeks and added the
tests that are causing the build to hang.
Seems that stability has improved - which might be casued
by some temporary problems when we marked the quarantined builds
or too "generous" way of marking test as quarantined, or
maybe improvement comes from the #10368 as the docker engine
and machines used to run the builds in GitHub experience far
less load (image builds are executed in separate builds) so
it might be that resource usage is decreased. Another reason
might be Github Actions stability improvements.
Or simply those tests are more stable when run isolation.
We might still add failing tests back as soon we see them behave
in a flaky way.
The remaining quarantined tests that need to be fixed:
* test_local_run (often hangs the build)
* test_retry_handling_job
* test_clear_multiple_external_task_marker
* test_should_force_kill_process
* test_change_state_for_tis_without_dagrun
* test_cli_webserver_background
We also move some of those tests to "heisentests" category
Those testst run fine in isolation but fail
the builds when run with all other tests:
* TestImpersonation tests
We might find that those heisentest can be fixed but for
now we are going to run them in isolation.
Also - since those quarantined tests are failing more often
the "num runs" to track for those has been decreased to 10
to keep track of 10 last runs only.
(cherry picked from commit b746f33fc66ce2ecdc6d72a9943fc2db00da0f45)
---
TESTING.rst | 12 ++++++++++++
tests/conftest.py | 23 ++++++++++++++++++++---
tests/core/test_impersonation_tests.py | 2 +-
3 files changed, 33 insertions(+), 4 deletions(-)
diff --git a/TESTING.rst b/TESTING.rst
index 296ef96..ae7fd23 100644
--- a/TESTING.rst
+++ b/TESTING.rst
@@ -415,6 +415,18 @@ Those tests are marked with ``@pytest.mark.quarantined``
annotation.
Those tests are skipped by default. You can enable them with
``--include-quarantined`` flag. You
can also decide to only run tests with ``-m quarantined`` flag to run only
those tests.
+Heisen tests
+------------
+
+Some of our tests are Heisentests. This means that they run fine in isolation
but when they run together with
+others they might fail the tests (this is likely due to resource
consumptions). Therefore we run those tests
+in isolation.
+
+Those tests are marked with ``@pytest.mark.heisentests`` annotation.
+Those tests are skipped by default. You can enable them with
``--include-heisentests`` flag. You
+can also decide to only run tests with ``-m heisentests`` flag to run only
those tests.
+
+
Running Tests with Kubernetes
=============================
diff --git a/tests/conftest.py b/tests/conftest.py
index e7de7fc..13dc670 100644
--- a/tests/conftest.py
+++ b/tests/conftest.py
@@ -1,4 +1,3 @@
-
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
@@ -98,6 +97,11 @@ def pytest_addoption(parser):
action="store_true",
help="Includes quarantined tests (marked with quarantined marker).
They are skipped by default.",
)
+ group.addoption(
+ "--include-heisentests",
+ action="store_true",
+ help="Includes heisentests (marked with heisentests marker). They are
skipped by default.",
+ )
def initial_db_init():
@@ -171,12 +175,15 @@ def pytest_configure(config):
"markers", "system(name): mark test to run with named system"
)
config.addinivalue_line(
- "markers", "long_running(name): mark test that run for a long time
(many minutes)"
+ "markers", "long_running: mark test that run for a long time (many
minutes)"
)
config.addinivalue_line(
"markers", "quarantined: mark test that are in quarantine (i.e. flaky,
need to be isolated and fixed)"
)
config.addinivalue_line(
+ "markers", "heisentests: mark test that should be run in isolation due
to resource consumption"
+ )
+ config.addinivalue_line(
"markers", "credential_file(name): mark tests that require credential
file in CREDENTIALS_DIR"
)
@@ -217,7 +224,7 @@ def skip_if_not_marked_with_system(selected_systems, item):
def skip_system_test(item):
for marker in item.iter_markers(name="system"):
pytest.skip("The test is skipped because it has system marker. "
- "System tests are only run when --systems flag "
+ "System tests are only run when --system flag "
"with the right system ({system}) is passed to pytest.
{item}".
format(system=marker.args[0], item=item))
@@ -236,6 +243,13 @@ def skip_quarantined_test(item):
format(item=item))
+def skip_heisen_test(item):
+ for _ in item.iter_markers(name="heisentests"):
+ pytest.skip("The test is skipped because it has heisentests marker. "
+ "And --include-heisentests flag is passed to pytest.
{item}".
+ format(item=item))
+
+
def skip_if_integration_disabled(marker, item):
integration_name = marker.args[0]
environment_variable_name = "INTEGRATION_" + integration_name.upper()
@@ -277,6 +291,7 @@ def pytest_runtest_setup(item):
include_long_running = item.config.getoption("--include-long-running")
include_quarantined = item.config.getoption("--include-quarantined")
+ include_heisentests = item.config.getoption("--include-heisentests")
for marker in item.iter_markers(name="integration"):
skip_if_integration_disabled(marker, item)
@@ -295,4 +310,6 @@ def pytest_runtest_setup(item):
skip_long_running_test(item)
if not include_quarantined:
skip_quarantined_test(item)
+ if not include_heisentests:
+ skip_heisen_test(item)
skip_if_credential_file_missing(item)
diff --git a/tests/core/test_impersonation_tests.py
b/tests/core/test_impersonation_tests.py
index 9539dac..798082d 100644
--- a/tests/core/test_impersonation_tests.py
+++ b/tests/core/test_impersonation_tests.py
@@ -109,7 +109,7 @@ def create_user():
)
[email protected]
[email protected]
class TestImpersonation(unittest.TestCase):
def setUp(self):
