This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git
The following commit(s) were added to refs/heads/main by this push:
new 6f41010269 Update documentation about System Tests (#37062)
6f41010269 is described below
commit 6f41010269e4a2c052b564f8f0f977abbaf5dfc7
Author: Jarek Potiuk <[email protected]>
AuthorDate: Sun Jan 28 21:33:39 2024 +0100
Update documentation about System Tests (#37062)
The documentation was a little out-dated.
---
contributing-docs/testing/system_tests.rst | 115 +++++++++--------------------
1 file changed, 35 insertions(+), 80 deletions(-)
diff --git a/contributing-docs/testing/system_tests.rst
b/contributing-docs/testing/system_tests.rst
index a8a1a9fc85..f9add8b377 100644
--- a/contributing-docs/testing/system_tests.rst
+++ b/contributing-docs/testing/system_tests.rst
@@ -20,16 +20,40 @@ Airflow System Tests
System tests need to communicate with external services/systems that are
available
if you have appropriate credentials configured for your tests.
-The system tests derive from the
``tests.test_utils.system_test_class.SystemTests`` class. They should also
-be marked with ``@pytest.marker.system(SYSTEM)`` where ``system`` designates
the system
-to be tested (for example, ``google.cloud``). These tests are skipped by
default.
-You can execute the system tests by providing the ``--system SYSTEM`` flag to
``pytest``. You can
-specify several --system flags if you want to execute tests for several
systems.
+.. contents:: :local:
+
+Purpose of System Tests
+-----------------------
+
+Airflow system tests are pretty special because they serve three purposes:
+
+* they are runnable tests that communicate with external services
+* they are example_dags that users can just copy&paste and use as starting
points for their own DAGs
+* the excerpts from these system tests are used to generate documentation
+
+Old System Tests
+----------------
+
+The system tests derive from the
``tests.test_utils.system_test_class.SystemTests`` class.
+
+Old versions of System tests should also be marked with
``@pytest.marker.system(SYSTEM)`` where ``system``
+designates the system to be tested (for example, ``google.cloud``). These
tests are skipped by default.
+
+For older version of the tests, you can execute the system tests by providing
the
+``--system SYSTEM`` flag to ``pytest``. You can specify several --system flags
if you want to execute
+tests for several systems.
The system tests execute a specified example DAG file that runs the DAG
end-to-end.
-.. contents:: :local:
+New System Tests
+----------------
+
+The new system tests follows
+[AIP-47 New Design of System
Tests](https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-47+New+design+of+Airflow+System+Tests)
+and those system tests do not require separate ``pytest`` flag to be executed,
they also don't need a separate
+class to run - all the code is kept in the system test class that is also an
executable DAG, it is discoverable
+by ``pytest`` and it is executable as Python script.
Environment for System Tests
----------------------------
@@ -74,13 +98,8 @@ run Google Cloud system tests.
RANDOM_POSTFIX=$(cat "${RANDOM_FILE}")
-
-To execute system tests, specify the ``--system SYSTEM``
-flag where ``SYSTEM`` is a system to run the system tests for. It can be
repeated.
-
-
Forwarding Authentication from the Host
-----------------------------------------------------
+---------------------------------------
For system tests, you can also forward authentication from the host to your
Breeze container. You can specify
the ``--forward-credentials`` flag when starting Breeze. Then, it will also
forward the most commonly used
@@ -94,75 +113,11 @@ Currently forwarded credentials are:
* credentials stored in ``${HOME}/.docker`` for ``docker`` client
* credentials stored in ``${HOME}/.snowsql`` for ``snowsql`` - SnowSQL
(Snowflake CLI client)
-Adding a New System Test
---------------------------
-
-We are working on automating system tests execution (AIP-4) but for now,
system tests are skipped when
-tests are run in our CI system. But to enable the test automation, we
encourage you to add system
-tests whenever an operator/hook/sensor is added/modified in a given system.
-
-* To add your own system tests, derive them from the
- ``tests.test_utils.system_tests_class.SystemTest`` class and mark with the
- ``@pytest.mark.system(SYSTEM_NAME)`` marker. The system name should follow
the path defined in
- the ``providers`` package (for example, the system tests from
``tests.providers.google.cloud``
- package should be marked with ``@pytest.mark.system("google.cloud")``.
-
-* If your system tests need some credential files to be available for an
- authentication with external systems, make sure to keep these credentials in
the
- ``files/airflow-breeze-config/keys`` directory. Mark your tests with
- ``@pytest.mark.credential_file(<FILE>)`` so that they are skipped if such a
credential file is not there.
- The tests should read the right credentials and authenticate them on their
own. The credentials are read
- in Breeze from the ``/files`` directory. The local "files" folder is mounted
to the "/files" folder in Breeze.
-
-* If your system tests are long-running ones (i.e., require more than 20-30
minutes
- to complete), mark them with the ```@pytest.markers.long_running`` marker.
- Such tests are skipped by default unless you specify the ``--long-running``
flag to pytest.
-
-* The system test itself (python class) does not have any logic. Such a test
runs
- the DAG specified by its ID. This DAG should contain the actual DAG logic
- to execute. Make sure to define the DAG in
``providers/<SYSTEM_NAME>/example_dags``. These example DAGs
- are also used to take some snippets of code out of them when documentation
is generated. So, having these
- DAGs runnable is a great way to make sure the documentation is describing a
working example. Inside
- your test class/test method, simply use
``self.run_dag(<DAG_ID>,<DAG_FOLDER>)`` to run the DAG. Then,
- the system class will take care about running the DAG. Note that the
DAG_FOLDER should be
- a subdirectory of the ``tests.test_utils.AIRFLOW_MAIN_FOLDER`` +
``providers/<SYSTEM_NAME>/example_dags``.
-
-
-A simple example of a system test is available in:
-
-``tests/providers/google/cloud/operators/test_compute_system.py``.
-
-It runs two DAGs defined in
``airflow.providers.google.cloud.example_dags.example_compute.py``.
-
-
-The typical system test session
--------------------------------
-
-Here is the typical session that you need to do to run system tests:
-
-1. Enter breeze
-
-.. code-block:: bash
-
- breeze down
- breeze --python 3.8 --db-reset --forward-credentials
-
-This will:
-
-* stop the whole environment (i.e. recreates metadata database from the
scratch)
-* run Breeze with:
-
- * python 3.8 version
- * resetting the Airflow database
- * forward your local credentials to Breeze
-
-3. Run the tests:
-
-.. code-block:: bash
+Running the System Tests
+------------------------
- pytest -o faulthandler_timeout=2400 \
- --system=google
tests/providers/google/cloud/operators/test_compute_system.py
+Running the tests and developing tests in `Test documentation
<../../tests/system/README.md>`__
------
+------
For other kinds of tests look at `Testing document <../09_testing.rst>`__
