michalslowikowski00 commented on a change in pull request #7774: [AIRFLOW-XXXX]
document system tests mechanism better
URL: https://github.com/apache/airflow/pull/7774#discussion_r395526260
##########
File path: TESTING.rst
##########
@@ -543,31 +587,197 @@ tests whenever an operator/hook/sensor is
added/modified in a given system.
the system class will take care about running the DAG. Note that the
DAG_FOLDER should be
a subdirectory of the ``tests.test_utils.AIRFLOW_MAIN_FOLDER`` +
``providers/<SYSTEM_NAME>/example_dags``.
-An example of a system test is available in:
-``airflow.tests.providers.google.operators.test_natunal_language_system.CloudNaturalLanguageExampleDagsTest``.
+A simple example of a system test is available in:
+
+``tests/providers/google/cloud/operators/test_compute_system.py``.
+
+It runs two DAGs defined in
``airflow.providers.google.cloud.example_dags.example_compute.py`` and
+``airflow.providers.google.cloud.example_dags.example_compute_igm.py``.
+
+Preparing backport packages for System Tests for Airflow 1.10.* series
+----------------------------------------------------------------------
+
+In order to run system tests with old Airflow version you need to prepare
backport packages. This
+can be done by running ``./scripts/ci/ci_prepare_bacport_packages.sh <PACKAGES
TO BUILD>``. For
+example the below command will build google postgres and mysql packages:
+
+.. code-block:: bash
+
+ ./scripts/ci/ci_prepare_bacport_packages.sh google postgres mysql
+
+Those packages will be prepared in ./dist folder. This folder is mapped to
/dist folder
+when you enter Breeze, so it is easy to automate installing of those packages
for testing.
-It runs the DAG defined in
``airflow.providers.google.cloud.example_dags.example_natural_language.py``.
-Running Tests for Older Airflow Versions
-----------------------------------------
+Installing backported for Airflow 1.10.* series
+-----------------------------------------------
The tests can be executed against the master version of Airflow but they also
work
with older versions. This is especially useful to test back-ported operators
from Airflow 2.0 to 1.10.* versions.
To run the tests for Airflow 1.10.* series, you need to run Breeze with
-``--install-airflow-version==<VERSION>`` to install a different version of
Airflow.
+``--install-airflow-version=<VERSION>`` to install a different version of
Airflow.
If ``current`` is specified (default), then the current version of Airflow is
used.
Otherwise, the released version of Airflow is installed.
-The commands make sure that the source version of master Airflow is removed
and the released version of
-Airflow from ``Pypi`` is installed. Note that tests sources are not removed
and they can be used
-to run tests (unit tests and system tests) against the freshly installed
version.
+The ``-install-airflow-version=<VERSION>`` command make sure that the current
(from sources) version of
+Airflow is removed and the released version of Airflow from ``Pypi`` is
installed. Note that tests sources
+are not removed and they can be used to run tests (unit tests and system
tests) against the
+freshly installed version.
+
+You should automate installing of the backport packages in your own
+``./files/airflow-breeze-config/variables.env`` file. You should make it
depend on
+``RUN_AIRFLOW_1_10`` variable value equals to "true" so that
+installation of backport packages is only performed when you install airflow
1.10.*.
+The backport packages are available in ``/dist`` directory if they were
prepared as described
+in the previous chapter.
+
+Typically the command in you variables.env file will be similar to:
+
+.. code-block:: bash
+
+ # install any packages from dist folder if they are available
+ if [[ ${RUN_AIRFLOW_1_10:=} == "true" ]]; then
+ pip install /dist/apache_airflow_providers_{google,postgres,mysql}*.whl
|| true
+ fi
+
+The command above will automatically install backported google, postgres amd
mysql packages if they
+were prepared before entering breeze.
+
+
+Running system tests for backported packages in Airflow 1.10.* series
+---------------------------------------------------------------------
+
+Once you installed 1.10.* Airflow version with ``--install-airflow-version``
and prepared and
+installed the required packages via ``variables.env`` it should be as easy as
running
+``pytest --systems=<SYSTEM_NAME> TEST_NAME``. Note that we have default
timeout for runnning
+system tests set to 8 minutes and some system tests might take much longer to
run and you might
+want to add ``-o faulthandler_timeout=2400`` (2400s = 40 minutes for example)
to your
+pytest command.
+
+Typical system test session
+---------------------------
+
+Here is the typical session that you need to do to run system tests:
+
+1. Prepare backport packages
+
+.. code-block:: bash
+
+ ./scripts/ci/ci_prepare_bacport_packages.sh google postgres mysql
+
+2. Enter breeze with installing Airflow 1.10.*, forwarding credentials and
installing
+ backported packages (you need appropriate line in
``./files/airflow-breeze-config/variables.env``)
+
+.. code-block:: bash
+
+ ./breeze --install-airflow-version 1.10.9 --python 3.6 --db-reset
--forward-credentials restart
+
+This will:
+
+* install Airflow 1.10.9
+* restarts the whole environment (i.e. recreates metadata database from the
scratch)
+* run Breeze with python 3.6 version
+* reset the Airflow database
+* forward your local credentials to Breeze
+
+3. Run the tests:
+
+.. code-block:: bash
+
+ pytest -o faulthandler_timeout=2400 --systems=google
tests/providers/google/cloud/operators/test_compute_system.py
+
+
+Iteration with System Tests if your resources are slow to create
+----------------------------------------------------------------
+
+When you want to iterate on system tests, you might want to create slow
resources first
+
+If you need to setup some external resources for your tests (for example
compute instances in Google Cloud)
+you should set them up and teardown in the setUp/tearDown methods of your
tests.
+Since those resources might be slow to create you might want to add some
helpers that
+set them up and tear them down separately via manual operations so that you
can iterate on
+the tests without waiting for setUp and tearDown with every test.
+
+In this case you should build in a mechanism to skip setUp and tearDown in
case you manually
+created the resources. Somewhat complex example of that can be found in
+``tests.providers.google.cloud.operators.test_cloud_sql_system.py`` and the
helper is
+available in
``tests.providers.google.cloud.operators.test_cloud_sql_system_helper.py``.
+
+When the helper is run with ``--action create`` to create cloud sql instances
which are very slow
+to create and set-up so that you can iterate on running the system tests
without
+loosing the time for creating theme every time. A temporary file is created to
prevent from
+setting up and tearing down the instances when running the test.
+
+This example also shows how you can use the random number generated at the
entry of Breeze if you
+have it in your variables.env (see previous chapter). In case of Cloud SQL you
cannot reuse the
+same instance name for a week so we generate random number that is used across
the whole session
+and store it in ``/random.txt`` file so that the names are unique during tests.
+
+NOTE! Do not forget to delete manually created resources before leaving the
Breeze session. They
+are usually expensive to run.
+
+Note that in case you have to update your backported operators or system tests
(they are part of
+the backport packageS) you need to rebuild the packages outside of breeze and
+``pip remove/pip install`` those packages in order to get them installed. This
is not needed
+if you run system tests with ``current`` airflow version, so it is better to
iterate with the
+system tests with the ``current`` version and fix all problems there and only
afterwards run
+the tests with Airflow 1.10.*
+
+Typical session then looks as follows:
+
+1. Prepare backport packages
+
+.. code-block:: bash
+
+ ./scripts/ci/ci_prepare_bacport_packages.sh google postgres mysql
+
+2. Enter breeze with installing Airflow 1.10.*, forwarding credentials and
installing
+ backported packages (you need appropriate line in
``./files/airflow-breeze-config/variables.env``)
+
+.. code-block:: bash
+
+ ./breeze --install-airflow-version 1.10.9 --python 3.6 --db-reset
--forward-credentials restart
+
+3. Run create action in helper (to create slowly created resources):
+
+.. code-block:: bash
+
+ python
tests/providers/google/cloud/operators/test_cloud_sql_system_helper.py --action
create
+
+4. Run the tests:
+
+.. code-block:: bash
+
+ pytest -o faulthandler_timeout=2400 --systems=google
tests/providers/google/cloud/operators/test_compute_system.py
+
+5. In case you are running backport packages tests you need to rebuild and
reinstall a package
+ every time you change the operators/hooks or example_dags. The example
below shows reinstallation
+ of the google package:
+
+In the host:
+
+.. code-block:: bash
+
+ ./scripts/ci/ci_prepare_backport_packages.sh google
+
+In the container (:
Review comment:
Typo? `(:`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
