o-nikolas commented on code in PR #31181:
URL: https://github.com/apache/airflow/pull/31181#discussion_r1190482268
##########
TESTING.rst:
##########
@@ -450,6 +450,69 @@ This prepares airflow .whl package in the dist folder.
breeze --use-airflow-version wheel --use-packages-from-dist
--skip-mounting-local-sources
+Airflow Docker Compose Tests
+============================
+
+Running Docker Compose Tests with breeze
+----------------------------------------
+
+We also test in CI whether docker compose that we expose in our documentation
via
+`Running Airflow in Docker
<https://airflow.apache.org/docs/apache-airflow/stable/howto/docker-compose/index.html>`_
+works as expected. Those tests are run in CI ("Test docker-compose quick
start")
+and you can run them locally as well.
+
+The way how the tests work:
Review Comment:
```suggestion
The way the tests work:
```
##########
TESTING.rst:
##########
@@ -450,6 +450,69 @@ This prepares airflow .whl package in the dist folder.
breeze --use-airflow-version wheel --use-packages-from-dist
--skip-mounting-local-sources
+Airflow Docker Compose Tests
+============================
+
+Running Docker Compose Tests with breeze
+----------------------------------------
+
+We also test in CI whether docker compose that we expose in our documentation
via
+`Running Airflow in Docker
<https://airflow.apache.org/docs/apache-airflow/stable/howto/docker-compose/index.html>`_
+works as expected. Those tests are run in CI ("Test docker-compose quick
start")
+and you can run them locally as well.
+
+The way how the tests work:
+
+1. They first build airflow production image
+2. Then they take the docker compose file of ours and use the image to start it
+3. Then they perform a simple DAG trigger tests that checks whether airflow is
up and processes simple DAG
+
+This is done in local environment, not in the Breeze CI image. It uses
``COMPOSE_PROJECT_NAME`` set to
+``quick-start`` to avoid conflicts with other docker compose deployments you
might have. and the whole
+test can be also run manually with ``pytest
docker_tests/test_docker_compose_quick_start.py`` command,
+providing that ``DOCKER_IMAGE`` environment variable is set to the image you
want to test.
+
+The complete test can be performed using breeze. The prerequisite to that
+is to have ``docker-compose`` (docker compose v1) or ``docker compose`` plugin
(docker compose v2)
+available on the path.
+
+Running complete test with breeze:
+
+.. code-block:: bash
+
+ breeze prod-image build --python 3.7
+ breeze testing docker-compose-tests
+
+In case the test fails, it will dump the logs from the running containers to
the console and it
+will shutdown the docker compose deployment. In case you want to debug the
docker-compose deployment
+created for the test, you can export ``SKIP_DOCKER_COMPOSE_DELETION`` variable
and the deployment
+will not be deleted after the test. If you want to debug the deployment using
``docker compose`` commands
+you should set ``COMPOSE_PROJECT_NAME`` to quick-start because this is what
the test uses:
+
+.. code-block:: bash
+
+ export COMPOSE_PROJECT_NAME=quick-start
+
+When the test is re-run it will automatically stop previous deployment and
start a new one.
+
+Running Docker Compose deployment manually
+------------------------------------------
+
+You can also (independently of Pytest test) run docker-compose deployment
manually with the image you built using
+the prod image build command above.
+
+.. code-block:: bash
+
+ export AIRFLOW_IMAGE_NAME=ghcr.io/apache/airflow/main/prod/python3.7:latest
+
+and follow the instructions in the
+`Running Airflow in Docker
<https://airflow.apache.org/docs/apache-airflow/stable/howto/docker-compose/index.html>`_
+but make sure to use the docker-compose file from the sources in
+``docs/apache-airflow/stable/howto/docker-compose/`` folder.
+
+Then, the usual ``docker compose`` and ``docker`` commands can be used to
debug such running instance.
+The test performs a simple API call to trigger a DAG and wait for it, but you
can follow our
+documentation to connect to such running docker compose instance and test it
manually.
Review Comment:
```suggestion
Then, the usual ``docker compose`` and ``docker`` commands can be used to
debug such running instances.
The test performs a simple API call to trigger a DAG and wait for it, but
you can follow our
documentation to connect to such running docker compose instances and test
it manually.
```
##########
TESTING.rst:
##########
@@ -450,6 +450,69 @@ This prepares airflow .whl package in the dist folder.
breeze --use-airflow-version wheel --use-packages-from-dist
--skip-mounting-local-sources
+Airflow Docker Compose Tests
+============================
+
+Running Docker Compose Tests with breeze
+----------------------------------------
+
+We also test in CI whether docker compose that we expose in our documentation
via
Review Comment:
```suggestion
We also test in CI whether the Docker Compose that we expose in our
documentation via
```
##########
TESTING.rst:
##########
@@ -450,6 +450,69 @@ This prepares airflow .whl package in the dist folder.
breeze --use-airflow-version wheel --use-packages-from-dist
--skip-mounting-local-sources
+Airflow Docker Compose Tests
+============================
+
+Running Docker Compose Tests with breeze
+----------------------------------------
+
+We also test in CI whether docker compose that we expose in our documentation
via
+`Running Airflow in Docker
<https://airflow.apache.org/docs/apache-airflow/stable/howto/docker-compose/index.html>`_
+works as expected. Those tests are run in CI ("Test docker-compose quick
start")
+and you can run them locally as well.
+
+The way how the tests work:
+
+1. They first build airflow production image
+2. Then they take the docker compose file of ours and use the image to start it
+3. Then they perform a simple DAG trigger tests that checks whether airflow is
up and processes simple DAG
+
+This is done in local environment, not in the Breeze CI image. It uses
``COMPOSE_PROJECT_NAME`` set to
+``quick-start`` to avoid conflicts with other docker compose deployments you
might have. and the whole
+test can be also run manually with ``pytest
docker_tests/test_docker_compose_quick_start.py`` command,
+providing that ``DOCKER_IMAGE`` environment variable is set to the image you
want to test.
+
+The complete test can be performed using breeze. The prerequisite to that
+is to have ``docker-compose`` (docker compose v1) or ``docker compose`` plugin
(docker compose v2)
+available on the path.
+
+Running complete test with breeze:
+
+.. code-block:: bash
+
+ breeze prod-image build --python 3.7
+ breeze testing docker-compose-tests
+
+In case the test fails, it will dump the logs from the running containers to
the console and it
+will shutdown the docker compose deployment. In case you want to debug the
docker-compose deployment
+created for the test, you can export ``SKIP_DOCKER_COMPOSE_DELETION`` variable
and the deployment
+will not be deleted after the test. If you want to debug the deployment using
``docker compose`` commands
+you should set ``COMPOSE_PROJECT_NAME`` to quick-start because this is what
the test uses:
Review Comment:
```suggestion
will shutdown the Docker Compose deployment. In case you want to debug the
Docker Compose deployment
created for the test, you can export ``SKIP_DOCKER_COMPOSE_DELETION``
variable and the deployment
will not be deleted after the test. If you want to debug the deployment
using ``docker compose`` commands
you should set ``COMPOSE_PROJECT_NAME`` to ``quick-start`` because this is
what the test uses:
```
##########
TESTING.rst:
##########
@@ -450,6 +450,69 @@ This prepares airflow .whl package in the dist folder.
breeze --use-airflow-version wheel --use-packages-from-dist
--skip-mounting-local-sources
+Airflow Docker Compose Tests
+============================
+
+Running Docker Compose Tests with breeze
Review Comment:
```suggestion
Running Docker Compose Tests with Breeze
```
##########
TESTING.rst:
##########
@@ -450,6 +450,69 @@ This prepares airflow .whl package in the dist folder.
breeze --use-airflow-version wheel --use-packages-from-dist
--skip-mounting-local-sources
+Airflow Docker Compose Tests
+============================
+
+Running Docker Compose Tests with breeze
+----------------------------------------
+
+We also test in CI whether docker compose that we expose in our documentation
via
+`Running Airflow in Docker
<https://airflow.apache.org/docs/apache-airflow/stable/howto/docker-compose/index.html>`_
+works as expected. Those tests are run in CI ("Test docker-compose quick
start")
+and you can run them locally as well.
+
+The way how the tests work:
+
+1. They first build airflow production image
+2. Then they take the docker compose file of ours and use the image to start it
+3. Then they perform a simple DAG trigger tests that checks whether airflow is
up and processes simple DAG
+
+This is done in local environment, not in the Breeze CI image. It uses
``COMPOSE_PROJECT_NAME`` set to
+``quick-start`` to avoid conflicts with other docker compose deployments you
might have. and the whole
+test can be also run manually with ``pytest
docker_tests/test_docker_compose_quick_start.py`` command,
+providing that ``DOCKER_IMAGE`` environment variable is set to the image you
want to test.
Review Comment:
```suggestion
This is done in a local environment, not in the Breeze CI image. It uses
``COMPOSE_PROJECT_NAME`` set to
``quick-start`` to avoid conflicts with other docker compose deployments you
might have. The whole
test can be also run manually with ``pytest
docker_tests/test_docker_compose_quick_start.py`` command,
provided that the ``DOCKER_IMAGE`` environment variable is set to the image
you want to test.
```
##########
TESTING.rst:
##########
@@ -450,6 +450,69 @@ This prepares airflow .whl package in the dist folder.
breeze --use-airflow-version wheel --use-packages-from-dist
--skip-mounting-local-sources
+Airflow Docker Compose Tests
+============================
+
+Running Docker Compose Tests with breeze
+----------------------------------------
+
+We also test in CI whether docker compose that we expose in our documentation
via
+`Running Airflow in Docker
<https://airflow.apache.org/docs/apache-airflow/stable/howto/docker-compose/index.html>`_
+works as expected. Those tests are run in CI ("Test docker-compose quick
start")
+and you can run them locally as well.
+
+The way how the tests work:
+
+1. They first build airflow production image
+2. Then they take the docker compose file of ours and use the image to start it
+3. Then they perform a simple DAG trigger tests that checks whether airflow is
up and processes simple DAG
Review Comment:
```suggestion
1. They first build the Airflow production image
2. Then they take the Docker Compose file of ours and use the image to start
it
3. Then they perform some simple DAG trigger tests which checks whether
Airflow is up and can process a simple DAG
```
##########
docker_tests/test_docker_compose_quick_start.py:
##########
@@ -154,9 +175,16 @@ def test_trigger_dag_and_wait_for_result(tmp_path_factory,
monkeypatch):
except Exception:
print(f"Current working directory: {os.getcwd()}")
run_command(["docker", "version"])
- run_command(["docker-compose", "version"])
+ run_command([*compose_command, "version"])
run_command(["docker", "ps"])
- run_command(["docker-compose", "logs"])
+ run_command([*compose_command, "logs"])
+ for container in ALL_CONTAINERS:
+ print("Health check for {container}")
+ result = run_command(
+ ["docker", "inspect", "--format", "{{json .State}}",
container], return_output=True
+ )
+ pprint(result)
raise
finally:
- run_command(["docker-compose", "down", "--volumes"])
+ if not os.environ.get("SKIP_DOCKER_COMPOSE_DELETION"):
Review Comment:
It would be awesome to add a Breeze cli flag/option as well, so you could
specify something like: `breeze testing docker-compose-tests --skip-deletion`,
not a requirement for this PR though.
##########
docker_tests/test_docker_compose_quick_start.py:
##########
@@ -103,6 +104,17 @@ def wait_for_terminal_dag_state(dag_id, dag_run_id):
break
+ALL_CONTAINERS = [
Review Comment:
Is there any way we can fetch this list programmatically instead of
hard-coding the names here?
##########
TESTING.rst:
##########
@@ -450,6 +450,69 @@ This prepares airflow .whl package in the dist folder.
breeze --use-airflow-version wheel --use-packages-from-dist
--skip-mounting-local-sources
+Airflow Docker Compose Tests
+============================
+
+Running Docker Compose Tests with breeze
+----------------------------------------
+
+We also test in CI whether docker compose that we expose in our documentation
via
+`Running Airflow in Docker
<https://airflow.apache.org/docs/apache-airflow/stable/howto/docker-compose/index.html>`_
+works as expected. Those tests are run in CI ("Test docker-compose quick
start")
+and you can run them locally as well.
+
+The way how the tests work:
+
+1. They first build airflow production image
+2. Then they take the docker compose file of ours and use the image to start it
+3. Then they perform a simple DAG trigger tests that checks whether airflow is
up and processes simple DAG
+
+This is done in local environment, not in the Breeze CI image. It uses
``COMPOSE_PROJECT_NAME`` set to
+``quick-start`` to avoid conflicts with other docker compose deployments you
might have. and the whole
+test can be also run manually with ``pytest
docker_tests/test_docker_compose_quick_start.py`` command,
+providing that ``DOCKER_IMAGE`` environment variable is set to the image you
want to test.
+
+The complete test can be performed using breeze. The prerequisite to that
+is to have ``docker-compose`` (docker compose v1) or ``docker compose`` plugin
(docker compose v2)
+available on the path.
Review Comment:
```suggestion
The complete test can be performed using Breeze. The prerequisite to that
is to have ``docker-compose`` (Docker Compose v1) or ``docker compose``
plugin (Docker Compose v2)
available on the path.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
