This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git
The following commit(s) were added to refs/heads/main by this push:
new ecc5b74 Stronger language about Docker Compose customizability
(#22304)
ecc5b74 is described below
commit ecc5b74528ed7e4ecf05c526feb2c0c85f463429
Author: Jarek Potiuk <[email protected]>
AuthorDate: Fri Mar 18 00:34:33 2022 +0100
Stronger language about Docker Compose customizability (#22304)
* Stronger language about Docker Compose customizability
Despite our warnings, our users continue treating the Docker
Compose that we exposed as something that should be easy to
extend and customize for their own needs, yet they continue
to struggle with some basic behaviour of containers, Docker Compose
and how they interact. This results in vast space of potential
problems as Docker Compose gives the user a false premise of
something that "just works" where it requires quite a deep
understanding on how it works.
When you get things wrong with Docker Compose, you often end up
with extremely confusing messages, that might suggest that the
problem is with Airflow, but really the problem is with how users
interact with their custom Docker images, registries, pulling,
networking, mounting volumes and plenty other things.
While this is the same with Kubernetes and Helm Chart, Helm Chart makes
it infinitely easier to customize in declarative way (this is what
our values.yaml does) and anything that has not been foreseen by Helm
Chart developers is "hard" by definition.
Docker Compose makes no such distinction. You really can't make Docker
Compose customizable by configuration, and any customization in it
requires modifying the compose file and for people who do not know
what they are doing will eventually lead to errors that they are not
able to diagnose and leads to creation of "Airlfow isssues", where they
should be brought to "Docker Compose" issues.
Example of that is here: https://github.com/apache/airflow/discussions/22301
where there are at least two issues that are not reproducible without
knowing in detail what the user has done, how the image was build
and distributed, and how the docker-compose installation interacted
with them. This leads to a terrible distraction for supporting
users of Airflow as the issues are really Docker Compose issues and
Airflow maintainers should not be involved in solving those.
This PR adds a bit stronger language and statement about the scope
and customizability of the Quick Start Docker Compose of ours. Not
only mentioning "Lack of Production Readiness" but also the
responsibility of the user to understand and diagnose docker compose
errors on their own and setting expectations that issues with Docker
Compose running should be directed elsewhere.
* Update docs/apache-airflow/start/docker.rst
* Update docs/apache-airflow/start/docker.rst
Co-authored-by: Daniel Standish
<[email protected]>
* Update docs/apache-airflow/start/docker.rst
Co-authored-by: Daniel Standish
<[email protected]>
Co-authored-by: Daniel Standish
<[email protected]>
---
docs/apache-airflow/start/docker.rst | 22 +++++++++++++++++++++-
1 file changed, 21 insertions(+), 1 deletion(-)
diff --git a/docs/apache-airflow/start/docker.rst
b/docs/apache-airflow/start/docker.rst
index fbab8ff..305415d 100644
--- a/docs/apache-airflow/start/docker.rst
+++ b/docs/apache-airflow/start/docker.rst
@@ -20,7 +20,27 @@
Running Airflow in Docker
#########################
-This quick-start guide will allow you to quickly start Airflow with
:doc:`CeleryExecutor </executor/celery>` in Docker. This is the fastest way to
start Airflow.
+This quick-start guide will allow you to quickly get Airflow up and running
with :doc:`CeleryExecutor </executor/celery>` in Docker. For running airflow
in production, we recommend running on kubernetes with the official helm chart.
+
+Customizing the quick-start Docker Compose
+==========================================
+
+DO NOT attempt to customize images and the Docker Compose if you do not know
exactly what you are doing,
+do not know Docker Compose, or are not prepared to debug and resolve problems
on your own. If you do not
+know Docker Compose and expect Airflow to **just work** beyond following
precisely the quick-start,
+then please use other ways of running Airflow - for example
:doc:`/start/local` for testing and trying
+and :doc:`Official Airflow Community Helm Chart<helm-chart:index>` for
production purposes.
+
+Even if many users think of Docker Compose as "ready to use", it is really a
developer tool, that requires
+the user to know very well how docker images, containers, docker compose
networking, volumes, naming, image
+building works. It is extremely easy to make mistakes that lead to difficult
to diagnose problems and if
+you are not ready to spend your own time on learning and diagnosing and
resolving those problems on your own
+do not follow this path. You have been warned.
+
+If you customize, or modify images, the compose file and see problem do not
expect you will get a lot of
+help with solving those problems in the Airflow support channels. Most of the
problems you will experience
+are Docker Compose related problems and if you need help in solving them,
there are dedicated channels in
+`Docker Compose <https://github.com/docker/compose>`_ that you can use.
Production readiness
====================
