This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git
The following commit(s) were added to refs/heads/main by this push:
new 8593d7f Improves quick-start docker-compose warnings and
documentation (#18164)
8593d7f is described below
commit 8593d7f62dd0449e355d92d3266a2272c9000aa1
Author: Jarek Potiuk <[email protected]>
AuthorDate: Sat Sep 11 17:50:43 2021 +0200
Improves quick-start docker-compose warnings and documentation (#18164)
The recently updated docker-compose had a bit broken behaviour
for non-Linux users. It expected the .env file to be created
always, but the instructions to create them were not working
on Windows. This fixes the problem by turning the error
into warning, and directing the users to the right instructions
per operating system.
Also the recent ``DUMB_INIT_SESS_ID`` was added for worker to
allow to handle signals properly also in our quick-start
docker-compose.
---
Dockerfile | 3 +++
docs/apache-airflow/start/docker-compose.yaml | 27 +++++++++++++++++++++++----
docs/apache-airflow/start/docker.rst | 22 ++++++++++++++++++++--
3 files changed, 46 insertions(+), 6 deletions(-)
diff --git a/Dockerfile b/Dockerfile
index 1890a87..0bc435c 100644
--- a/Dockerfile
+++ b/Dockerfile
@@ -479,6 +479,9 @@ LABEL org.apache.airflow.distro="debian" \
org.opencontainers.image.title="Production Airflow Image" \
org.opencontainers.image.description="Reference, production-ready Apache
Airflow image"
+
+# See
https://airflow.apache.org/docs/docker-stack/entrypoint.html#signal-propagation
+# to learn more about the way how signals are handled by the image
ENV DUMB_INIT_SETSID="1"
ENTRYPOINT ["/usr/bin/dumb-init", "--", "/entrypoint"]
diff --git a/docs/apache-airflow/start/docker-compose.yaml
b/docs/apache-airflow/start/docker-compose.yaml
index b83673a..af09742 100644
--- a/docs/apache-airflow/start/docker-compose.yaml
+++ b/docs/apache-airflow/start/docker-compose.yaml
@@ -138,6 +138,11 @@ services:
interval: 10s
timeout: 10s
retries: 5
+ environment:
+ <<: *airflow-common-env
+ # Required to handle warm shutdown of the celery workers properly
+ # See
https://airflow.apache.org/docs/docker-stack/entrypoint.html#signal-propagation
+ DUMB_INIT_SETSID: "0"
restart: always
depends_on:
<<: *airflow-common-depends-on
@@ -161,6 +166,7 @@ services:
airflow-init:
<<: *airflow-common
entrypoint: /bin/bash
+ # yamllint disable rule:line-length
command:
- -c
- |
@@ -172,15 +178,20 @@ services:
min_airflow_version=2.2.0
min_airflow_version_comparable=$$(ver $${min_airflow_version})
if (( airflow_version_comparable < min_airflow_version_comparable ));
then
+ echo
echo -e "\033[1;31mERROR!!!: Too old Airflow version
$${airflow_version}!\e[0m"
echo "The minimum Airflow version supported:
$${min_airflow_version}. Only use this or higher!"
+ echo
exit 1
fi
if [[ -z "${AIRFLOW_UID}" ]]; then
- echo -e "\033[1;31mERROR!!!: AIRFLOW_UID not set!\e[0m"
- echo "Please follow these instructions to set AIRFLOW_UID and
AIRFLOW_GID environment variables:
-
https://airflow.apache.org/docs/apache-airflow/stable/start/docker.html#initializing-environment";
- exit 1
+ echo
+ echo -e "\033[1;33mWARNING!!!: AIRFLOW_UID not set!\e[0m"
+ echo "If you are on Linux, you SHOULD follow the instructions below
to set "
+ echo "AIRFLOW_UID and AIRFLOW_GID environment variables, otherwise
files will be owned by root."
+ echo "For other operating systems you can get rid of the warning
with manually created .env file:"
+ echo " See:
https://airflow.apache.org/docs/apache-airflow/stable/start/docker.html#setting-the-right-airflow-user";
+ echo
fi
one_meg=1048576
mem_available=$$(($$(getconf _PHYS_PAGES) * $$(getconf PAGE_SIZE) /
one_meg))
@@ -188,18 +199,24 @@ services:
disk_available=$$(df / | tail -1 | awk '{print $$4}')
warning_resources="false"
if (( mem_available < 4000 )) ; then
+ echo
echo -e "\033[1;33mWARNING!!!: Not enough memory available for
Docker.\e[0m"
echo "At least 4GB of memory required. You have $$(numfmt --to iec
$$((mem_available * one_meg)))"
+ echo
warning_resources="true"
fi
if (( cpus_available < 2 )); then
+ echo
echo -e "\033[1;33mWARNING!!!: Not enough CPUS available for
Docker.\e[0m"
echo "At least 2 CPUs recommended. You have $${cpus_available}"
+ echo
warning_resources="true"
fi
if (( disk_available < one_meg * 10 )); then
+ echo
echo -e "\033[1;33mWARNING!!!: Not enough Disk space available for
Docker.\e[0m"
echo "At least 10 GBs recommended. You have $$(numfmt --to iec
$$((disk_available * 1024 )))"
+ echo
warning_resources="true"
fi
if [[ $${warning_resources} == "true" ]]; then
@@ -207,10 +224,12 @@ services:
echo -e "\033[1;33mWARNING!!!: You have not enough resources to run
Airflow (see above)!\e[0m"
echo "Please follow the instructions to increase amount of resources
available:"
echo "
https://airflow.apache.org/docs/apache-airflow/stable/start/docker.html#before-you-begin";
+ echo
fi
mkdir -p /sources/logs /sources/dags /sources/plugins
chown -R "${AIRFLOW_UID}:${AIRFLOW_GID}" /sources/{logs,dags,plugins}
exec /entrypoint airflow version
+ # yamllint enable rule:line-length
environment:
<<: *airflow-common-env
_AIRFLOW_DB_UPGRADE: 'true'
diff --git a/docs/apache-airflow/start/docker.rst
b/docs/apache-airflow/start/docker.rst
index 0b041fb..f96f42d 100644
--- a/docs/apache-airflow/start/docker.rst
+++ b/docs/apache-airflow/start/docker.rst
@@ -110,9 +110,15 @@ apt packages and more can be found in :doc:`Building the
image <docker-stack:bui
Initializing Environment
========================
-Before starting Airflow for the first time, You need to prepare your
environment, i.e. create the necessary files, directories and initialize the
database.
+Before starting Airflow for the first time, You need to prepare your
environment, i.e. create the necessary
+files, directories and initialize the database.
-On **all operating systems**, the quick-start needs to know your host user id
and needs to have group id set to ``0``. You have to make sure to configure
them for the docker-compose:
+Setting the right Airflow user
+------------------------------
+
+On **Linux**, the quick-start needs to know your host user id and needs to
have group id set to ``0``.
+Otherwise the files created in ``dags``, ``logs`` and ``plugins`` will be
created with ``root`` iser.
+You have to make sure to configure them for the docker-compose:
.. code-block:: bash
@@ -121,6 +127,18 @@ On **all operating systems**, the quick-start needs to
know your host user id an
See :ref:`Docker Compose environment variables <docker-compose-env-variables>`
+For other operating systems, you will get warning that ``AIRFLOW_UID`` is not
set, but you can
+ignore it. You can also manually create the ``.env`` file in the same folder
your
+``docker-compose.yaml`` is placed with this content to get rid of the warning:
+
+.. code-block:: text
+
+ AIRFLOW_UID=50000
+ AIRFLOW_GID=0
+
+Initialize the database
+-----------------------
+
On **all operating systems**, you need to run database migrations and create
the first user account. To do it, run.
.. code-block:: bash
