This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch v2-1-test in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 696352baa10e093d6350d6d8d40137a3d65c31a7 Author: Kamil BreguĊa <[email protected]> AuthorDate: Mon Jul 19 14:46:18 2021 +0200 Support secret backends/airflow.cfg for celery broker in entrypoint_prod.sh (#17069) * Support secret backends in entrypoint_prod.sh * Update entrypoint_prod.sh (cherry picked from commit c1ff75efe5a2b5638148c4d0359cdb1a83b67806) --- docs/docker-stack/entrypoint.rst | 9 ++------- scripts/in_container/prod/entrypoint_prod.sh | 17 ++++++++--------- 2 files changed, 10 insertions(+), 16 deletions(-) diff --git a/docs/docker-stack/entrypoint.rst b/docs/docker-stack/entrypoint.rst index 0a0de5d..dad13f4 100644 --- a/docs/docker-stack/entrypoint.rst +++ b/docs/docker-stack/entrypoint.rst @@ -105,7 +105,7 @@ To disable check, set ``CONNECTION_CHECK_MAX_COUNT=0``. Waits for celery broker connection ---------------------------------- -In case Postgres or MySQL DB is used, and one of the ``scheduler``, ``celery``, ``worker``, or ``flower`` +In case CeleryExecutor is used, and one of the ``scheduler``, ``celery``, ``worker``, or ``flower`` commands are used the entrypoint will wait until the celery broker DB connection is available. The script detects backend type depending on the URL schema and assigns default port numbers if not specified @@ -120,12 +120,7 @@ Supported schemes: * ``postgres://`` - default port 5432 * ``mysql://`` - default port 3306 -Waiting for connection involves checking if a matching port is open. -The host information is derived from the variables :envvar:`AIRFLOW__CELERY__BROKER_URL` and -:envvar:`AIRFLOW__CELERY__BROKER_URL_CMD`. If :envvar:`AIRFLOW__CELERY__BROKER_URL_CMD` variable -is passed to the container, it is evaluated as a command to execute and result of this evaluation is used -as :envvar:`AIRFLOW__CELERY__BROKER_URL`. The :envvar:`AIRFLOW__CELERY__BROKER_URL_CMD` variable -takes precedence over the :envvar:`AIRFLOW__CELERY__BROKER_URL` variable. +Waiting for connection involves checking if a matching port is open. The host information is derived from the Airflow configuration. .. _entrypoint:commands: diff --git a/scripts/in_container/prod/entrypoint_prod.sh b/scripts/in_container/prod/entrypoint_prod.sh index 9d8e74a..d4a2e8e 100755 --- a/scripts/in_container/prod/entrypoint_prod.sh +++ b/scripts/in_container/prod/entrypoint_prod.sh @@ -195,15 +195,14 @@ function upgrade_db() { airflow db upgrade || true } -function wait_for_celery_backend() { +function wait_for_celery_broker() { # Verifies connection to Celery Broker - if [[ -n "${AIRFLOW__CELERY__BROKER_URL_CMD=}" ]]; then - wait_for_connection "$(eval "${AIRFLOW__CELERY__BROKER_URL_CMD}")" - else - AIRFLOW__CELERY__BROKER_URL=${AIRFLOW__CELERY__BROKER_URL:=} - if [[ -n ${AIRFLOW__CELERY__BROKER_URL=} ]]; then - wait_for_connection "${AIRFLOW__CELERY__BROKER_URL}" - fi + local executor + executor="$(airflow config get-value core executor)" + if [[ "${executor}" == "CeleryExecutor" ]]; then + local connection_url + connection_url="$(airflow config get-value celery broker_url)" + wait_for_connection "${connection_url}" fi } @@ -322,7 +321,7 @@ fi # Note: the broker backend configuration concerns only a subset of Airflow components if [[ ${AIRFLOW_COMMAND} =~ ^(scheduler|celery|worker|flower)$ ]] \ && [[ "${CONNECTION_CHECK_MAX_COUNT}" -gt "0" ]]; then - wait_for_celery_backend + wait_for_celery_broker fi exec "airflow" "${@}"
