I don't see you have configured the AIRFLOW_HOME environment variable. Try setting it to the absolute path so that it can find your dags. Also, there is an environment variable where you can manually pass the path to the DAGs folder. AIRFLOW__CORE__DAGS_FOLDER
On Mon, Aug 9, 2021, 14:39 Anthony Joyce <[email protected]> wrote: > Hi all, > > This is my first time the email list so I thank you in advance for the > help. > > Here is my situation: > > We are running Airflow 1.10.10..a pretty old version at this point without > a container and built from pip on CentOS. Instead of updating anaconda and > dealing with dependency hell, I decided to download the official > apache/airflow docker container and try to configure it to my already > existing meta database and DAGs. It seems the container initialized > successfully picking up our Variables and Connections via our existing > Postgres-13 meta database with all containers healthy at this point. > However, I am having a problem connecting our external Airflow DAGs folder > (~/airflow/dags). I have copied the information in our ~/airflow/dags > folder in the ./dags folder but that doesn’t seem to help. > > Do you all have any advice/suggestions regarding this issue? > > Here is the redacted docker-compose.yaml file below. > > Thank you for the help! > > Best, > > Anthony > > *version*: '3' > *x-airflow-common*: > &airflow-common > *image*: ${AIRFLOW_IMAGE_NAME:-apache/airflow:2.1.2} > *environment*: > &airflow-common-env > *AIRFLOW__CORE__EXECUTOR*: CeleryExecutor > *AIRFLOW__CORE__SQL_ALCHEMY_CONN*: postgresql+psycopg2://airflow@ > {host}/airflow > *AIRFLOW__CELERY__RESULT_BACKEND*: db+postgresql://airflow@ > {host}/airflow > *AIRFLOW__CELERY__BROKER_URL*: redis://:@redis:6379/0 > *AIRFLOW__CORE__FERNET_KEY*: {key} > *AIRFLOW__CORE__DAGS_ARE_PAUSED_AT_CREATION*: 'true' > *AIRFLOW__CORE__LOAD_EXAMPLES*: 'false' > *AIRFLOW__API__AUTH_BACKEND*: 'airflow.api.auth.backend.basic_auth' > *_PIP_ADDITIONAL_REQUIREMENTS*: ${_PIP_ADDITIONAL_REQUIREMENTS:-} > *volumes*: > - ./dags:/opt/airflow/dags > - ./logs:/opt/airflow/logs > #- /home/etl/airflow/plugins/:/opt/airflow/plugins > *user*: "${AIRFLOW_UID:-50000}:${AIRFLOW_GID:-0}" > *depends_on*: > *redis*: > *condition*: service_healthy > *postgres*: > *condition*: service_healthy > > *services*: > *postgres*: > *image*: postgres:13 > *environment*: > *POSTGRES_USER*: airflow > *POSTGRES_PASSWORD*: airflow > *POSTGRES_DB*: airflow > *volumes*: > - postgres-db-volume:/localdata/pgdata > *healthcheck*: > *test*: ["CMD", "pg_isready", "-U", "airflow"] > *interval*: 5s > *retries*: 5 > *restart*: always > > *redis*: > *image*: redis:latest > *ports*: > - 6379:6379 > *healthcheck*: > *test*: ["CMD", "redis-cli", "ping"] > *interval*: 5s > *timeout*: 30s > *retries*: 50 > *restart*: always > > *airflow-webserver*: > <<: *airflow-common > *command*: webserver > *ports*: > - 8080:8080 > *healthcheck*: > *test*: ["CMD", "curl", "--fail", "http://localhost:8080/health";] > *interval*: 10s > *timeout*: 10s > *retries*: 5 > *restart*: always > > *airflow-scheduler*: > <<: *airflow-common > *command*: scheduler > *healthcheck*: > *test*: ["CMD-SHELL", 'airflow jobs check --job-type SchedulerJob > --hostname "$${HOSTNAME}"'] > *interval*: 10s > *timeout*: 10s > *retries*: 5 > *restart*: always > > *airflow-worker*: > <<: *airflow-common > *command*: celery worker > *healthcheck*: > *test*: > - "CMD-SHELL" > - 'celery --app airflow.executors.celery_executor.app inspect > ping -d "celery@$${HOSTNAME}"' > *interval*: 10s > *timeout*: 10s > *retries*: 5 > *restart*: always > > *airflow-init*: > <<: *airflow-common > *command*: version > *environment*: > <<: *airflow-common-env > *_AIRFLOW_DB_UPGRADE*: 'true' > *_AIRFLOW_WWW_USER_CREATE*: 'true' > *_AIRFLOW_WWW_USER_USERNAME*: ${_AIRFLOW_WWW_USER_USERNAME:-airflow} > *_AIRFLOW_WWW_USER_PASSWORD*: ${_AIRFLOW_WWW_USER_PASSWORD:-airflow} > > *flower*: > <<: *airflow-common > *command*: celery flower > *ports*: > - 5555:5555 > *healthcheck*: > *test*: ["CMD", "curl", "--fail", "http://localhost:5555/";] > *interval*: 10s > *timeout*: 10s > *retries*: 5 > *restart*: always > > *volumes*: > *postgres**-db-volume*: > > > >
