GitHub user enchant3dmango edited a comment on the discussion: Airflow ci/cd pipeline to rebuild airflow
If you want to pause DAGs in your CI/CD pipeline, you can use this [CLI](https://airflow.apache.org/docs/apache-airflow/stable/cli-and-env-variables-ref.html#pause). It will be something like the following. You can use `docker compose up --build -d` instead of separating the build and restart command. ```yaml rebuild_project: stage: rebuild script: - ssh ${DEPLOY_USER}@${DEPLOY_SERVER} << 'EOF' cd ${DEPLOY_PATH} # Pause all DAGs docker compose exec airflow-webserver airflow dags pause <regex-for-your dags> --treat-dag-id-as-regex # Wait until all tasks are finished while docker compose exec airflow-webserver airflow tasks list_instances | grep -q running; do echo "Waiting for running tasks to finish..." sleep 10 done # Stop Airflow cleanly docker compose down # Rebuild and restart docker compose up --build -d # Unpause previously paused DAGs docker compose exec airflow-webserver airflow dags unpause <regex-for-your dags> --treat-dag-id-as-regex EOF rules: - if: '$CI_COMMIT_BRANCH == "main"' changes: - requirements.txt - start.sh - docker-compose.yaml - Dockerfile - plugins/**/* - config/**/* ``` GitHub link: https://github.com/apache/airflow/discussions/47149#discussioncomment-12340565 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
