GitHub user nilskre created a discussion: Airflow development workflow and DAG Deployment from feature branches
Advice needed how to use Airflow in an enterprise setup. ## Current setup We have a monorepo for DAGs and two Airflow instances: - `prod` -> DAGs from the main branch - `dev` -> DAGs from the dev branch Deployment uses the Git-Sync sidecar, which works fine. New DAGs are developed (and tested with unit tests) on feature branches. Some issues only become visible, when running the tasks on Airflow dev/prod. Therefore before merging the feature branch into dev/main, we want to test the DAG on dev/prod. ## Challenge Before merging a feature branch into dev or main, we want to test the new DAG on the corresponding Airflow instance. Goal: Deploy DAGs from a feature branch to dev or prod for pre-merge testing without affecting the main DAGs. ## Question What is the recommended workflow to achieve this in an enterprise Airflow setup? - Can we deploy feature-branch DAGs safely to dev/prod? - Are there best practices or patterns for testing feature branches on live Airflow instances? - Any tooling, Git workflows, or Airflow configurations that make this easier? - Any advice or references to best practices would be highly appreciated. Thanks in advance! GitHub link: https://github.com/apache/airflow/discussions/54669 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
