dnskr opened a new pull request #16291: URL: https://github.com/apache/airflow/pull/16291
This PR fixes the following issue: **Airflow Helm chart cannot be installed by Argo CD because of "migrate-database-job" hook** --- [Argo CD](https://argo-cd.readthedocs.io/en/stable/) is a popular continuous delivery tool for Kubernetes. It is used by many teams to deploy and manage applications in Kubernetes clusters. ### Problem description There are 3 objects dependent on "migrate-database-job" hook: - [scheduler](https://github.com/apache/airflow/blob/main/chart/templates/scheduler/scheduler-deployment.yaml#L106) - [webserver](https://github.com/apache/airflow/blob/main/chart/templates/webserver/webserver-deployment.yaml#L100) - [worker](https://github.com/apache/airflow/blob/main/chart/templates/webserver/webserver-deployment.yaml#L100) In the current implementation "migrate-database-job" hook is triggered before the main containers of scheduler, webserver and worker are started. So they have "wait-for-airflow-migrations" init-containers which wait for "migrate-database-job" to be done. Hook "wait-for-airflow-migrations" is implemented by [Helm hooks](helm.sh/docs/topics/charts_hooks). However, Argo CD has its own hooks implementation and maps them to Helm hooks with some differences/restrictions. With the current chart implementation Argo CD never triggers "migrate-database-job" hook because `"helm.sh/hook": post-install` is mapped to `"argocd.argoproj.io/hook": PostSync` which means that the hook has to be executed `after all Sync hooks completed and were successful, a successful application, and all resources in a Healthy state`([link to the doc](https://argo-cd.readthedocs.io/en/stable/user-guide/resource_hooks/#usage)). This condition is never **true** because scheduler, webserver and worker stay in init state waiting for "migrate-database-job" to finish. This PR adds Argo CD hook annotations to override Helm hook annotations. ### How to reproduce Install Argo CD ``` helm repo add argo https://argoproj.github.io/argo-helm helm repo update helm install argocd argo/argo-cd --version 3.6.6 ``` Create Airflow application manifest ``` apiVersion: argoproj.io/v1alpha1 kind: Application metadata: name: airflow finalizers: - resources-finalizer.argocd.argoproj.io spec: project: default source: repoURL: https://github.com/apache/airflow.git targetRevision: main path: chart helm: version: v3 valueFiles: - values.yaml values: | executor: KubernetesExecutor postgresql: enabled: true persistence: enabled: false flower: enabled: false redis: enabled: false statsd: enabled: false pgbouncer: enabled: false destination: server: https://kubernetes.default.svc namespace: default syncPolicy: automated: prune: true selfHeal: true ``` Create application ``` kubectl create -f airflow.yaml ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
