hussein-awala opened a new pull request, #27124: URL: https://github.com/apache/airflow/pull/27124
This PR can solve [this issue](https://github.com/apache/airflow/issues/27080) Before this [PR](https://github.com/apache/airflow/pull/23711) which added the standalone Dag Processors to the helm chart, the git sync logic was: - if `dags.gitSync.enabled` then: 1. add `git_sync_container-init` and `git_sync_container` to scheduler pods containers 2. load the volume `dags` from: - a PVC if `dags.persistence.enabled`: claim existing PVC if `dags.persistence.existingClaim` is defined or create a new PVC if not - `emptyDir` volume if not 4. mount the volume `dags` to scheduler container as readOnly 5. mount the volume `dags` to `git_sync_container` - elif `dags.persistence.enabled` then: - load the volume `dags` from an existing PVC if `dags.persistence.existingClaim` is defined or create a new PVC if not - add this volume to schedulers container So when `dags.gitSync.enabled` we always had a `git_sync_container` running in the schedulers pods, and the `dags.persistence.enabled` is used to specify the volume type After the PR, we create the `git_sync_container` in the scheduler pods only if: - executor is one of the Local Executors and `gitSync` is enabled - or `dagProcessor` is not enabled and `gitSync` is enabled which I find logical ([code](https://github.com/gmsantos/airflow/blob/main/chart/templates/scheduler/scheduler-deployment.yaml#L221-L223)) But we create the `git_sync_container` in the `dag-processor` only if `gitSync` is enabled and `dags.persistence` is not enabled ([code](https://github.com/gmsantos/airflow/blob/main/chart/templates/dag-processor/dag-processor-deployment.yaml#L189-L191)) I think we can always create `git_sync_container` in the `dag-processor` pod when it is enabled and `gitSync` is enabled, and we use `dags.persistence` to switch between PVC and emptyDir -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
