hussein-awala commented on PR #27124: URL: https://github.com/apache/airflow/pull/27124#issuecomment-1289710933
> What is the reason you need persistence when you have git Sync? What does it give you more than just gitSync? The main reason is pulling dags from different sources (some dags from git repo and other dags from CI/CD for ex). To avoid losing the other dags after a pod restart, we need to activate the persistence. To achieve my goal, I need also to replace this block https://github.com/apache/airflow/blob/c1a9cf81e60f636ba4fb9bbf970f4312dedb9d20/chart/templates/_helpers.yaml#L413-L419 by: ``` {{ define "airflow_dags" -}} {{ (printf "%s/dags" .Values.airflowHome) }} {{- end -}} ``` in this case, I could add the other dags to a new folder `/dags/other_dags`, the `git-sync` could pull the git repo dags to `/dags/repo`, and the `dag-processor` will process all the dags in the folder `/dags`. (I will test it tomorrow before commit my code) Later, I will propose a new feature: add support to sync the dags from different git repos and adding dags from other sources to separate sub folders in dags folder, and if we don't want to add git dags to the persistent volume, we can just change the volume mount path and add an empty volume for git dags: ``` /dags: /persistent_dags -> persitent volume: /folder1 /folder2 ... /git_dags -> empty dir /repo1 /repo2 ... ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
