potiuk commented on PR #34331: URL: https://github.com/apache/airflow/pull/34331#issuecomment-1826437589
I would strongly discourage using both git-sync and persistence. I am very close to propose (I did have a PR in the past about it) such combo should be forbidden because it leads to very brittle behaviour and you might have alot of problems with it. So I would rather not make it easier for people using it, to be honest. I still failed to see where it might bring any use and there are multiple problems such setup might cause - depending on - for example - whether your persistence solution has provisioned IOPS and how "distributed" your setup is and what kind of the PVC you configure. You can read more about it here: https://airflow.apache.org/docs/helm-chart/stable/manage-dags-files.html#notes-for-combining-git-sync-and-persistence Also if you would like to read more and see if your solution might work better (depending on the way which PVC you are using and what kind of K8S you run and how distributed your cluster is) you can read about it in my blog post where I explained why combining Persistence and Git Sync is a bad idea: https://medium.com/apache-airflow/shared-volumes-in-airflow-the-good-the-bad-and-the-ugly-22e9f681afca Generally you are shooting yourself in the foot by enabling both at the same time. I am not sure we want to fix anything about git-sync + persistence combo to be honest. I don't thinlk we should make it easier for our users to shoot htemselves in their foot. I'd rather see a PR disabling this combo. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
