potiuk commented on issue #9937: URL: https://github.com/apache/airflow/issues/9937#issuecomment-663014369
I think we are going a bit different direction. Rather than making Airflow more complex, we are integrating more and more external services using existing tools. For example we have the helm chart in `charts` folder which allows you to set-up Airflow on Kubernetes cluster easily. And one of the Helm Chart configuration options is to integrate `git sync` sidecar where syncing git via regular process running in separate container but sharing DAG volume is performed. This is more "sustainable" way of doing such integrations. That's the philosophy of "a tool that only does one job well" rather than integrate all the possible options in the airflow itself. What you are describing here is more like a "s3-sync sidecar" that you might want to build instead of extending Airflow. Which (not too surprising) someone (or many someones) already did https://github.com/namshi/s3-sidecar , https://gitlab.koehn.com/docker/s3-sync, https://medium.com/nuvo-group-tech/copy-files-and-directories-betwe en-kubernetes-and-s3-d290ded9a5e0 and many more. So rather than adding such functionality to airflow maybe you would like to test/review and contribute back such a sidecar solution to the helm chart of ours ? Also we are going to have officially supported docker-compose #8548 in the future and using such side-car could be an option there, so that you would not have to use Kubernetes. I thin this is much more reasonable approach. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
