potiuk commented on PR #43408: URL: https://github.com/apache/airflow/pull/43408#issuecomment-2441215247
Just to put it in context: in Airflow 2, all the DAGs in DAG folder are in the same "namespace" - they can see each other and import each other. You can potentially bundle things together in a DAG .zip file (and this is the way how you can currently also bundle DAG and it's dependencies so that they are isolated between different DAGs) - but generally speaking when you have a DAG, imports will use the DAG_FOLDER as the part of PYTHONPATH where all imports are made from for your custom code. I don't see why you would not be able to do what you want to do with submodules to be honest. But maybe I am wrong. Also in Airflow 3 this is going to change a lot and in a completely different way that your proposal is heading. We have AIP-66 being implemetned: https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=294816356 where you will be able to specify different sources for different bundles, and this all will be handled pretty differently - and the community is going to focus their efforts on supporting this - way more forward looking approach, which has much greater flexibility and will be supported. And if you want to spend your efforts on making it happen - I would suggest contributing to those efforts, rather than adding support for multiple git-sync containers. Even if you have a good case yourself to have support for multiple git-syncs ratther than submodules I don't think it is something we would like to support here, you could probably fork Airfow's chart and publish your version if you want to expose it to others and support it for others using it, but I think that would be quite a bit of burden for us to support it here in the community, while we already not only defined what is the way forward but are also in the middle of implementing it for airflow 3. @jedcunningham -> since you are mostly engaged in AIP-66, is my thinking going in the right direction? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
