I think one of the big benefits of zipped dags is that you can handle different versions of common libraries that might be used in different DAGs. You can bundle the versions of libraries together. The DAG bundles with GIT pretty much replace that property of .zip files.
But DAG versioning we have will not be yet functional for people who are - for example - using S3 and other object storages and would not want to modify all their workflows and switch to git - they will not have "real" versioning support for a while, this is a "poor man's" way of doing pretty much the same thing that separate bundles from separate branches/repos would do. But I think long term we might implement it also for object storage bundles. J. On Tue, Feb 4, 2025 at 1:18 AM Daniel Standish <daniel.stand...@astronomer.io.invalid> wrote: > Yeah i don't think we're removing zipped dags. It is very late in the game > re 3.0. It was just a thought, partly I was just curious if anyone using > it and if so how. But I'm glad I asked because, maybe you will have an > idea or two for dag versioning. Dag versioning will be in 3.0 but, I don't > think it will be the last iteration of the feature :) > > On Mon, Feb 3, 2025 at 3:54 PM Sam Wheating <samwheat...@gmail.com> wrote: > > > Hey Daniel, > > > > I haven't really been keeping up with the latest AIPs, but I just had a > > read through the DAG versioning proposal and it sounds like a lot of this > > functionality will probably be offered by DAG bundles. Thanks for > pointing > > this out! > > > > I'll see if I can get involved here and maybe help out with the > > implementation, or at least provide some feedback on the design. > Meanwhile, > > until this is available (It sounds like this will be Airflow 3.1 or > later?) > > I would suggest that we continue supporting zip-packaged DAGs. > > > > Sam > > > > On Mon, Feb 3, 2025 at 3:17 PM Daniel Standish > > <daniel.stand...@astronomer.io.invalid> wrote: > > > > > Thanks Sam > > > > > > I'm curious how the dag versioning features will interact with your > > > workflow. i'm not sure that it will immediately solve all of your use > > > case. i imagine you would probably have some good feedback on how to > > > improve / make it more useful. > > > > > > > > > > > > On Mon, Feb 3, 2025 at 3:04 PM Sam Wheating <samwheat...@gmail.com> > > wrote: > > > > > > > Hi Daniel, > > > > > > > > I can speak on behalf of Reddit Ads - we use this feature extensively > > in > > > > our pre-production environment. > > > > > > > > In short, we create per-branch zip archives which contain all custom > > > > operators and shared code as well as any DAGs modified in that > branch. > > > This > > > > allows multiple developers to concurrently test different changes to > > the > > > > same DAGs or operators in a shared remote airflow deployment - since > > all > > > > imports are localized to the zip archive there's no interference > > between > > > > different branches. > > > > > > > > I can provide more details if needed, but I just wanted to say that > > this > > > > feature has been very useful for us. If there's a better way of > > providing > > > > isolated development namespaces in a shared environment, I would love > > to > > > > hear about it. > > > > > > > > Sam > > > > > > > > On Mon, Feb 3, 2025 at 11:45 AM Daniel Standish > > > > <daniel.stand...@astronomer.io.invalid> wrote: > > > > > > > > > Recall, we can put dags inside of zip files and airflow will unzip > > and > > > > find > > > > > the dags. > > > > > > > > > > Do people actually use these? Is there still a reason to support > > this > > > > > feature? > > > > > > > > > > > > > > >
