+1 to what Jarek has said I usually see zipped dags used in conjunction with blob storage, typically when dags are stored in multiple repos (either dags are split by use case, or teams are sharing a deployment). It's not perfect, but one of the benefits of zipping your dags is that if in zip_a I have a dag.py, and in zip_b, there is also a dag.py, there's no problem. I don't know if the git bundle backend would accommodate that.
Another reason people may not want to switch to the git bundle backend is github rate limits. If you're consolidating dags from 10 git repos in a bucket, you're only watching for changes in the bucket. By switching, you're now watching 10+ repos. At scale, you'll run into problems. We see that today with some of our high scale on-prem customers at Astronomer and have built out our own solutions to accommodate.. On Tue, Feb 4, 2025 at 1:33 AM Jarek Potiuk <ja...@potiuk.com> wrote: > I think one of the big benefits of zipped dags is that you can handle > different versions of common libraries that might be used in different > DAGs. You can bundle the versions of libraries together. The DAG bundles > with GIT pretty much replace that property of .zip files. > > But DAG versioning we have will not be yet functional for people who are - > for example - using S3 and other object storages and would not want to > modify all their workflows and switch to git - they will not have "real" > versioning support for a while, this is a "poor man's" way of doing pretty > much the same thing that separate bundles from separate branches/repos > would do. But I think long term we might implement it also for object > storage bundles. > > J. > > > On Tue, Feb 4, 2025 at 1:18 AM Daniel Standish > <daniel.stand...@astronomer.io.invalid> wrote: > > > Yeah i don't think we're removing zipped dags. It is very late in the > game > > re 3.0. It was just a thought, partly I was just curious if anyone using > > it and if so how. But I'm glad I asked because, maybe you will have an > > idea or two for dag versioning. Dag versioning will be in 3.0 but, I > don't > > think it will be the last iteration of the feature :) > > > > On Mon, Feb 3, 2025 at 3:54 PM Sam Wheating <samwheat...@gmail.com> > wrote: > > > > > Hey Daniel, > > > > > > I haven't really been keeping up with the latest AIPs, but I just had a > > > read through the DAG versioning proposal and it sounds like a lot of > this > > > functionality will probably be offered by DAG bundles. Thanks for > > pointing > > > this out! > > > > > > I'll see if I can get involved here and maybe help out with the > > > implementation, or at least provide some feedback on the design. > > Meanwhile, > > > until this is available (It sounds like this will be Airflow 3.1 or > > later?) > > > I would suggest that we continue supporting zip-packaged DAGs. > > > > > > Sam > > > > > > On Mon, Feb 3, 2025 at 3:17 PM Daniel Standish > > > <daniel.stand...@astronomer.io.invalid> wrote: > > > > > > > Thanks Sam > > > > > > > > I'm curious how the dag versioning features will interact with your > > > > workflow. i'm not sure that it will immediately solve all of your > use > > > > case. i imagine you would probably have some good feedback on how to > > > > improve / make it more useful. > > > > > > > > > > > > > > > > On Mon, Feb 3, 2025 at 3:04 PM Sam Wheating <samwheat...@gmail.com> > > > wrote: > > > > > > > > > Hi Daniel, > > > > > > > > > > I can speak on behalf of Reddit Ads - we use this feature > extensively > > > in > > > > > our pre-production environment. > > > > > > > > > > In short, we create per-branch zip archives which contain all > custom > > > > > operators and shared code as well as any DAGs modified in that > > branch. > > > > This > > > > > allows multiple developers to concurrently test different changes > to > > > the > > > > > same DAGs or operators in a shared remote airflow deployment - > since > > > all > > > > > imports are localized to the zip archive there's no interference > > > between > > > > > different branches. > > > > > > > > > > I can provide more details if needed, but I just wanted to say that > > > this > > > > > feature has been very useful for us. If there's a better way of > > > providing > > > > > isolated development namespaces in a shared environment, I would > love > > > to > > > > > hear about it. > > > > > > > > > > Sam > > > > > > > > > > On Mon, Feb 3, 2025 at 11:45 AM Daniel Standish > > > > > <daniel.stand...@astronomer.io.invalid> wrote: > > > > > > > > > > > Recall, we can put dags inside of zip files and airflow will > unzip > > > and > > > > > find > > > > > > the dags. > > > > > > > > > > > > Do people actually use these? Is there still a reason to support > > > this > > > > > > feature? > > > > > > > > > > > > > > > > > > > > >
