I've also witnessed the zipped DAGs feature to be used quite a bit - in scenarios similar to what Jarek & Constance described, and also to e.g. avoid downloading a multitude of files from blob storage (less effective cost & performance wise).
On Wed, Feb 5, 2025 at 6:08 PM Constance Martineau <consta...@astronomer.io.invalid> wrote: > +1 to what Jarek has said > > I usually see zipped dags used in conjunction with blob storage, typically > when dags are stored in multiple repos (either dags are split by use case, > or teams are sharing a deployment). It's not perfect, but one of the > benefits of zipping your dags is that if in zip_a I have a dag.py, and in > zip_b, there is also a dag.py, there's no problem. I don't know if the git > bundle backend would accommodate that. > > Another reason people may not want to switch to the git bundle backend is > github rate limits. If you're consolidating dags from 10 git repos in a > bucket, you're only watching for changes in the bucket. By switching, > you're now watching 10+ repos. At scale, you'll run into problems. We see > that today with some of our high scale on-prem customers at Astronomer and > have built out our own solutions to accommodate.. > > On Tue, Feb 4, 2025 at 1:33 AM Jarek Potiuk <ja...@potiuk.com> wrote: > > > I think one of the big benefits of zipped dags is that you can handle > > different versions of common libraries that might be used in different > > DAGs. You can bundle the versions of libraries together. The DAG bundles > > with GIT pretty much replace that property of .zip files. > > > > But DAG versioning we have will not be yet functional for people who are > - > > for example - using S3 and other object storages and would not want to > > modify all their workflows and switch to git - they will not have "real" > > versioning support for a while, this is a "poor man's" way of doing > pretty > > much the same thing that separate bundles from separate branches/repos > > would do. But I think long term we might implement it also for object > > storage bundles. > > > > J. > > > > > > On Tue, Feb 4, 2025 at 1:18 AM Daniel Standish > > <daniel.stand...@astronomer.io.invalid> wrote: > > > > > Yeah i don't think we're removing zipped dags. It is very late in the > > game > > > re 3.0. It was just a thought, partly I was just curious if anyone > using > > > it and if so how. But I'm glad I asked because, maybe you will have an > > > idea or two for dag versioning. Dag versioning will be in 3.0 but, I > > don't > > > think it will be the last iteration of the feature :) > > > > > > On Mon, Feb 3, 2025 at 3:54 PM Sam Wheating <samwheat...@gmail.com> > > wrote: > > > > > > > Hey Daniel, > > > > > > > > I haven't really been keeping up with the latest AIPs, but I just > had a > > > > read through the DAG versioning proposal and it sounds like a lot of > > this > > > > functionality will probably be offered by DAG bundles. Thanks for > > > pointing > > > > this out! > > > > > > > > I'll see if I can get involved here and maybe help out with the > > > > implementation, or at least provide some feedback on the design. > > > Meanwhile, > > > > until this is available (It sounds like this will be Airflow 3.1 or > > > later?) > > > > I would suggest that we continue supporting zip-packaged DAGs. > > > > > > > > Sam > > > > > > > > On Mon, Feb 3, 2025 at 3:17 PM Daniel Standish > > > > <daniel.stand...@astronomer.io.invalid> wrote: > > > > > > > > > Thanks Sam > > > > > > > > > > I'm curious how the dag versioning features will interact with your > > > > > workflow. i'm not sure that it will immediately solve all of your > > use > > > > > case. i imagine you would probably have some good feedback on how > to > > > > > improve / make it more useful. > > > > > > > > > > > > > > > > > > > > On Mon, Feb 3, 2025 at 3:04 PM Sam Wheating <samwheat...@gmail.com > > > > > > wrote: > > > > > > > > > > > Hi Daniel, > > > > > > > > > > > > I can speak on behalf of Reddit Ads - we use this feature > > extensively > > > > in > > > > > > our pre-production environment. > > > > > > > > > > > > In short, we create per-branch zip archives which contain all > > custom > > > > > > operators and shared code as well as any DAGs modified in that > > > branch. > > > > > This > > > > > > allows multiple developers to concurrently test different changes > > to > > > > the > > > > > > same DAGs or operators in a shared remote airflow deployment - > > since > > > > all > > > > > > imports are localized to the zip archive there's no interference > > > > between > > > > > > different branches. > > > > > > > > > > > > I can provide more details if needed, but I just wanted to say > that > > > > this > > > > > > feature has been very useful for us. If there's a better way of > > > > providing > > > > > > isolated development namespaces in a shared environment, I would > > love > > > > to > > > > > > hear about it. > > > > > > > > > > > > Sam > > > > > > > > > > > > On Mon, Feb 3, 2025 at 11:45 AM Daniel Standish > > > > > > <daniel.stand...@astronomer.io.invalid> wrote: > > > > > > > > > > > > > Recall, we can put dags inside of zip files and airflow will > > unzip > > > > and > > > > > > find > > > > > > > the dags. > > > > > > > > > > > > > > Do people actually use these? Is there still a reason to > support > > > > this > > > > > > > feature? > > > > > > > > > > > > > > > > > > > > > > > > > > > >
