Thanks for the reply Ephraim! I really appreciate you taking the time to respond on the PR and point out exactly what you disagree with (and I think there's a lot of merit there after a quick look) and are willing to iterate on it :) I'll get to your PR comments and see how far we can get that change set, likely after my vacation!
I'm also glad that we align on the mapping table as a close second. Between the two I'm confident we'll land on a solution that is good for Airflow and easy/safe to use for users! Cheers, Niko P.S. I'm reserving further comment on externally storing the manifest in the user's bucket/block storage. You make good points, and I have some further thoughts/rebuttals, but if we agree that it's a last-resort third option, then let's debate its pros and cons only if we can't make headway on the first two options. On Thu, Jul 30, 2026 at 1:07 PM Ephraim Anierobi <[email protected]> wrote: > Hey Niko, thanks for the detailed reply, and enjoy the vacation! > > I have posted the specifics in the PR as you requested. The short version > is that I disagree with points 3 and 4, I think it should be worked on. The > changed branch only runs when the Dag itself is unchanged, and the manifest > covers the whole bucket, so unrelated file changes reach every Dag in the > bundle. And the task instance check has no state filter, so it means "has > ever run" and not "has running tasks". Details with code references are on > the PR. > > I am glad we land close on the mapping table. It is my second choice too, > ahead of keeping the manifest on DagVersion, so that feels like real common > ground if the external path does not get support. > > On write access and brittleness, I think both designs put some state in > user hands, just different state. VersionId pins depend on versioning being > enabled and on the bucket's retention rules. A wrong lifecycle rule would > break old pins there too, and the manifest in the DB cannot see that until > a fetch fails. In the external design, the state lives in a bucket the team > owns and can lock down. That is the same pattern as the Terraform state > bucket Jens mentioned earlier. The only write access needed is to that > releases area, never to the Dag source. And every fetched file is checked > against its hash, so if something is deleted or changed the run fails with > a clear error instead of running the wrong code. Where the state lives is a > real choice, but I don't think either option takes the user out of the > picture. > > Regards > Ephraim > > On Thu, 30 Jul 2026 at 17:45, Niko Oliveira <[email protected]> wrote: > > > Hey Ephraim thanks for weighing in! > > > > I very much appreciate it :) > > > > Indeed, I have that PR out with a suggested change for object store > Bundles > > to allow useful versioning. I understand that it might sound a bit scary > > initially, but if you read through the (I hope carefully written) > > description and follow-up comments, I think the actual impact is much > > smaller. I didn't see many specific counterpoints against the proposal > > itself, only a quick pivot to something else. Can you share some of your > > analysis of the proposal in the PR itself? The proposed change only opens > > the aperture of when a new DagVersion is created by a very small margin: > > > > 1) The Bundle being used must be one that leverages version_data (the > > manifest in the S3 case). Meaning current bundles like Git and Local are > > *unaffected* by these changes entirely > > 2) The bundle above must have versioning enabled. So regular unversioned > > use of the S3 bundle for example is *unaffected* > > 3) The Dags in the bundle must have meaningful changes that > > actually require a new version > > 4) The Dag must have running tasks associated with it. > > 5) Lastly, the bucket must actually be deployed to (of course) which is > > less frequent than file changes on disk or in a Git repository (where > many > > other unrelated changes are often merged). So an intentional deployment > to > > a bucket (perhaps, see points above) creating a new DagVersion is a > > reasonable expectation, I think. > > > > The documentation will state that these bundles may create DagVersions on > > bucket pushes, and I think given that existing tooling for cleaning up > > these versions exists, it can be easily managed. > > > > In the PR I only see "I worry" or "the problem is real and well > explained," > > rather than a deep consideration of the proposal itself. So given the > > above, which points do you strictly disagree with? Perhaps a discussion > on > > the dev call would allow a more nuanced conversation? > > > > I would like this thread to hopefully revert to discussing the PR itself, > > as requested above, but just quick hits on your proposals: > > > > A separate table would be my second best choice here. Although it adds > more > > schema to manage, more tables to ensure stay clean and don't overrun. It > is > > a better user experience. > > > > On your PoC for storing the manifest in the datastore itself, that must > > have taken some time, and I appreciate the effort! But it was always > known > > to be technically possible, that was never the issue. The issues > are/were: > > 1) Needing write access to the datastore. Enterprise users we've talked > > with find this an absolute no, this system should not be allowed write > > access to the datastore itself. 2) The more pieces you put in the users' > > hands, the more brittle the design becomes. They can accidentally delete > or > > modify these manifests, have poorly configured retention rules, and need > to > > develop their own tools for reaping old versions, etc. There is a higher > > likelihood that you will receive issues and "support requests" for a > piece > > of the design that is out of our hands. > > > > > > Again, I really appreciate you taking the time to weigh in! > > > > FYI: I'm heading off on a vacation starting this evening. So future > replies > > will be delayed :) > > > > Cheers, > > Niko > > > > > > On Wed, Jul 29, 2026 at 10:07 AM Ephraim Anierobi < > > [email protected]> wrote: > > > > > Hi all, > > > > > > I want to revive this thread. Niko has a PR open now that touches this > > area > > > (https://github.com/apache/airflow/pull/70207) and the direction > > question > > > we discussed here is coming up again in its review, so it feels better > to > > > continue that discussion on the list than on the PR. > > > > > > First a small clarification for the record. Reading back through the > > > thread, what did not get support was storing the manifests through DB > > > Manager, so provider owned tables and provider owned migrations. A > table > > in > > > core Airflow models was not rejected. Jens even said that if the DB is > > the > > > choice then a common solution in core would be easier than DB handling > > in a > > > provider. On the PR, Jed suggested a mapping table keyed on bundle name > > and > > > version, and I don't think this thread rules that out. > > > > > > That said, my own preference is still the direction Jarek and Jens > argued > > > for here, which is to keep this state out of the database and put it in > > the > > > store itself. The database then only keeps the version string that each > > run > > > already records, and the store can always turn that string back into > the > > > code. Nothing in the database ever becomes the only copy of something > > that > > > is needed to rebuild a bundle. > > > > > > I wanted to see if that works in practice rather than argue it in > theory, > > > so I built an experiment. The repo is > > > https://github.com/ephraimbuddy/airflow-manifest-bundle and there is a > > > design doc at > > > > > > > > > https://github.com/ephraimbuddy/airflow-manifest-bundle/blob/main/docs/design.md > > > . > > > The bundle version is a sha256 content hash and the hash alone is > enough > > to > > > find the code again, the same way a git sha resolves against the object > > > store. Snapshots and their manifests live under a published root, which > > can > > > be an s3 prefix or a shared filesystem. The manifest object is written > > last > > > so a snapshot only becomes visible once it is complete. The release > > pointer > > > uses S3 conditional writes instead of a lock file. The dag source > prefix > > > stays read only in every mode and workers need nothing more than > > GetObject > > > on the releases prefix, and nothing at all in the filesystem mode. It > > also > > > waits for the source to be stable before it publishes, so a half synced > > > bucket never becomes a release. It runs unchanged on Airflow 3.0.0 > > through > > > 3.3.x with no core changes, which I think shows the external path does > > not > > > need anything from core. > > > > > > On why I used a content hash instead of S3 versionIds in the > experiment. > > I > > > wanted the version string alone to be enough to find and check the > code. > > A > > > content hash can always be recomputed from the files, so the manifest > is > > > never the only copy of anything and there is no extra state that needs > a > > > safe home. That is what lets the whole storage question fall away. > Every > > > fetched file is checked against its hash, so the chain verifies itself > > end > > > to end. A hash also works the same on any storage, so the published > root > > > can be a bucket today, a different bucket later, or a shared > filesystem, > > > and nothing needs versioning enabled on the source. VersionIds have a > > real > > > advantage in that nothing gets copied. If that matters there is a > > possible > > > hybrid where the file entries are versionId references but the manifest > > > itself is stored content addressed in the bucket. That keeps the use of > > > native S3 versioning and still keeps the state out of the DB. > > > > > > To be clear I am not pushing the package itself. I am happy to help > move > > > whatever pieces are useful into the amazon provider, and Niko, happy to > > > align with whatever you have in flight. The goal is one good versioned > S3 > > > bundle. > > > > > > Thoughts? > > > > > > - Ephraim > > > > > > On Fri, 18 Jul 2025 at 06:44, Zhe You Liu <[email protected]> wrote: > > > > > > > > This is a good solution. It goes along the idea of a "generic" > > solution > > > > > that does not need an "amazon specific" table and DB manager. If > the > > > > > manifest serialized field can be used for all other "bundles" (even > > if > > > > > manifest format itself is specific to S3 bundle), I am very happy > > with > > > > that > > > > > solution. > > > > > > > > Glad to hear that. Let’s wait for input from others as well, there > may > > be > > > > concerns I haven’t fully considered. > > > > > > > > > One thing to consider (but this is entirely up to the S3 bundle > > > > > implementation) is handling versioning of such manifest during > > > > > serialization/deserialization to allow downgrading and upgrading > the > > > > > provider seamlessly. > > > > > > > > Nice point and I agree. Regardless of which approach we take (storing > > in > > > > the DB or in object storage), we’ll need to handle serialization > > properly > > > > and ensure backward compatibility. > > > > > > > > Best, > > > > Jason Liu > > > > > > > > > > > > On 2025/07/18 04:59:45 Jarek Potiuk wrote: > > > > > > In my opinion, we can simply add an optional `manifest` field (or > > > > another > > > > > suitable name). I don’t think we need to introduce a new table via > > > > > DbManager; an additional field for storing metadata about the > > external > > > > > state (such as prefix and object versions for all dags in the > bundle, > > > in > > > > > the case of S3DagBundle) should suffice. We could introduce a new > > > parent > > > > > subclass, such as `RemoteDagBundle` or `ObjectStoreDagBundle`, in > the > > > > > common provider to define the structure for serializing and > > > deserializing > > > > > the `manifest` field. > > > > > > > > > > This is a good solution. It goes along the idea of a "generic" > > solution > > > > > that does not need an "amazon specific" table and DB manager. If > the > > > > > manifest serialized field can be used for all other "bundles" (even > > if > > > > > manifest format itself is specific to S3 bundle), I am very happy > > with > > > > that > > > > > solution. One thing to consider (but this is entirely up to the S3 > > > bundle > > > > > implementation) is handling versioning of such manifest during > > > > > serialization/deserialization to allow downgrading and upgrading > the > > > > > provider seamlessly. > > > > > > > > > > > > > > > > > > > > On Fri, Jul 18, 2025 at 5:56 AM Zhe You Liu <[email protected]> > > > wrote: > > > > > > > > > > > Sorry for the late response. > > > > > > > > > > > > Both approaches work for me; I just wanted to share my opinion as > > we > > > > > > settle on a final decision. > > > > > > > > > > > > From my perspective, the DagBundle acts as a client that pulls > > > external > > > > > > state and stores only the version identifier in the Airflow > > metadata > > > > DB. > > > > > > > > > > > > For example, with GitDagBundle, the Git repository serves as the > > > > external > > > > > > storage. The GitDagBundle pulls DAG files locally and stores the > > > commit > > > > > > hash as the `version` field in `DagBundleModel.version`. > > > > > > > > > > > > 1. If we choose to store the manifest in the Airflow metadata DB: > > > > > > > > > > > > In my opinion, we can simply add an optional `manifest` field (or > > > > another > > > > > > suitable name). I don’t think we need to introduce a new table > via > > > > > > DbManager; an additional field for storing metadata about the > > > external > > > > > > state (such as prefix and object versions for all dags in the > > bundle, > > > > in > > > > > > the case of S3DagBundle) should suffice. We could introduce a new > > > > parent > > > > > > subclass, such as `RemoteDagBundle` or `ObjectStoreDagBundle`, in > > the > > > > > > common provider to define the structure for serializing and > > > > deserializing > > > > > > the `manifest` field. > > > > > > > > > > > > 2. If we decide to store the manifest outside the Airflow > metadata > > > DB: > > > > > > > > > > > > We will need to clarify: > > > > > > > > > > > > a) The required parameters for all DagBundles that pull DAGs from > > > > object > > > > > > storage. Based on the discussion above, we would need the > > `conn_id`, > > > > > > `bucket`, and `prefix` for the manifest file. > > > > > > > > > > > > b) The interface for calculating the bundle version based on the > > > > external > > > > > > state or DAG content hash. > > > > > > > > > > > > Here is a concrete example of how the manifest could be stored: > > > > > > > > https://github.com/apache/airflow/pull/46621#issuecomment-3078208467 > > > > > > > > > > > > Thank you all for the insightful discussion! > > > > > > > > > > > > Best, > > > > > > Jason > > > > > > > > > > > > On 2025/07/10 21:56:31 "Oliveira, Niko" wrote: > > > > > > > Thanks for the reply Jarek :) > > > > > > > > > > > > > > Indeed we have different philosophies about this so we will > > > certainly > > > > > > keep going in circles about where to draw the line on making > things > > > > easy > > > > > > and enjoyable to use, whether to intentionally add friction or > not, > > > > etc, > > > > > > etc. > > > > > > > > > > > > > > I think if we have optional paths to take and it's not > immensely > > > > harder > > > > > > we should err on the side of making OSS Airflow as good as it can > > be, > > > > > > despite whatever managed services we have in the community. I'm > not > > > > sure > > > > > > where it has come from recently but this new push to make Airflow > > > > > > intentionally hard to use so that managed services stay in > business > > > is > > > > a > > > > > > bit unsettling. We're certainly not asking for that, and those > > around > > > > that > > > > > > I've chatted to (since I'm now seeing this mentioned frequently) > > are > > > > also > > > > > > not asking for this. I'm curious where this new pressure is > coming > > > > from and > > > > > > why you feel it recently. > > > > > > > > > > > > > > But regardless of the curiosity above, I'll return to the > drawing > > > > board, > > > > > > and see what else can be done for this particular problem. If > there > > > are > > > > > > other Bundle types who need to solve the same problem perhaps we > > can > > > > find a > > > > > > more acceptable implementation in Airflow core to support this. > And > > > if > > > > not, > > > > > > I'll proceed with externalizing the storage of the S3 Bundle > > version > > > > > > metadata outside of Airflow. > > > > > > > > > > > > > > Cheers, > > > > > > > Niko > > > > > > > > > > > > > > ________________________________ > > > > > > > From: Jarek Potiuk <[email protected]> > > > > > > > Sent: Wednesday, July 9, 2025 11:59:06 PM > > > > > > > To: [email protected] > > > > > > > Subject: RE: [EXT] S3 Dag Bundle Versions and DB Manager > > > > > > > > > > > > > > CAUTION: This email originated from outside of the > organization. > > Do > > > > not > > > > > > click links or open attachments unless you can confirm the sender > > and > > > > know > > > > > > the content is safe. > > > > > > > > > > > > > > > > > > > > > > > > > > > > AVERTISSEMENT: Ce courrier électronique provient d’un > expéditeur > > > > > > externe. Ne cliquez sur aucun lien et n’ouvrez aucune pièce > jointe > > si > > > > vous > > > > > > ne pouvez pas confirmer l’identité de l’expéditeur et si vous > > n’êtes > > > > pas > > > > > > certain que le contenu ne présente aucun risque. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > To me, I'm always working from a user perspective. My goal is > > to > > > > make > > > > > > > their lives easier, their deployments easier, the product the > > most > > > > > > > enjoyable for them to use. To me, the best user experience is > > that > > > > they > > > > > > > should enable bundle versioning and it should just work with as > > > > little or > > > > > > > no extra steps and with as little infra as possible, and with > the > > > > fewest > > > > > > > possible pit falls for them to fall into. From a user > > perspective, > > > > > > they've > > > > > > > already provisioned a database for airflow metadata, why is > this > > > > portion > > > > > > of > > > > > > > metadata leaking out to other forms of external storage? Now > this > > > is > > > > > > > another resource they now need to be aware of and manage the > > > > lifecycle of > > > > > > > (or allow us write access into their accounts to manage for > > them). > > > > > > > > > > > > > > > > > > > > > *TL;DR; I think our goal in open-source is to have frictionless > > and > > > > "out > > > > > > of > > > > > > > the box" experience only for basic cases, but not for more > > complex > > > > > > > deployments.* > > > > > > > > > > > > > > It's a long read if you want to read it .. so beware :). > > > > > > > > > > > > > > I think that is an important "optimization goal" for sure to > > > provide > > > > > > > frictionless and enjoyable experience - but I think it's one of > > > many > > > > > > goals > > > > > > > that are sometimes contradicting with long term open-source > > project > > > > > > > sustainability and it's very import to clarify which "user" we > > are > > > > > > talking > > > > > > > about. > > > > > > > > > > > > > > To be honest, I am not sure that our goal should be "airflow > > should > > > > work > > > > > > > out of the box in case of integration with external services in > > > > > > production' > > > > > > > if it complicates our code and makes it service-dependent - > and > > as > > > > Jens > > > > > > > noticed, if we can come up with a "generic" thing that can be > > > > reusable > > > > > > > across multiple services, we can invest more in making it works > > > "out > > > > of > > > > > > the > > > > > > > box", but if you anyhow need to integrate and make work with > > > external > > > > > > > service, it adds very little "deployment complexity" to use > > another > > > > piece > > > > > > > of the service - and this is basically the job of deployment > > > manager > > > > > > > anyway. > > > > > > > > > > > > > > The "just work" goal as I see it should only cover those > > individual > > > > users > > > > > > > who want to try and use airflow in it's basic form and > > "standalone" > > > > > > > configuration - not for "deployment managers". > > > > > > > > > > > > > > I think yes - our goal should be to make things extremely easy > > for > > > > users > > > > > > > who want to use airflow in its basic form where things should > > > **just > > > > > > > work**. Like "docker run -it apache/airflow standalone" - this > is > > > > what > > > > > > > currently **just works**, 0 configuration, 0 work for external > > > > > > > integrations, and we even had a discussion that we could make > it > > > "low > > > > > > > production ready" (which I think we could - just implement > > > automated > > > > > > > backup/recovery of sqlite db and maybe document mounting a > folder > > > > with > > > > > > DAGs > > > > > > > and db, better handling of logs rather than putting them as > mixed > > > > output > > > > > > on > > > > > > > stdout and we are practically done). But when you add "S3" as > the > > > dag > > > > > > > storage you already need to make a lot of decisions - mostly > > about > > > > > > service > > > > > > > accounts, security, access, versioning, backup of the s3 > objects, > > > > etc. > > > > > > etc. > > > > > > > And that's not a "standalone user' case - that is a "deployment > > > > manager" > > > > > > > work (where "deployment manager" is a role - not necessarily > > title > > > > of the > > > > > > > job you have. > > > > > > > > > > > > > > I think - and that is a bit of philosophical - but I've been > > > talking > > > > > > about > > > > > > > it to Maciek Obuchowski yesterday - that there is a pretty > clear > > > > boundary > > > > > > > of what open-source solutions delivers and it should match > > > > expectations > > > > > > of > > > > > > > people using it. Maintainers and community developing > open-source > > > > should > > > > > > > mostly deliver a working, generic solutions that are extendable > > > with > > > > > > > various deployment options and we should make it possible for > > those > > > > > > > deployments to happen - and provide building blocks for them. > But > > > > it's > > > > > > > "deployment manager" work to make sure to put things together > and > > > > make it > > > > > > > works. And we should not do it "for them". It's their job to > > figure > > > > out > > > > > > how > > > > > > > to configure and set-up things, make backups, set security > > > boundaries > > > > > > etc. > > > > > > > - we should make it possible, document the options, document > > > security > > > > > > model > > > > > > > and make it "easy" to configure things - but there should not > be > > an > > > > > > > expectation from the deploiyment manager that it "just works". > > > > > > > > > > > > > > And I think your approach is perfectly fine - but only for > > "managed > > > > > > > services" - there, indeed manage service user's expectations > can > > be > > > > that > > > > > > > things "just work" and they are willing to pay for it with real > > > > money, > > > > > > > rather than their time and effort to make it so. And there I > > think, > > > > those > > > > > > > who deliver such a service should have the "just work" as > primary > > > > goal - > > > > > > > also because users will have such expectations - because they > > > > actually > > > > > > pay > > > > > > > for it to "just work". Not so much for open-source product - > > where > > > > "just > > > > > > > work" often involves complexity, additional maintenance > overhead > > > and > > > > > > making > > > > > > > opinionated decisions on "how it just works". For those > "managed > > > > service" > > > > > > > teams - "just work" is very much a primary goal. But for "open > > > > source > > > > > > > community" - having such a goal is actually not good - it's > > > > dangerous > > > > > > > because it might result in wrong expectations from the users. > If > > we > > > > start > > > > > > > making airflow "just works" in all kinds of deployment with > zero > > > work > > > > > > from > > > > > > > the users who want to deploy it in production and at scale, > they > > > will > > > > > > > expect it to happen for everything - why don't we have > automated > > > log > > > > > > > trimming, why don't we have automated backup of the Database, > why > > > > don't > > > > > > we > > > > > > > auto vacuum the db, why don't we provide one-click deployment > > > option > > > > on > > > > > > > AWS. GCS. Azure, why don't we provide DDOS protection in our > > > > webserver, > > > > > > why > > > > > > > don't we ..... you name it. > > > > > > > > > > > > > > That's a bit of philosophy - those are the same assumptions and > > > goals > > > > > > that > > > > > > > I had in mind when designing multi-team - and there it's also > why > > > we > > > > had > > > > > > > different views - I just feel that some level of friction is a > > > > "property" > > > > > > > of open-source product. > > > > > > > > > > > > > > Also a bit of "business" side - this is also "good" for those > who > > > > provide > > > > > > > managed services and airflow to keep sustainable open-source > > > business > > > > > > model > > > > > > > working - because what people are paying them is precisely to > > > > "remove the > > > > > > > friction". If take the "frictionless user experience" goal > case > > to > > > > > > extreme > > > > > > > - Airflow would essentially be killed IMHO. Imagine if Airflow > > > would > > > > be > > > > > > > frictioness for all kinds of deployments and had "everything" > > > > working out > > > > > > > of the box. There would be no business for any of the managed > > > > services > > > > > > > (because users would not need to pay for it). Then we would > only > > > have > > > > > > users > > > > > > > who expect thigns to "just work" and most of them would not > even > > > > think > > > > > > > about contributing back. And there would be no managed services > > > > people > > > > > > > (like you) whose job is paid by the services - or people like > me > > > who > > > > > > work > > > > > > > with and get money from several of those - which would > basically > > > slow > > > > > > down > > > > > > > active development and maintenance for Airflow to a halt - > > because > > > > even > > > > > > if > > > > > > > we had a lot of people willing to contribute, maintainers would > > > have > > > > very > > > > > > > little - own - time to keep things running. There is a fine > > balance > > > > that > > > > > > we > > > > > > > keep now between the open-source and stakeholders, and > > open-source > > > > > > product > > > > > > > "friction" is an important property that the balance is built > on. > > > > > > > > > > > > > > J. > > > > > > > > > > > > > > > > > > > > > On Wed, Jul 9, 2025 at 9:21 PM Oliveira, Niko > > > > > > <[email protected]> > > > > > > > wrote: > > > > > > > > > > > > > > > To me, I'm always working from a user perspective. My goal is > > to > > > > make > > > > > > > > their lives easier, their deployments easier, the product the > > > most > > > > > > > > enjoyable for them to use. To me, the best user experience is > > > that > > > > they > > > > > > > > should enable bundle versioning and it should just work with > as > > > > little > > > > > > or > > > > > > > > no extra steps and with as little infra as possible, and with > > the > > > > > > fewest > > > > > > > > possible pit falls for them to fall into. From a user > > > perspective, > > > > > > they've > > > > > > > > already provisioned a database for airflow metadata, why is > > this > > > > > > portion of > > > > > > > > metadata leaking out to other forms of external storage? Now > > this > > > > is > > > > > > > > another resource they now need to be aware of and manage the > > > > lifecycle > > > > > > of > > > > > > > > (or allow us write access into their accounts to manage for > > > them). > > > > > > > > > > > > > > > > Ultimately, we should not be afraid of doing sometimes > > difficult > > > > work > > > > > > to > > > > > > > > make a good product for our users, it's for them in the end > :) > > > > > > > > > > > > > > > > However, I see your perspectives as well, making our code and > > DB > > > > > > > > management more complex is more work and complication for us. > > And > > > > from > > > > > > the > > > > > > > > feedback so far I'm out voted, so I'm happy as always to > > disagree > > > > and > > > > > > > > commit, and do as you wish :) > > > > > > > > > > > > > > > > Thanks for the feedback everyone! > > > > > > > > > > > > > > > > Cheers, > > > > > > > > Niko > > > > > > > > > > > > > > > > ________________________________ > > > > > > > > From: Jens Scheffler <[email protected]> > > > > > > > > Sent: Wednesday, July 9, 2025 12:07:08 PM > > > > > > > > To: [email protected] > > > > > > > > Subject: RE: [EXT] S3 Dag Bundle Versions and DB Manager > > > > > > > > > > > > > > > > CAUTION: This email originated from outside of the > > organization. > > > > Do not > > > > > > > > click links or open attachments unless you can confirm the > > sender > > > > and > > > > > > know > > > > > > > > the content is safe. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > AVERTISSEMENT: Ce courrier électronique provient d’un > > expéditeur > > > > > > externe. > > > > > > > > Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si > > vous > > > > ne > > > > > > pouvez > > > > > > > > pas confirmer l’identité de l’expéditeur et si vous n’êtes > pas > > > > certain > > > > > > que > > > > > > > > le contenu ne présente aucun risque. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > My 2ct on the discussions are similar like the opinions > before. > > > > > > > > > > > > > > > > From my Edge3 experience migrating DB from provider - even > if > > > > > > > > technically enabled - is a bit of a pain. Adding a lot of > > > > boilerplate, > > > > > > > > you need to consider your provider should also still be > > > compatible > > > > with > > > > > > > > AF2 (I assume) and once a user wants to downgrade it is a bit > > of > > > > manual > > > > > > > > effort to downgrade DB as well. > > > > > > > > > > > > > > > > As long as we are not adding a generic Key/Value store to > core > > > > (similar > > > > > > > > liek Variables but for general purpose internal use not > exposed > > > to > > > > > > users > > > > > > > > - but then in case of trougbleshooting how to "manage/admin > > it?) > > > I > > > > > > would > > > > > > > > also see it like Terraform - a secondary bucked for state os > > > cheap > > > > and > > > > > > > > convenient. Yes write access would be needed but only for > > > Airflow. > > > > And > > > > > > > > as it is separated from other should not be a general > security > > > > harm... > > > > > > > > just a small deployment complexity. And I assume versining is > > > > optional. > > > > > > > > So no requirement to have it on per default and if a user > wants > > > to > > > > move > > > > > > > > to/enable versioing then just the state bucket would need to > be > > > > added > > > > > > to > > > > > > > > Bundle-config? > > > > > > > > > > > > > > > > TLDR I would favor a bucket, else if DB is the choice then a > > > common > > > > > > > > solution in core might be easier than a DB handling in > > provider. > > > > But > > > > > > > > would also not block any other, just from point of complexity > > I'd > > > > not > > > > > > > > favor provider specifc DB tables. > > > > > > > > > > > > > > > > Jens > > > > > > > > > > > > > > > > On 09.07.25 19:57, Jarek Potiuk wrote: > > > > > > > > > What about the DynamoDB idea ? What you are trying to > > trade-off > > > > is > > > > > > > > "writing > > > > > > > > > to airflow metadata DB" with "writing to another DB" > really. > > So > > > > yes > > > > > > it > > > > > > > > is - > > > > > > > > > another thing you will need to have access to write to - > > other > > > > than > > > > > > > > Airflow > > > > > > > > > DB, but it's really the question should the boundaries be > on > > > > > > "Everything > > > > > > > > > writable should be in Airflow" vs. "Everything writable > > should > > > > be in > > > > > > the > > > > > > > > > "cloud" that the integration is about. > > > > > > > > > > > > > > > > > > Yes - it makes the management using S3 versioning a bit > more > > > > > > "write-y" - > > > > > > > > > but on the other hand it does allow to confine complexity > to > > a > > > > pure > > > > > > > > > "amazon" provider - with practically 0 impact on Airflow > > core > > > > and > > > > > > > > airflow > > > > > > > > > DB. Which I really like to be honest. > > > > > > > > > > > > > > > > > > And yes "co-location" is also my goal. And I think this is > a > > > > perfect > > > > > > way > > > > > > > > to > > > > > > > > > explain it as well why it is better to keep "S3 versioning" > > > > close to > > > > > > "S3" > > > > > > > > > and not to Airflow - especially that there will be a lot of > > > > > > "S3-specific" > > > > > > > > > things in the state that are not easy to abstract and have > > > > "common" > > > > > > for > > > > > > > > > other Airflow versioning implementations. > > > > > > > > > > > > > > > > > > You can think about it this way: > > > > > > > > > > > > > > > > > > Airflow has already done its job with abstractions - > > versioning > > > > > > changes > > > > > > > > and > > > > > > > > > metadata DB is implemented in Airflow DB. If there are any > > > > missing > > > > > > pieces > > > > > > > > > in the abstraction that will be usable across multiple > > > > > > implementations of > > > > > > > > > versioning, we should - of course - add it to Airflow > > metadata > > > > DB - > > > > > > in > > > > > > > > the > > > > > > > > > way that they can be used by those different > implementations. > > > > But the > > > > > > > > code > > > > > > > > > to manage and use it should be in airflow-core. > > > > > > > > > If there is anything specific for the implementation of S3 > / > > > > Amazon > > > > > > > > > integration -> it should be implemented independently from > > > > Airflow > > > > > > > > Metadata > > > > > > > > > DB. There are many complexities in managing and upgrading > > core > > > DB > > > > > > and we > > > > > > > > > should not use the db to make provider-specific things. The > > > > > > discussion > > > > > > > > > about shared code and isolation is interesting in this > > context. > > > > > > Because I > > > > > > > > > think we might get to the point when we go deeper and > deeper > > in > > > > this > > > > > > > > > direction that we will have (and we already do it more or > > less) > > > > NO > > > > > > > > > (regular) providers needed with whatever CLI or tooling we > > will > > > > need > > > > > > to > > > > > > > > > manage the Metadata DB. FAB and Edge are currently > > exceptions - > > > > but > > > > > > they > > > > > > > > > are by no means "regular" providers. > > > > > > > > > > > > > > > > > > So I'd say - if while designing/ implementing S3 versioning > > you > > > > will > > > > > > see > > > > > > > > > that part of the implementation can be abstracted away and > > > added > > > > to > > > > > > the > > > > > > > > > core and used by other implementations - 100% - let's add > it > > to > > > > the > > > > > > core. > > > > > > > > > But only then. If it is something that only Amazon provider > > > needs > > > > > > and S3 > > > > > > > > > needs - let's make it use Amazon **whatever** as backing > > > storage. > > > > > > > > > > > > > > > > > > I would even say - talk to the Google team and try to come > up > > > > with an > > > > > > > > > abstraction that can be used for versioning in both S3 and > > GCS, > > > > > > agree on > > > > > > > > > it, and let's see if this abstraction should find its way > to > > > the > > > > > > core. > > > > > > > > That > > > > > > > > > would be my proposal. > > > > > > > > > > > > > > > > > > J. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Jul 9, 2025 at 7:37 PM Oliveira, Niko > > > > > > > > <[email protected]> > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > >> Thanks for engaging folks! > > > > > > > > >> > > > > > > > > >> I don’t love the idea of using another bucket. For one, > this > > > > means > > > > > > > > Airflow > > > > > > > > >> needs write access to S3 which is not ideal; some > > > > users/customers > > > > > > are > > > > > > > > very > > > > > > > > >> sensitive about ever allowing write access to things. And > > two, > > > > you > > > > > > will > > > > > > > > >> commonly get issues with a design that leaks state into > > > customer > > > > > > managed > > > > > > > > >> accounts/resources, they may delete the bucket not knowing > > > what > > > > it > > > > > > is, > > > > > > > > they > > > > > > > > >> may not migrate it to a new account or region if they ever > > > > move. I > > > > > > think > > > > > > > > >> it’s best for the data to be stored transparently to the > > user > > > > and > > > > > > > > >> co-located with the data it strongly relates to (i.e. the > > dag > > > > runs > > > > > > that > > > > > > > > are > > > > > > > > >> associated with those bundle versions). > > > > > > > > >> > > > > > > > > >> Is using DB Manager completely unacceptable these days? > What > > > are > > > > > > folks' > > > > > > > > >> thoughts on that? > > > > > > > > >> > > > > > > > > >> Cheers, > > > > > > > > >> Niko > > > > > > > > >> > > > > > > > > >> ________________________________ > > > > > > > > >> From: Jarek Potiuk <[email protected]> > > > > > > > > >> Sent: Wednesday, July 9, 2025 6:23:54 AM > > > > > > > > >> To: [email protected] > > > > > > > > >> Subject: RE: [EXT] S3 Dag Bundle Versions and DB Manager > > > > > > > > >> > > > > > > > > >> CAUTION: This email originated from outside of the > > > > organization. Do > > > > > > not > > > > > > > > >> click links or open attachments unless you can confirm the > > > > sender > > > > > > and > > > > > > > > know > > > > > > > > >> the content is safe. > > > > > > > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > > >> AVERTISSEMENT: Ce courrier électronique provient d’un > > > expéditeur > > > > > > > > externe. > > > > > > > > >> Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe > si > > > > vous ne > > > > > > > > pouvez > > > > > > > > >> pas confirmer l’identité de l’expéditeur et si vous n’êtes > > pas > > > > > > certain > > > > > > > > que > > > > > > > > >> le contenu ne présente aucun risque. > > > > > > > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > > >>> Another option also would be Using dynamodb table? that > > also > > > > > > supports > > > > > > > > >> snapshots and i feel it works very well with state > > management. > > > > > > > > >> > > > > > > > > >> Yep that would also work. > > > > > > > > >> > > > > > > > > >> Anything "Amazon" to keep state would do. I think that it > > > > should be > > > > > > our > > > > > > > > >> "default" approach that if we have to keep state and the > > state > > > > is > > > > > > > > connected > > > > > > > > >> with specific "provider's" implementation, it's best to > not > > > keep > > > > > > state > > > > > > > > in > > > > > > > > >> Airflow, but in the "integration" that the provider works > > with > > > > if > > > > > > > > possible. > > > > > > > > >> We cannot do it in "generic" case because we do not know > > what > > > > > > > > >> "integrations" the user has - but since this is > "provider's" > > > > > > > > functionality, > > > > > > > > >> using anything else that the given integration provides > > makes > > > > > > perfect > > > > > > > > >> sense. > > > > > > > > >> > > > > > > > > >> J. > > > > > > > > >> > > > > > > > > >> > > > > > > > > >> On Wed, Jul 9, 2025 at 3:12 PM Pavankumar Gopidesu < > > > > > > > > >> [email protected]> > > > > > > > > >> wrote: > > > > > > > > >> > > > > > > > > >>> Agree another s3 bucket also works here > > > > > > > > >>> > > > > > > > > >>> Another option also would be Using dynamodb table? that > > also > > > > > > supports > > > > > > > > >>> snapshots and i feel it works very well with state > > > management. > > > > > > > > >>> > > > > > > > > >>> > > > > > > > > >>> Pavan > > > > > > > > >>> > > > > > > > > >>> On Wed, Jul 9, 2025 at 2:06 PM Jarek Potiuk < > > > [email protected]> > > > > > > wrote: > > > > > > > > >>> > > > > > > > > >>>> One of the options would be to use a similar approach as > > > > terraform > > > > > > > > >> uses - > > > > > > > > >>>> i.e. use dedicated "metadata" state storage in a > DIFFERENT > > > s3 > > > > > > bucket > > > > > > > > >> than > > > > > > > > >>>> DAG files. Since we know there must be an S3 available > > > > > > (obviously) - > > > > > > > > it > > > > > > > > >>>> seems not too excessive to assume that there might be > > > another > > > > > > bucket, > > > > > > > > >>>> independent of the DAG bucket where the state is stored > - > > > same > > > > > > bucket > > > > > > > > >>> (and > > > > > > > > >>>> dedicated connection id) could even be used to store > state > > > for > > > > > > > > multiple > > > > > > > > >>> S3 > > > > > > > > >>>> dag bundles - each Dag bundle could have a dedicated > > object > > > > > > storing > > > > > > > > the > > > > > > > > >>>> state. The metadata is not huge, so continuously reading > > and > > > > > > replacing > > > > > > > > >> it > > > > > > > > >>>> should not be an issue. > > > > > > > > >>>> > > > > > > > > >>>> What's nice about it - this single object could even > > > > > > **actually** > > > > > > > > use > > > > > > > > >> S3 > > > > > > > > >>>> versioning to keep historical state - to optimize > things > > > and > > > > > > keep a > > > > > > > > >> log > > > > > > > > >>> of > > > > > > > > >>>> changes potentially. > > > > > > > > >>>> > > > > > > > > >>>> J. > > > > > > > > >>>> > > > > > > > > >>>> On Wed, Jul 9, 2025 at 3:01 AM Oliveira, Niko > > > > > > > > >>> <[email protected] > > > > > > > > >>>> wrote: > > > > > > > > >>>> > > > > > > > > >>>>> Hey folks, > > > > > > > > >>>>> > > > > > > > > >>>>> tl;dr I’d like to get some thoughts on a proposal to > use > > DB > > > > > > Manager > > > > > > > > >> for > > > > > > > > >>>> S3 > > > > > > > > >>>>> Dag Bundle versioning. > > > > > > > > >>>>> > > > > > > > > >>>>> The initial commit for S3 Dag Bundles was recently > merged > > > [1] > > > > > > but it > > > > > > > > >>>> lacks > > > > > > > > >>>>> Bundle versioning (since this isn’t trivial with > > something > > > > like > > > > > > S3, > > > > > > > > >>> like > > > > > > > > >>>> it > > > > > > > > >>>>> is with Git). The proposed solution involves building a > > > > snapshot > > > > > > of > > > > > > > > >> the > > > > > > > > >>>> S3 > > > > > > > > >>>>> bucket at the time each Bundle version is created, > noting > > > the > > > > > > version > > > > > > > > >>> of > > > > > > > > >>>>> all the objects in the bucket (using S3’s native bucket > > > > > > versioning > > > > > > > > >>>> feature) > > > > > > > > >>>>> and creating a manifest to store those versions and > then > > > > giving > > > > > > that > > > > > > > > >>>> whole > > > > > > > > >>>>> manifest itself some unique id/version/uuid. These > > > manifests > > > > now > > > > > > need > > > > > > > > >>> to > > > > > > > > >>>> be > > > > > > > > >>>>> stored somewhere for future use/retrieval. The proposal > > is > > > to > > > > > > use the > > > > > > > > >>>>> Airflow database using the DB Manager feature. Other > > > options > > > > > > include > > > > > > > > >>>> using > > > > > > > > >>>>> the local filesystem to store them (but this obviously > > wont > > > > work > > > > > > in > > > > > > > > >>>>> Airflow’s distributed architecture) or the S3 bucket > > itself > > > > (but > > > > > > this > > > > > > > > >>>>> requires write access to the bucket and we will always > be > > > at > > > > the > > > > > > > > >> mercy > > > > > > > > >>> of > > > > > > > > >>>>> the user accidentally deleting/modifying the manifests > as > > > > they > > > > > > try to > > > > > > > > >>>>> manage the lifecycle of their bucket, they should not > > need > > > > to be > > > > > > > > >> aware > > > > > > > > >>> of > > > > > > > > >>>>> or need to account for this metadata). So the Airflow > DB > > > > works > > > > > > nicely > > > > > > > > >>> as > > > > > > > > >>>> a > > > > > > > > >>>>> persistent and internally accessible location for this > > > data. > > > > > > > > >>>>> > > > > > > > > >>>>> But I’m aware of the complexities of using the DB > Manager > > > > and the > > > > > > > > >>>>> discussion we had during the last dev call about > > providers > > > > > > vending DB > > > > > > > > >>>>> tables (concerning migrations and ensuring smooth > > upgrades > > > or > > > > > > > > >>> downgrades > > > > > > > > >>>> of > > > > > > > > >>>>> the schema). So I wanted to reach out to see what folks > > > > thought. > > > > > > I > > > > > > > > >> have > > > > > > > > >>>>> talked to Jed, the Bundle Master (tm), and we haven’t > > come > > > up > > > > > > with > > > > > > > > >>>> anything > > > > > > > > >>>>> else that solves the problem as cleanly, so the DB > > Manager > > > is > > > > > > still > > > > > > > > >> my > > > > > > > > >>>> top > > > > > > > > >>>>> choice. I think what we go with will pave the way for > > other > > > > > > Bundle > > > > > > > > >>>>> providers of a similar type as well, so it's worth > > thinking > > > > > > deeply > > > > > > > > >>> about > > > > > > > > >>>>> this decision. > > > > > > > > >>>>> > > > > > > > > >>>>> Let me know what you think and thanks for your time! > > > > > > > > >>>>> > > > > > > > > >>>>> Cheers, > > > > > > > > >>>>> Niko > > > > > > > > >>>>> > > > > > > > > >>>>> [1] https://github.com/apache/airflow/pull/46621 > > > > > > > > >>>>> > > > > > > > > > > > > > > > > > > > > --------------------------------------------------------------------- > > > > > > > > To unsubscribe, e-mail: [email protected] > > > > > > > > For additional commands, e-mail: [email protected] > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > --------------------------------------------------------------------- > > > > > > To unsubscribe, e-mail: [email protected] > > > > > > For additional commands, e-mail: [email protected] > > > > > > > > > > > > > > > > > > > > > > > > > --------------------------------------------------------------------- > > > > To unsubscribe, e-mail: [email protected] > > > > For additional commands, e-mail: [email protected] > > > > > > > > > > > > > >
