Original thread: https://lists.apache.org/thread/5cvdrdgsqr28p4lxzbwjwzx58yg9vbcy
On 2026/07/29 17:06:49 Ephraim Anierobi wrote: > Hi all, > > I want to revive this thread. Niko has a PR open now that touches this area > (https://github.com/apache/airflow/pull/70207) and the direction question > we discussed here is coming up again in its review, so it feels better to > continue that discussion on the list than on the PR. > > First a small clarification for the record. Reading back through the > thread, what did not get support was storing the manifests through DB > Manager, so provider owned tables and provider owned migrations. A table in > core Airflow models was not rejected. Jens even said that if the DB is the > choice then a common solution in core would be easier than DB handling in a > provider. On the PR, Jed suggested a mapping table keyed on bundle name and > version, and I don't think this thread rules that out. > > That said, my own preference is still the direction Jarek and Jens argued > for here, which is to keep this state out of the database and put it in the > store itself. The database then only keeps the version string that each run > already records, and the store can always turn that string back into the > code. Nothing in the database ever becomes the only copy of something that > is needed to rebuild a bundle. > > I wanted to see if that works in practice rather than argue it in theory, > so I built an experiment. The repo is > https://github.com/ephraimbuddy/airflow-manifest-bundle and there is a > design doc at > https://github.com/ephraimbuddy/airflow-manifest-bundle/blob/main/docs/design.md. > The bundle version is a sha256 content hash and the hash alone is enough to > find the code again, the same way a git sha resolves against the object > store. Snapshots and their manifests live under a published root, which can > be an s3 prefix or a shared filesystem. The manifest object is written last > so a snapshot only becomes visible once it is complete. The release pointer > uses S3 conditional writes instead of a lock file. The dag source prefix > stays read only in every mode and workers need nothing more than GetObject > on the releases prefix, and nothing at all in the filesystem mode. It also > waits for the source to be stable before it publishes, so a half synced > bucket never becomes a release. It runs unchanged on Airflow 3.0.0 through > 3.3.x with no core changes, which I think shows the external path does not > need anything from core. > > On why I used a content hash instead of S3 versionIds in the experiment. I > wanted the version string alone to be enough to find and check the code. A > content hash can always be recomputed from the files, so the manifest is > never the only copy of anything and there is no extra state that needs a > safe home. That is what lets the whole storage question fall away. Every > fetched file is checked against its hash, so the chain verifies itself end > to end. A hash also works the same on any storage, so the published root > can be a bucket today, a different bucket later, or a shared filesystem, > and nothing needs versioning enabled on the source. VersionIds have a real > advantage in that nothing gets copied. If that matters there is a possible > hybrid where the file entries are versionId references but the manifest > itself is stored content addressed in the bucket. That keeps the use of > native S3 versioning and still keeps the state out of the DB. > > To be clear I am not pushing the package itself. I am happy to help move > whatever pieces are useful into the amazon provider, and Niko, happy to > align with whatever you have in flight. The goal is one good versioned S3 > bundle. > > Thoughts? > > - Ephraim > > On Fri, 18 Jul 2025 at 06:44, Zhe You Liu <[email protected]> wrote: > > > > This is a good solution. It goes along the idea of a "generic" solution > > > that does not need an "amazon specific" table and DB manager. If the > > > manifest serialized field can be used for all other "bundles" (even if > > > manifest format itself is specific to S3 bundle), I am very happy with > > that > > > solution. > > > > Glad to hear that. Let’s wait for input from others as well, there may be > > concerns I haven’t fully considered. > > > > > One thing to consider (but this is entirely up to the S3 bundle > > > implementation) is handling versioning of such manifest during > > > serialization/deserialization to allow downgrading and upgrading the > > > provider seamlessly. > > > > Nice point and I agree. Regardless of which approach we take (storing in > > the DB or in object storage), we’ll need to handle serialization properly > > and ensure backward compatibility. > > > > Best, > > Jason Liu > > > > > > On 2025/07/18 04:59:45 Jarek Potiuk wrote: > > > > In my opinion, we can simply add an optional `manifest` field (or > > another > > > suitable name). I don’t think we need to introduce a new table via > > > DbManager; an additional field for storing metadata about the external > > > state (such as prefix and object versions for all dags in the bundle, in > > > the case of S3DagBundle) should suffice. We could introduce a new parent > > > subclass, such as `RemoteDagBundle` or `ObjectStoreDagBundle`, in the > > > common provider to define the structure for serializing and deserializing > > > the `manifest` field. > > > > > > This is a good solution. It goes along the idea of a "generic" solution > > > that does not need an "amazon specific" table and DB manager. If the > > > manifest serialized field can be used for all other "bundles" (even if > > > manifest format itself is specific to S3 bundle), I am very happy with > > that > > > solution. One thing to consider (but this is entirely up to the S3 bundle > > > implementation) is handling versioning of such manifest during > > > serialization/deserialization to allow downgrading and upgrading the > > > provider seamlessly. > > > > > > > > > > > > On Fri, Jul 18, 2025 at 5:56 AM Zhe You Liu <[email protected]> wrote: > > > > > > > Sorry for the late response. > > > > > > > > Both approaches work for me; I just wanted to share my opinion as we > > > > settle on a final decision. > > > > > > > > From my perspective, the DagBundle acts as a client that pulls external > > > > state and stores only the version identifier in the Airflow metadata > > DB. > > > > > > > > For example, with GitDagBundle, the Git repository serves as the > > external > > > > storage. The GitDagBundle pulls DAG files locally and stores the commit > > > > hash as the `version` field in `DagBundleModel.version`. > > > > > > > > 1. If we choose to store the manifest in the Airflow metadata DB: > > > > > > > > In my opinion, we can simply add an optional `manifest` field (or > > another > > > > suitable name). I don’t think we need to introduce a new table via > > > > DbManager; an additional field for storing metadata about the external > > > > state (such as prefix and object versions for all dags in the bundle, > > in > > > > the case of S3DagBundle) should suffice. We could introduce a new > > parent > > > > subclass, such as `RemoteDagBundle` or `ObjectStoreDagBundle`, in the > > > > common provider to define the structure for serializing and > > deserializing > > > > the `manifest` field. > > > > > > > > 2. If we decide to store the manifest outside the Airflow metadata DB: > > > > > > > > We will need to clarify: > > > > > > > > a) The required parameters for all DagBundles that pull DAGs from > > object > > > > storage. Based on the discussion above, we would need the `conn_id`, > > > > `bucket`, and `prefix` for the manifest file. > > > > > > > > b) The interface for calculating the bundle version based on the > > external > > > > state or DAG content hash. > > > > > > > > Here is a concrete example of how the manifest could be stored: > > > > https://github.com/apache/airflow/pull/46621#issuecomment-3078208467 > > > > > > > > Thank you all for the insightful discussion! > > > > > > > > Best, > > > > Jason > > > > > > > > On 2025/07/10 21:56:31 "Oliveira, Niko" wrote: > > > > > Thanks for the reply Jarek :) > > > > > > > > > > Indeed we have different philosophies about this so we will certainly > > > > keep going in circles about where to draw the line on making things > > easy > > > > and enjoyable to use, whether to intentionally add friction or not, > > etc, > > > > etc. > > > > > > > > > > I think if we have optional paths to take and it's not immensely > > harder > > > > we should err on the side of making OSS Airflow as good as it can be, > > > > despite whatever managed services we have in the community. I'm not > > sure > > > > where it has come from recently but this new push to make Airflow > > > > intentionally hard to use so that managed services stay in business is > > a > > > > bit unsettling. We're certainly not asking for that, and those around > > that > > > > I've chatted to (since I'm now seeing this mentioned frequently) are > > also > > > > not asking for this. I'm curious where this new pressure is coming > > from and > > > > why you feel it recently. > > > > > > > > > > But regardless of the curiosity above, I'll return to the drawing > > board, > > > > and see what else can be done for this particular problem. If there are > > > > other Bundle types who need to solve the same problem perhaps we can > > find a > > > > more acceptable implementation in Airflow core to support this. And if > > not, > > > > I'll proceed with externalizing the storage of the S3 Bundle version > > > > metadata outside of Airflow. > > > > > > > > > > Cheers, > > > > > Niko > > > > > > > > > > ________________________________ > > > > > From: Jarek Potiuk <[email protected]> > > > > > Sent: Wednesday, July 9, 2025 11:59:06 PM > > > > > To: [email protected] > > > > > Subject: RE: [EXT] S3 Dag Bundle Versions and DB Manager > > > > > > > > > > CAUTION: This email originated from outside of the organization. Do > > not > > > > click links or open attachments unless you can confirm the sender and > > know > > > > the content is safe. > > > > > > > > > > > > > > > > > > > > AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur > > > > externe. Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si > > vous > > > > ne pouvez pas confirmer l’identité de l’expéditeur et si vous n’êtes > > pas > > > > certain que le contenu ne présente aucun risque. > > > > > > > > > > > > > > > > > > > > > To me, I'm always working from a user perspective. My goal is to > > make > > > > > their lives easier, their deployments easier, the product the most > > > > > enjoyable for them to use. To me, the best user experience is that > > they > > > > > should enable bundle versioning and it should just work with as > > little or > > > > > no extra steps and with as little infra as possible, and with the > > fewest > > > > > possible pit falls for them to fall into. From a user perspective, > > > > they've > > > > > already provisioned a database for airflow metadata, why is this > > portion > > > > of > > > > > metadata leaking out to other forms of external storage? Now this is > > > > > another resource they now need to be aware of and manage the > > lifecycle of > > > > > (or allow us write access into their accounts to manage for them). > > > > > > > > > > > > > > > *TL;DR; I think our goal in open-source is to have frictionless and > > "out > > > > of > > > > > the box" experience only for basic cases, but not for more complex > > > > > deployments.* > > > > > > > > > > It's a long read if you want to read it .. so beware :). > > > > > > > > > > I think that is an important "optimization goal" for sure to provide > > > > > frictionless and enjoyable experience - but I think it's one of many > > > > goals > > > > > that are sometimes contradicting with long term open-source project > > > > > sustainability and it's very import to clarify which "user" we are > > > > talking > > > > > about. > > > > > > > > > > To be honest, I am not sure that our goal should be "airflow should > > work > > > > > out of the box in case of integration with external services in > > > > production' > > > > > if it complicates our code and makes it service-dependent - and as > > Jens > > > > > noticed, if we can come up with a "generic" thing that can be > > reusable > > > > > across multiple services, we can invest more in making it works "out > > of > > > > the > > > > > box", but if you anyhow need to integrate and make work with external > > > > > service, it adds very little "deployment complexity" to use another > > piece > > > > > of the service - and this is basically the job of deployment manager > > > > > anyway. > > > > > > > > > > The "just work" goal as I see it should only cover those individual > > users > > > > > who want to try and use airflow in it's basic form and "standalone" > > > > > configuration - not for "deployment managers". > > > > > > > > > > I think yes - our goal should be to make things extremely easy for > > users > > > > > who want to use airflow in its basic form where things should **just > > > > > work**. Like "docker run -it apache/airflow standalone" - this is > > what > > > > > currently **just works**, 0 configuration, 0 work for external > > > > > integrations, and we even had a discussion that we could make it "low > > > > > production ready" (which I think we could - just implement automated > > > > > backup/recovery of sqlite db and maybe document mounting a folder > > with > > > > DAGs > > > > > and db, better handling of logs rather than putting them as mixed > > output > > > > on > > > > > stdout and we are practically done). But when you add "S3" as the dag > > > > > storage you already need to make a lot of decisions - mostly about > > > > service > > > > > accounts, security, access, versioning, backup of the s3 objects, > > etc. > > > > etc. > > > > > And that's not a "standalone user' case - that is a "deployment > > manager" > > > > > work (where "deployment manager" is a role - not necessarily title > > of the > > > > > job you have. > > > > > > > > > > I think - and that is a bit of philosophical - but I've been talking > > > > about > > > > > it to Maciek Obuchowski yesterday - that there is a pretty clear > > boundary > > > > > of what open-source solutions delivers and it should match > > expectations > > > > of > > > > > people using it. Maintainers and community developing open-source > > should > > > > > mostly deliver a working, generic solutions that are extendable with > > > > > various deployment options and we should make it possible for those > > > > > deployments to happen - and provide building blocks for them. But > > it's > > > > > "deployment manager" work to make sure to put things together and > > make it > > > > > works. And we should not do it "for them". It's their job to figure > > out > > > > how > > > > > to configure and set-up things, make backups, set security boundaries > > > > etc. > > > > > - we should make it possible, document the options, document security > > > > model > > > > > and make it "easy" to configure things - but there should not be an > > > > > expectation from the deploiyment manager that it "just works". > > > > > > > > > > And I think your approach is perfectly fine - but only for "managed > > > > > services" - there, indeed manage service user's expectations can be > > that > > > > > things "just work" and they are willing to pay for it with real > > money, > > > > > rather than their time and effort to make it so. And there I think, > > those > > > > > who deliver such a service should have the "just work" as primary > > goal - > > > > > also because users will have such expectations - because they > > actually > > > > pay > > > > > for it to "just work". Not so much for open-source product - where > > "just > > > > > work" often involves complexity, additional maintenance overhead and > > > > making > > > > > opinionated decisions on "how it just works". For those "managed > > service" > > > > > teams - "just work" is very much a primary goal. But for "open > > source > > > > > community" - having such a goal is actually not good - it's > > dangerous > > > > > because it might result in wrong expectations from the users. If we > > start > > > > > making airflow "just works" in all kinds of deployment with zero work > > > > from > > > > > the users who want to deploy it in production and at scale, they will > > > > > expect it to happen for everything - why don't we have automated log > > > > > trimming, why don't we have automated backup of the Database, why > > don't > > > > we > > > > > auto vacuum the db, why don't we provide one-click deployment option > > on > > > > > AWS. GCS. Azure, why don't we provide DDOS protection in our > > webserver, > > > > why > > > > > don't we ..... you name it. > > > > > > > > > > That's a bit of philosophy - those are the same assumptions and goals > > > > that > > > > > I had in mind when designing multi-team - and there it's also why we > > had > > > > > different views - I just feel that some level of friction is a > > "property" > > > > > of open-source product. > > > > > > > > > > Also a bit of "business" side - this is also "good" for those who > > provide > > > > > managed services and airflow to keep sustainable open-source business > > > > model > > > > > working - because what people are paying them is precisely to > > "remove the > > > > > friction". If take the "frictionless user experience" goal case to > > > > extreme > > > > > - Airflow would essentially be killed IMHO. Imagine if Airflow would > > be > > > > > frictioness for all kinds of deployments and had "everything" > > working out > > > > > of the box. There would be no business for any of the managed > > services > > > > > (because users would not need to pay for it). Then we would only have > > > > users > > > > > who expect thigns to "just work" and most of them would not even > > think > > > > > about contributing back. And there would be no managed services > > people > > > > > (like you) whose job is paid by the services - or people like me who > > > > work > > > > > with and get money from several of those - which would basically slow > > > > down > > > > > active development and maintenance for Airflow to a halt - because > > even > > > > if > > > > > we had a lot of people willing to contribute, maintainers would have > > very > > > > > little - own - time to keep things running. There is a fine balance > > that > > > > we > > > > > keep now between the open-source and stakeholders, and open-source > > > > product > > > > > "friction" is an important property that the balance is built on. > > > > > > > > > > J. > > > > > > > > > > > > > > > On Wed, Jul 9, 2025 at 9:21 PM Oliveira, Niko > > > > <[email protected]> > > > > > wrote: > > > > > > > > > > > To me, I'm always working from a user perspective. My goal is to > > make > > > > > > their lives easier, their deployments easier, the product the most > > > > > > enjoyable for them to use. To me, the best user experience is that > > they > > > > > > should enable bundle versioning and it should just work with as > > little > > > > or > > > > > > no extra steps and with as little infra as possible, and with the > > > > fewest > > > > > > possible pit falls for them to fall into. From a user perspective, > > > > they've > > > > > > already provisioned a database for airflow metadata, why is this > > > > portion of > > > > > > metadata leaking out to other forms of external storage? Now this > > is > > > > > > another resource they now need to be aware of and manage the > > lifecycle > > > > of > > > > > > (or allow us write access into their accounts to manage for them). > > > > > > > > > > > > Ultimately, we should not be afraid of doing sometimes difficult > > work > > > > to > > > > > > make a good product for our users, it's for them in the end :) > > > > > > > > > > > > However, I see your perspectives as well, making our code and DB > > > > > > management more complex is more work and complication for us. And > > from > > > > the > > > > > > feedback so far I'm out voted, so I'm happy as always to disagree > > and > > > > > > commit, and do as you wish :) > > > > > > > > > > > > Thanks for the feedback everyone! > > > > > > > > > > > > Cheers, > > > > > > Niko > > > > > > > > > > > > ________________________________ > > > > > > From: Jens Scheffler <[email protected]> > > > > > > Sent: Wednesday, July 9, 2025 12:07:08 PM > > > > > > To: [email protected] > > > > > > Subject: RE: [EXT] S3 Dag Bundle Versions and DB Manager > > > > > > > > > > > > CAUTION: This email originated from outside of the organization. > > Do not > > > > > > click links or open attachments unless you can confirm the sender > > and > > > > know > > > > > > the content is safe. > > > > > > > > > > > > > > > > > > > > > > > > AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur > > > > externe. > > > > > > Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous > > ne > > > > pouvez > > > > > > pas confirmer l’identité de l’expéditeur et si vous n’êtes pas > > certain > > > > que > > > > > > le contenu ne présente aucun risque. > > > > > > > > > > > > > > > > > > > > > > > > My 2ct on the discussions are similar like the opinions before. > > > > > > > > > > > > From my Edge3 experience migrating DB from provider - even if > > > > > > technically enabled - is a bit of a pain. Adding a lot of > > boilerplate, > > > > > > you need to consider your provider should also still be compatible > > with > > > > > > AF2 (I assume) and once a user wants to downgrade it is a bit of > > manual > > > > > > effort to downgrade DB as well. > > > > > > > > > > > > As long as we are not adding a generic Key/Value store to core > > (similar > > > > > > liek Variables but for general purpose internal use not exposed to > > > > users > > > > > > - but then in case of trougbleshooting how to "manage/admin it?) I > > > > would > > > > > > also see it like Terraform - a secondary bucked for state os cheap > > and > > > > > > convenient. Yes write access would be needed but only for Airflow. > > And > > > > > > as it is separated from other should not be a general security > > harm... > > > > > > just a small deployment complexity. And I assume versining is > > optional. > > > > > > So no requirement to have it on per default and if a user wants to > > move > > > > > > to/enable versioing then just the state bucket would need to be > > added > > > > to > > > > > > Bundle-config? > > > > > > > > > > > > TLDR I would favor a bucket, else if DB is the choice then a common > > > > > > solution in core might be easier than a DB handling in provider. > > But > > > > > > would also not block any other, just from point of complexity I'd > > not > > > > > > favor provider specifc DB tables. > > > > > > > > > > > > Jens > > > > > > > > > > > > On 09.07.25 19:57, Jarek Potiuk wrote: > > > > > > > What about the DynamoDB idea ? What you are trying to trade-off > > is > > > > > > "writing > > > > > > > to airflow metadata DB" with "writing to another DB" really. So > > yes > > > > it > > > > > > is - > > > > > > > another thing you will need to have access to write to - other > > than > > > > > > Airflow > > > > > > > DB, but it's really the question should the boundaries be on > > > > "Everything > > > > > > > writable should be in Airflow" vs. "Everything writable should > > be in > > > > the > > > > > > > "cloud" that the integration is about. > > > > > > > > > > > > > > Yes - it makes the management using S3 versioning a bit more > > > > "write-y" - > > > > > > > but on the other hand it does allow to confine complexity to a > > pure > > > > > > > "amazon" provider - with practically 0 impact on Airflow core > > and > > > > > > airflow > > > > > > > DB. Which I really like to be honest. > > > > > > > > > > > > > > And yes "co-location" is also my goal. And I think this is a > > perfect > > > > way > > > > > > to > > > > > > > explain it as well why it is better to keep "S3 versioning" > > close to > > > > "S3" > > > > > > > and not to Airflow - especially that there will be a lot of > > > > "S3-specific" > > > > > > > things in the state that are not easy to abstract and have > > "common" > > > > for > > > > > > > other Airflow versioning implementations. > > > > > > > > > > > > > > You can think about it this way: > > > > > > > > > > > > > > Airflow has already done its job with abstractions - versioning > > > > changes > > > > > > and > > > > > > > metadata DB is implemented in Airflow DB. If there are any > > missing > > > > pieces > > > > > > > in the abstraction that will be usable across multiple > > > > implementations of > > > > > > > versioning, we should - of course - add it to Airflow metadata > > DB - > > > > in > > > > > > the > > > > > > > way that they can be used by those different implementations. > > But the > > > > > > code > > > > > > > to manage and use it should be in airflow-core. > > > > > > > If there is anything specific for the implementation of S3 / > > Amazon > > > > > > > integration -> it should be implemented independently from > > Airflow > > > > > > Metadata > > > > > > > DB. There are many complexities in managing and upgrading core DB > > > > and we > > > > > > > should not use the db to make provider-specific things. The > > > > discussion > > > > > > > about shared code and isolation is interesting in this context. > > > > Because I > > > > > > > think we might get to the point when we go deeper and deeper in > > this > > > > > > > direction that we will have (and we already do it more or less) > > NO > > > > > > > (regular) providers needed with whatever CLI or tooling we will > > need > > > > to > > > > > > > manage the Metadata DB. FAB and Edge are currently exceptions - > > but > > > > they > > > > > > > are by no means "regular" providers. > > > > > > > > > > > > > > So I'd say - if while designing/ implementing S3 versioning you > > will > > > > see > > > > > > > that part of the implementation can be abstracted away and added > > to > > > > the > > > > > > > core and used by other implementations - 100% - let's add it to > > the > > > > core. > > > > > > > But only then. If it is something that only Amazon provider needs > > > > and S3 > > > > > > > needs - let's make it use Amazon **whatever** as backing storage. > > > > > > > > > > > > > > I would even say - talk to the Google team and try to come up > > with an > > > > > > > abstraction that can be used for versioning in both S3 and GCS, > > > > agree on > > > > > > > it, and let's see if this abstraction should find its way to the > > > > core. > > > > > > That > > > > > > > would be my proposal. > > > > > > > > > > > > > > J. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Jul 9, 2025 at 7:37 PM Oliveira, Niko > > > > > > <[email protected]> > > > > > > > wrote: > > > > > > > > > > > > > >> Thanks for engaging folks! > > > > > > >> > > > > > > >> I don’t love the idea of using another bucket. For one, this > > means > > > > > > Airflow > > > > > > >> needs write access to S3 which is not ideal; some > > users/customers > > > > are > > > > > > very > > > > > > >> sensitive about ever allowing write access to things. And two, > > you > > > > will > > > > > > >> commonly get issues with a design that leaks state into customer > > > > managed > > > > > > >> accounts/resources, they may delete the bucket not knowing what > > it > > > > is, > > > > > > they > > > > > > >> may not migrate it to a new account or region if they ever > > move. I > > > > think > > > > > > >> it’s best for the data to be stored transparently to the user > > and > > > > > > >> co-located with the data it strongly relates to (i.e. the dag > > runs > > > > that > > > > > > are > > > > > > >> associated with those bundle versions). > > > > > > >> > > > > > > >> Is using DB Manager completely unacceptable these days? What are > > > > folks' > > > > > > >> thoughts on that? > > > > > > >> > > > > > > >> Cheers, > > > > > > >> Niko > > > > > > >> > > > > > > >> ________________________________ > > > > > > >> From: Jarek Potiuk <[email protected]> > > > > > > >> Sent: Wednesday, July 9, 2025 6:23:54 AM > > > > > > >> To: [email protected] > > > > > > >> Subject: RE: [EXT] S3 Dag Bundle Versions and DB Manager > > > > > > >> > > > > > > >> CAUTION: This email originated from outside of the > > organization. Do > > > > not > > > > > > >> click links or open attachments unless you can confirm the > > sender > > > > and > > > > > > know > > > > > > >> the content is safe. > > > > > > >> > > > > > > >> > > > > > > >> > > > > > > >> AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur > > > > > > externe. > > > > > > >> Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si > > vous ne > > > > > > pouvez > > > > > > >> pas confirmer l’identité de l’expéditeur et si vous n’êtes pas > > > > certain > > > > > > que > > > > > > >> le contenu ne présente aucun risque. > > > > > > >> > > > > > > >> > > > > > > >> > > > > > > >>> Another option also would be Using dynamodb table? that also > > > > supports > > > > > > >> snapshots and i feel it works very well with state management. > > > > > > >> > > > > > > >> Yep that would also work. > > > > > > >> > > > > > > >> Anything "Amazon" to keep state would do. I think that it > > should be > > > > our > > > > > > >> "default" approach that if we have to keep state and the state > > is > > > > > > connected > > > > > > >> with specific "provider's" implementation, it's best to not keep > > > > state > > > > > > in > > > > > > >> Airflow, but in the "integration" that the provider works with > > if > > > > > > possible. > > > > > > >> We cannot do it in "generic" case because we do not know what > > > > > > >> "integrations" the user has - but since this is "provider's" > > > > > > functionality, > > > > > > >> using anything else that the given integration provides makes > > > > perfect > > > > > > >> sense. > > > > > > >> > > > > > > >> J. > > > > > > >> > > > > > > >> > > > > > > >> On Wed, Jul 9, 2025 at 3:12 PM Pavankumar Gopidesu < > > > > > > >> [email protected]> > > > > > > >> wrote: > > > > > > >> > > > > > > >>> Agree another s3 bucket also works here > > > > > > >>> > > > > > > >>> Another option also would be Using dynamodb table? that also > > > > supports > > > > > > >>> snapshots and i feel it works very well with state management. > > > > > > >>> > > > > > > >>> > > > > > > >>> Pavan > > > > > > >>> > > > > > > >>> On Wed, Jul 9, 2025 at 2:06 PM Jarek Potiuk <[email protected]> > > > > wrote: > > > > > > >>> > > > > > > >>>> One of the options would be to use a similar approach as > > terraform > > > > > > >> uses - > > > > > > >>>> i.e. use dedicated "metadata" state storage in a DIFFERENT s3 > > > > bucket > > > > > > >> than > > > > > > >>>> DAG files. Since we know there must be an S3 available > > > > (obviously) - > > > > > > it > > > > > > >>>> seems not too excessive to assume that there might be another > > > > bucket, > > > > > > >>>> independent of the DAG bucket where the state is stored - same > > > > bucket > > > > > > >>> (and > > > > > > >>>> dedicated connection id) could even be used to store state for > > > > > > multiple > > > > > > >>> S3 > > > > > > >>>> dag bundles - each Dag bundle could have a dedicated object > > > > storing > > > > > > the > > > > > > >>>> state. The metadata is not huge, so continuously reading and > > > > replacing > > > > > > >> it > > > > > > >>>> should not be an issue. > > > > > > >>>> > > > > > > >>>> What's nice about it - this single object could even > > > > **actually** > > > > > > use > > > > > > >> S3 > > > > > > >>>> versioning to keep historical state - to optimize things and > > > > keep a > > > > > > >> log > > > > > > >>> of > > > > > > >>>> changes potentially. > > > > > > >>>> > > > > > > >>>> J. > > > > > > >>>> > > > > > > >>>> On Wed, Jul 9, 2025 at 3:01 AM Oliveira, Niko > > > > > > >>> <[email protected] > > > > > > >>>> wrote: > > > > > > >>>> > > > > > > >>>>> Hey folks, > > > > > > >>>>> > > > > > > >>>>> tl;dr I’d like to get some thoughts on a proposal to use DB > > > > Manager > > > > > > >> for > > > > > > >>>> S3 > > > > > > >>>>> Dag Bundle versioning. > > > > > > >>>>> > > > > > > >>>>> The initial commit for S3 Dag Bundles was recently merged [1] > > > > but it > > > > > > >>>> lacks > > > > > > >>>>> Bundle versioning (since this isn’t trivial with something > > like > > > > S3, > > > > > > >>> like > > > > > > >>>> it > > > > > > >>>>> is with Git). The proposed solution involves building a > > snapshot > > > > of > > > > > > >> the > > > > > > >>>> S3 > > > > > > >>>>> bucket at the time each Bundle version is created, noting the > > > > version > > > > > > >>> of > > > > > > >>>>> all the objects in the bucket (using S3’s native bucket > > > > versioning > > > > > > >>>> feature) > > > > > > >>>>> and creating a manifest to store those versions and then > > giving > > > > that > > > > > > >>>> whole > > > > > > >>>>> manifest itself some unique id/version/uuid. These manifests > > now > > > > need > > > > > > >>> to > > > > > > >>>> be > > > > > > >>>>> stored somewhere for future use/retrieval. The proposal is to > > > > use the > > > > > > >>>>> Airflow database using the DB Manager feature. Other options > > > > include > > > > > > >>>> using > > > > > > >>>>> the local filesystem to store them (but this obviously wont > > work > > > > in > > > > > > >>>>> Airflow’s distributed architecture) or the S3 bucket itself > > (but > > > > this > > > > > > >>>>> requires write access to the bucket and we will always be at > > the > > > > > > >> mercy > > > > > > >>> of > > > > > > >>>>> the user accidentally deleting/modifying the manifests as > > they > > > > try to > > > > > > >>>>> manage the lifecycle of their bucket, they should not need > > to be > > > > > > >> aware > > > > > > >>> of > > > > > > >>>>> or need to account for this metadata). So the Airflow DB > > works > > > > nicely > > > > > > >>> as > > > > > > >>>> a > > > > > > >>>>> persistent and internally accessible location for this data. > > > > > > >>>>> > > > > > > >>>>> But I’m aware of the complexities of using the DB Manager > > and the > > > > > > >>>>> discussion we had during the last dev call about providers > > > > vending DB > > > > > > >>>>> tables (concerning migrations and ensuring smooth upgrades or > > > > > > >>> downgrades > > > > > > >>>> of > > > > > > >>>>> the schema). So I wanted to reach out to see what folks > > thought. > > > > I > > > > > > >> have > > > > > > >>>>> talked to Jed, the Bundle Master (tm), and we haven’t come up > > > > with > > > > > > >>>> anything > > > > > > >>>>> else that solves the problem as cleanly, so the DB Manager is > > > > still > > > > > > >> my > > > > > > >>>> top > > > > > > >>>>> choice. I think what we go with will pave the way for other > > > > Bundle > > > > > > >>>>> providers of a similar type as well, so it's worth thinking > > > > deeply > > > > > > >>> about > > > > > > >>>>> this decision. > > > > > > >>>>> > > > > > > >>>>> Let me know what you think and thanks for your time! > > > > > > >>>>> > > > > > > >>>>> Cheers, > > > > > > >>>>> Niko > > > > > > >>>>> > > > > > > >>>>> [1] https://github.com/apache/airflow/pull/46621 > > > > > > >>>>> > > > > > > > > > > > > > > --------------------------------------------------------------------- > > > > > > To unsubscribe, e-mail: [email protected] > > > > > > For additional commands, e-mail: [email protected] > > > > > > > > > > > > > > > > > > > > > > > > > --------------------------------------------------------------------- > > > > To unsubscribe, e-mail: [email protected] > > > > For additional commands, e-mail: [email protected] > > > > > > > > > > > > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: [email protected] > > For additional commands, e-mail: [email protected] > > > > > --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
