I've reviewed it again today and left some comments. Overall, it looks great and would be a great feature add in Airflow. I think you can start an official voting thread on this AIP.
On Sun, Jul 19, 2026 at 7:32 PM Piyush Maheshwari < [email protected]> wrote: > Hi everyone, > Thanks for the reviews on AIP-109 ( > > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-109+DAG+Version+Pinning > ). > Over the past few weeks, I implemented the feature in our fork (nearly > done). > Based on the insights and learnings from this process, I have further > refined the AIP while addressing all comments. > > Please review the updated version. I'll wait to collect some feedback and > then take it to a vote. > I can start contributing PRs as soon as the AIP is approved. > > Thanks and regards, > Piyush > > On Wed, Jun 3, 2026 at 2:01 PM Ephraim Anierobi <[email protected] > > > wrote: > > > Hi Piyush, > > > > Thanks for the AIP. I’ve added a few comments to the doc. Please take a > > look, as I think we should iron out those areas and fully understand the > > implications before moving forward. > > > > Regards > > - Ephraim > > > > On Tue, 26 May 2026 at 18:24, Przemysław Mirowski <[email protected]> > > wrote: > > > > > I checked the API - +1. Thanks for writing this up! > > > > > > Thanks Nathan for mentioning the #63884 PR. It is nice addition > (already > > > merged), and I think that it will be really useful for users who got > used > > > to Airflow 2 retry mechanism. I think also that your PR and API-109 are > > > complementary as one is focusing on the versioning behaviour for > retrying > > > task, the latter is focusing on versioning behaviour for new Dag Runs. > > > ________________________________ > > > From: Christos Bisias <[email protected]> > > > Sent: 25 May 2026 09:46 > > > To: [email protected] <[email protected]> > > > Subject: Re: [DISCUSS] DAG Version Pinning for Deployment Gating > > (Building > > > on AIP-63) > > > > > > Hello, > > > > > > I'm a little late on the discussion but I just came across the AIP and > I > > > like this idea. I've actually been thinking of working on something > > > similar, to allow people to handle a bad rollout by reverting to old > code > > > for that run without a full slow release pipeline. And this covers it. > > > > > > In my opinion, this seems more like a natural step towards what dag > > > versioning is supposed to do, than a new feature. > > > > > > Thank you, > > > Christos > > > > > > > > > On Mon, May 25, 2026 at 8:15 AM Piyush Maheshwari < > > > [email protected]> wrote: > > > > > > > Hi everyone, > > > > I wanted to send a gentle nudge to review AIP-109: DAG Version > Pinning > > ( > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-109+DAG+Version+Pinning > > > > ). > > > > If there are no major concerns, I would like to take this to a vote > > soon. > > > > > > > > Thanks, > > > > Piyush > > > > > > > > On Fri, May 15, 2026 at 7:44 AM Piyush Maheshwari < > > > > [email protected]> wrote: > > > > > > > > > Thanks for the note, Sumit. Based on the feedback, I've drafted an > > AIP > > > > > that is now up for review. > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-109+DAG+Version+Pinning > > > > > > > > > > Would like to get the community's feedback on the same. > > > > > > > > > > Nathan, I remember seeing your work (the issue and an older PR) > while > > > > > reviewing all ongoing work related to DAG versions. I agree with > the > > > PR's > > > > > intent, although I haven't reviewed it yet. > > > > > I understand your PR makes the version-pinned execution behavior of > > > > reruns > > > > > and backfills configurable. > > > > > However, this discussion revolves around the behavior for new runs > > > only. > > > > > We need the capability to pin a DAG to a specific version for > future > > > runs > > > > > instead. > > > > > Hope that clarifies. > > > > > > > > > > Regards, > > > > > Piyush > > > > > > > > > > On Thu, May 14, 2026 at 5:11 PM Nathan Hadfield < > > > > [email protected]> > > > > > wrote: > > > > > > > > > >> Hello, > > > > >> > > > > >> I saw AIP-109 that was created in relation to this discussion and > > > > thought > > > > >> I’d better mention this PR that I’ve been working on for a while > and > > > is > > > > >> close to being approved. > > > > >> > > > > >> https://github.com/apache/airflow/pull/63884 > > > > >> > > > > >> It is very much related to the motivations described and > implements > > > the > > > > >> desire for control over the behaviour when clearing/backfilling > > runs. > > > > >> > > > > >> Happy to discuss best steps for this here or on the PR. > > > > >> > > > > >> Cheers, > > > > >> > > > > >> Nathan > > > > >> > > > > >> From: Przemysław Mirowski <[email protected]> > > > > >> Date: Tuesday, 28 April 2026 at 21:54 > > > > >> To: [email protected] <[email protected]> > > > > >> Subject: Re: [DISCUSS] DAG Version Pinning for Deployment Gating > > > > >> (Building on AIP-63) > > > > >> > > > > >> This Message Is From an External Sender > > > > >> This message came from outside your organization. > > > > >> > > > > >> > > > > >> > P.S. In my opinion, what can be done in/around git, should be > done > > > > >> there. Recreation of CI/CD in any form inside of Airflow itself is > > > > >> something which should not be done. > > > > >> > I'm glad we agree on this :) I suppose we just disagree on what > is > > > > >> possible outside of Airflow :p > > > > >> > > > > >> I think that we just disagree on what the issue is, not on what is > > > > >> possible/should be outside of Airflow. > > > > >> > > > > >> > I think we are trying to duplicate what we already have in Git. > > > > >> > > > > >> Not really if we are only referring to version pinning. As far as > I > > am > > > > >> aware of how things are working, there is no possibility to > > determine > > > > that > > > > >> Dag after e.g. deployment on 1:00:00 PM will be exactly parsed and > > > used > > > > >> since 1:01:00 PM forward. Basically, what version pinning would > > > provide > > > > is > > > > >> the full control of the time since the given version will be used > > > > >> (currently we can only have more-or-less timing which in some > cases, > > > is > > > > not > > > > >> sufficient). The "quick revert" is the consequence of having above > > > > >> possibility. > > > > >> > > > > >> Looking at the general concerns, with having that feature or not, > > > users > > > > >> can pretty easily test things on production, but it just requires > > more > > > > time > > > > >> between iterations without it. IMHO it will not change the need > for > > > > >> Airflow-related platform teams which makes sure, by > > > standards/policies, > > > > >> that things are properly tested before production deployment. I > > think > > > > that > > > > >> assumption that some users will misuse this feature is true (like > > with > > > > most > > > > >> of the features really), but on the other hand it would provide > more > > > > >> control for other users. The other solution possibly would be to > > make > > > > Dag > > > > >> Processor work more on "events" instead of "simple" parsing loop > (I > > > > recall > > > > >> that there was some PR couple years ago with PoC of that, but I > > > couldn't > > > > >> quickly find it). > > > > >> > > > > >> ________________________________ > > > > >> From: Jarek Potiuk <[email protected]> > > > > >> Sent: 28 April 2026 17:02 > > > > >> To: [email protected] <[email protected]> > > > > >> Subject: Re: [DISCUSS] DAG Version Pinning for Deployment Gating > > > > >> (Building on AIP-63) > > > > >> > > > > >> Same concerns. I think we are trying to duplicate what we already > > have > > > > in > > > > >> Git—branches and reverts, for example—by moving what should be > > managed > > > > as > > > > >> part of the development process to Airflow UI. > > > > >> > > > > >> Almost everything you describe can be done with: > > > > >> > > > > >> * having a dev/staging system configured properly to use > dev/staging > > > > >> branches > > > > >> * Having a process of managing development and a proper branching > > > > strategy > > > > >> * single git command (for example, `git revert XXXX` followed by > > push > > > to > > > > >> the right branch) > > > > >> > > > > >> J. > > > > >> > > > > >> > > > > >> On Tue, Apr 28, 2026 at 10:34 AM Pierre Jeambrun < > > > [email protected] > > > > > > > > > >> wrote: > > > > >> > > > > >> > At first glance I tend to agree with Jens and Niko. > > > > >> > > > > > >> > I understand the request, but I agree that this resolves CI/CD > and > > > > >> testing > > > > >> > issues that should probably be remain outside Airflow. > > > > >> > > > > > >> > On Mon, Apr 27, 2026 at 7:43 PM Oliveira, Niko < > > [email protected] > > > > > > > > >> > wrote: > > > > >> > > > > > >> > > Hey folks! > > > > >> > > > > > > >> > > > P.S. In my opinion, what can be done in/around git, should > be > > > done > > > > >> > > there. Recreation of CI/CD in any form inside of Airflow > itself > > is > > > > >> > > something which should not be done. > > > > >> > > > > > > >> > > I'm glad we agree on this :) I suppose we just disagree on > what > > is > > > > >> > > possible outside of Airflow :p > > > > >> > > > > > > >> > > But at this point I will bow out of the conversation and let > > > others > > > > >> weigh > > > > >> > > in. I'm not fully convinced any of these requested behaviours > > > > require > > > > >> > > changes to Airflow (I think that's just masking some dev ops > > > work). > > > > >> But > > > > >> > > also I'm not completely opposed to the change either, I'm more > > on > > > > the > > > > >> > > fence, so if others love the feature by all means implement > it! > > :) > > > > >> > > > > > > >> > > Cheers, > > > > >> > > Niko > > > > >> > > ________________________________ > > > > >> > > From: Przemysław Mirowski <[email protected]> > > > > >> > > Sent: Thursday, April 23, 2026 3:06 PM > > > > >> > > To: [email protected] <[email protected]> > > > > >> > > Subject: RE: [EXT] [DISCUSS] DAG Version Pinning for > Deployment > > > > Gating > > > > >> > > (Building on AIP-63) > > > > >> > > > > > > >> > > CAUTION: This email originated from outside of the > organization. > > > Do > > > > >> not > > > > >> > > click links or open attachments unless you can confirm the > > sender > > > > and > > > > >> > know > > > > >> > > the content is safe. > > > > >> > > > > > > >> > > > > > > >> > > > > > > >> > > AVERTISSEMENT: Ce courrier électronique provient d’un > expéditeur > > > > >> externe. > > > > >> > > Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si > > vous > > > ne > > > > >> > pouvez > > > > >> > > pas confirmer l’identité de l’expéditeur et si vous n’êtes pas > > > > certain > > > > >> > que > > > > >> > > le contenu ne présente aucun risque. > > > > >> > > > > > > >> > > > > > > >> > > > > > > >> > > Hi, > > > > >> > > > > > > >> > > I think that CI/CD and version pining are a little two > different > > > > >> things > > > > >> > > here. In a use cases with some critical systems involved, the > > > > >> situation > > > > >> > > when the Dag changes the version to the latest without > > possibility > > > > to > > > > >> > > determine when it will exactly happen (CI/CD will have some > > > > >> more-or-less > > > > >> > > time to deploy the change, the same goes for Dag Processor > > parsing > > > > >> time) > > > > >> > is > > > > >> > > rather hard to do and in some systems it can make change > > > deployment > > > > >> > harder > > > > >> > > and less safe. Of course, the ideal solution would be to have > > > proper > > > > >> > > non-prod environment, which is fully representative in > > comparison > > > to > > > > >> > > production (in some cases exposing non-prod to prod > > > > data/traffic/etc. > > > > >> is, > > > > >> > > just, not an option - e.g. security), but it is not always > > > possible > > > > >> to do > > > > >> > > due to various reasons like costs, licenses, space and/or > > vendors. > > > > I'm > > > > >> > > agreeing especially with point 5 of Piyush latest message. > > Having > > > > >> above > > > > >> > in > > > > >> > > mind, I think that version pinning would be a nice addition to > > the > > > > Dag > > > > >> > > Versioning feature with an assumption that it is for critical > > > > Airflow > > > > >> > Dags > > > > >> > > when full control of the Dags version change time is required > > > (maybe > > > > >> > there > > > > >> > > is also another way to achieve that). > > > > >> > > > > > > >> > > P.S. In my opinion, what can be done in/around git, should be > > done > > > > >> there. > > > > >> > > Recreation of CI/CD in any form inside of Airflow itself is > > > > something > > > > >> > which > > > > >> > > should not be done. > > > > >> > > ________________________________ > > > > >> > > From: Oliveira, Niko <[email protected]> > > > > >> > > Sent: 23 April 2026 01:50 > > > > >> > > To: [email protected] <[email protected]> > > > > >> > > Subject: Re: [DISCUSS] DAG Version Pinning for Deployment > Gating > > > > >> > (Building > > > > >> > > on AIP-63) > > > > >> > > > > > > >> > > Hey Piyush, > > > > >> > > > > > > >> > > Thanks for your reply, I do love how clearly it is written > and I > > > see > > > > >> > > exactly the problem you're trying to solve! > > > > >> > > > > > > >> > > I'm still just not convinced this needs to be done in Airflow, > > at > > > > >> least > > > > >> > > not with a first class feature. As interesting as I think your > > > > >> > microservice > > > > >> > > analogy is, Airflow is not a microservice component, it is a > > > (very, > > > > >> very) > > > > >> > > fancy cron scheduler. And I'm not sure the complexity is worth > > the > > > > use > > > > >> > > case. Since any new code added to Airflow must be maintained > by > > > this > > > > >> > > community and we must be cautious that any new pieces serves > > > enough > > > > >> use > > > > >> > > cases/users to make it worth it. > > > > >> > > To me this should either be managed outside of an individual > > > Airflow > > > > >> > > environment e.g. you have an entirely separate > staging/gamma/dev > > > > >> Airflow > > > > >> > > environment, which is exposed to some level of production > > traffic > > > > (to > > > > >> > > borrow your microservice analogy) until it can graduate to the > > > > >> production > > > > >> > > environment. And if you really need on the fly toggling of a > > > > version, > > > > >> as > > > > >> > > you say, Airflow does this quite responsively, if you deploy a > > new > > > > >> > version > > > > >> > > of your dags it will parse and start using that new version > > > > >> immediately > > > > >> > > (the problem you're trying to solve can be a benefit here). > You > > > can > > > > >> even > > > > >> > > have multiple versions of your dags deployed at once and use > > > > >> > configuration > > > > >> > > to control which dag directory Airflow reads from (or > > move/symlink > > > > >> Dags > > > > >> > in > > > > >> > > and out of the Dags directory as needed from a known good or > > > pinned > > > > >> > > source). Or use variables or some other parameter store to > > control > > > > >> other > > > > >> > > pieces of runtime behaviour inside the Dags themselves. > Between > > > > CI/CD, > > > > >> > dev > > > > >> > > ops and making use of existing Airflow primitives I think you > > can > > > > >> achieve > > > > >> > > what you're looking for. > > > > >> > > > > > > >> > > But as always, this is open and community based software, so > I'm > > > > >> happy to > > > > >> > > disagree and commit if the rest of the community thinks this > is > > a > > > > >> > valuable > > > > >> > > feature :) > > > > >> > > > > > > >> > > Cheers, > > > > >> > > Niko > > > > >> > > ________________________________ > > > > >> > > From: Piyush Maheshwari <[email protected]> > > > > >> > > Sent: Tuesday, April 21, 2026 10:46 PM > > > > >> > > To: [email protected] <[email protected]> > > > > >> > > Subject: RE: [EXT] [DISCUSS] DAG Version Pinning for > Deployment > > > > Gating > > > > >> > > (Building on AIP-63) > > > > >> > > > > > > >> > > CAUTION: This email originated from outside of the > organization. > > > Do > > > > >> not > > > > >> > > click links or open attachments unless you can confirm the > > sender > > > > and > > > > >> > know > > > > >> > > the content is safe. > > > > >> > > > > > > >> > > > > > > >> > > > > > > >> > > AVERTISSEMENT: Ce courrier électronique provient d’un > expéditeur > > > > >> externe. > > > > >> > > Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si > > vous > > > ne > > > > >> > pouvez > > > > >> > > pas confirmer l’identité de l’expéditeur et si vous n’êtes pas > > > > certain > > > > >> > que > > > > >> > > le contenu ne présente aucun risque. > > > > >> > > > > > > >> > > > > > > >> > > > > > > >> > > Hi Ephraim, Jarek, Jens, and Niko, > > > > >> > > > > > > >> > > Thank you for the candid feedback. I want to clarify a few > > things, > > > > as > > > > >> I > > > > >> > > completely agree with Jens and Niko that "testing in > production" > > > is > > > > an > > > > >> > > anti-pattern. That is absolutely not the intention here. > > > > >> > > > > > > >> > > 1. I view this as bringing standard microservice-like > deployment > > > > >> maturity > > > > >> > > to DAGs. > > > > >> > > Before service deployments in our org, code is tested locally, > > in > > > a > > > > >> dev > > > > >> > > environment, and via strict unit/e2e integration tests before > it > > > > ever > > > > >> > makes > > > > >> > > it to main. But even after merging and passing those CI > > pipelines, > > > > we > > > > >> > still > > > > >> > > use load tests, pre-prod soak times, shadow traffic, and gated > > > > >> production > > > > >> > > rollouts with automated rollback triggers. Having deployment > > gates > > > > for > > > > >> > the > > > > >> > > production environment doesn't mean the pre-merge checks > weren't > > > > >> strict > > > > >> > or > > > > >> > > that the change wasn't tested beforehand -- it just allows us > to > > > > place > > > > >> > > additional safety gates for the code to take effect, exactly > > like > > > in > > > > >> the > > > > >> > > service world. > > > > >> > > > > > > >> > > 2. The core issue we are trying to solve is that Airflow > > currently > > > > >> > > inseparably links Code Distribution (a file arriving on the > > > > >> dag-processor > > > > >> > > and being parsed) with Release Activation (the scheduler > > executing > > > > >> that > > > > >> > > code). > > > > >> > > To extend the microservices analogy, I can think of the DAG > > > > processor > > > > >> > > parsing all files as "building the artifact(s)," while the > > > scheduler > > > > >> and > > > > >> > > executor acting on the DAG versions created thereafter as > > > > "deploying" > > > > >> or > > > > >> > > running the changed code. > > > > >> > > We simply want to decouple the build from the deployment. This > > > does > > > > >> not > > > > >> > > mean that the code arriving on the dag-processor will be > tested > > > for > > > > >> the > > > > >> > > first time straight in production. It should've already > passed a > > > set > > > > >> of > > > > >> > > checks in the CI pipeline. > > > > >> > > > > > > >> > > 3. It is also worth calling out that Airflow already supports > > this > > > > >> > > decoupled behavior at the run level for task re-runs and > > > > mid-execution > > > > >> > DAG > > > > >> > > version bumps (by pinning the version for the rest of the > > > execution > > > > or > > > > >> > the > > > > >> > > rerun). We are simply trying to expose this existing > capability > > at > > > > the > > > > >> > DAG > > > > >> > > level so users can govern which version new scheduled runs are > > > > created > > > > >> > > with. > > > > >> > > > > > > >> > > 4. I also agree that Airflow itself should not be aware of our > > > CI/CD > > > > >> > > pipeline, nor would it manage the deployment orchestration or > > > > testing. > > > > >> > > For our requirements, I just need Airflow to expose APIs to > > deploy > > > > >> (pin) > > > > >> > a > > > > >> > > DAG version, and to remove the pin (to restore/enable the > > default > > > > >> > > "auto-deploy latest" behavior). > > > > >> > > Beyond that, we intend to use an external release orchestrator > > > that > > > > >> can > > > > >> > > explicitly tell Airflow when a parsed version is actually > > allowed > > > to > > > > >> run. > > > > >> > > Until that API call is made, the previously pinned version > > remains > > > > >> > active. > > > > >> > > This ensures we don't introduce assumptions or awareness of > the > > > > >> presence > > > > >> > of > > > > >> > > any external gating mechanisms to Airflow. > > > > >> > > Also note that the intention is to keep the default > auto-deploy > > > > >> behavior > > > > >> > > unless a user (or a system on their behalf) explicitly asks > > > Airflow > > > > to > > > > >> > pin > > > > >> > > a DAG to a specific version. > > > > >> > > > > > > >> > > 5. Most importantly, this feature provides an incident > response > > > > >> > "rollback" > > > > >> > > behavior. If a bad DAG version slips through CI/CD into > > > production, > > > > >> > either > > > > >> > > an on-call engineer or a rollback-trigger (airflow-external) > can > > > > >> > instantly > > > > >> > > roll back to the previous pinned version via the API/UI to > > > mitigate. > > > > >> > > Without this, users have to revert the code in Git and wait > for > > > the > > > > >> > entire > > > > >> > > CI/CD pipeline and file-sync process to run, which is often > too > > > slow > > > > >> > during > > > > >> > > an outage. > > > > >> > > > > > > >> > > 6. Jarek - You are right, database schema changes can be > > discussed > > > > >> later. > > > > >> > > My intention was only to share a very brief summary of how I > > > deemed > > > > >> it to > > > > >> > > be technically feasible for early feedback. I did briefly > share > > > the > > > > >> > > high-level use cases ("Safe Deployment Gating" and "Instant > > > > >> Rollbacks") > > > > >> > in > > > > >> > > the original mail, but I completely agree that aligning on the > > UX > > > > >> first > > > > >> > > would be a good next step. > > > > >> > > > > > > >> > > If there are no major remaining concerns after this response, > I > > > can > > > > >> draft > > > > >> > > and share an AIP to detail the UX, followed by a high-level > > > > proposal, > > > > >> > > caveats and next steps. > > > > >> > > > > > > >> > > Thanks for your time. > > > > >> > > Regards, > > > > >> > > Piyush > > > > >> > > > > > > >> > > On Tue, Apr 21, 2026 at 5:59 PM Oliveira, Niko < > > > [email protected] > > > > > > > > > >> > > wrote: > > > > >> > > > > > > >> > > > I am with Jens on this one. I think we're complicating > Airflow > > > to > > > > >> get > > > > >> > > > around a bad practice. If stability of your Dags is critical > > and > > > > >> they > > > > >> > are > > > > >> > > > highly versioned then I think as Jens suggested running them > > > > >> through a > > > > >> > > > pipeline that first deploys them to a dev or gamma > environment > > > > which > > > > >> > > > verifies that quality of the Dags is what you expect. If > > > something > > > > >> > slips > > > > >> > > > through, then it's just normal software practices of either > > > > >> reverting > > > > >> > and > > > > >> > > > rolling back or rolling forward with a fix pushed through > the > > > > >> > pipeline. I > > > > >> > > > don't think Airflow should be aware of that process or > > > opinionated > > > > >> > about > > > > >> > > it. > > > > >> > > > > > > > >> > > > Cheers, > > > > >> > > > Niko > > > > >> > > > ------------------------------ > > > > >> > > > *From:* Jens Scheffler <[email protected]> > > > > >> > > > *Sent:* Monday, April 20, 2026 11:17 AM > > > > >> > > > *To:* [email protected] <[email protected]> > > > > >> > > > *Subject:* RE: [EXT] [DISCUSS] DAG Version Pinning for > > > Deployment > > > > >> > Gating > > > > >> > > > (Building on AIP-63) > > > > >> > > > > > > > >> > > > CAUTION: This email originated from outside of the > > organization. > > > > Do > > > > >> not > > > > >> > > > click links or open attachments unless you can confirm the > > > sender > > > > >> and > > > > >> > > know > > > > >> > > > the content is safe. > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > > >> > > > AVERTISSEMENT: Ce courrier électronique provient d’un > > expéditeur > > > > >> > externe. > > > > >> > > > Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si > > > vous > > > > ne > > > > >> > > pouvez > > > > >> > > > pas confirmer l’identité de l’expéditeur et si vous n’êtes > pas > > > > >> certain > > > > >> > > que > > > > >> > > > le contenu ne présente aucun risque. > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > > >> > > > Hi, > > > > >> > > > > > > > >> > > > I am still quite sceptical. Yes, if such pinning is made, > then > > > per > > > > >> Dag > > > > >> > a > > > > >> > > > change need to be possible via UI and API. But I still see > it > > as > > > > >> > > > checken-and-egg - so you want to run a pinned version but > then > > > how > > > > >> do > > > > >> > > > you test the changes (w/o moving a version pin)? Then again > > some > > > > >> test > > > > >> > > > mode is needed or per run you need to make a "test run" with > > > > another > > > > >> > > > version. Smells a bit like mis-using a production system for > > > > >> testing. > > > > >> > > > > > > > >> > > > On the other hand, yes if all Dags share the same Git repo > > then > > > > >> merging > > > > >> > > > a branch to some other will switch all Dags at the same > time. > > > > Still > > > > >> you > > > > >> > > > could utilize standard Git tools and cherry-pick individual > > > > changes > > > > >> and > > > > >> > > > no force to always make a full rollout. At least 80% > possible > > > with > > > > >> > > > standard CI/CD tools and Git. > > > > >> > > > > > > > >> > > > TLDR I see the danger that instead of a proper CI/CD and > test > > > > system > > > > >> > > > such a feature might feel like you can easily test on a > > > production > > > > >> > > > system. Effectively it would be needed allowing to start a > Dag > > > > with > > > > >> any > > > > >> > > > version to also be able to jump back as a reversion. Even > > > though, > > > > >> yes, > > > > >> > > > agree, all is technically possible. > > > > >> > > > > > > > >> > > > Jens > > > > >> > > > > > > > >> > > > On 20.04.26 16:40, Jarek Potiuk wrote: > > > > >> > > > > +1 to what Ephraim wrote. I think that was a natural next > > step > > > > we > > > > >> > > > > discussed, but it needs significant refinement, starting > > with > > > > the > > > > >> > > actual > > > > >> > > > > use cases it should serve and the UX for user > interaction. I > > > > think > > > > >> > > > related > > > > >> > > > > database changes are pretty secondary. Use cases cover > runs, > > > > >> re-runs, > > > > >> > > > > backfills, CI testing, rollbacks, etc. Following the > > > > >> "documentation > > > > >> > > > first" > > > > >> > > > > approach discussed in separate thread, describing the > > context > > > > and > > > > >> > > > intention > > > > >> > > > > of what we want to achieve is much more important than DB > > > schema > > > > >> > > changes. > > > > >> > > > > Once we know which use cases we want to serve, the DB > schema > > > > >> changes > > > > >> > > and > > > > >> > > > > other related items will emerge naturally. > > > > >> > > > > > > > > >> > > > > On Mon, Apr 20, 2026 at 3:15 PM Ephraim Anierobi < > > > > >> > > > [email protected]> > > > > >> > > > > wrote: > > > > >> > > > > > > > > >> > > > >> Hi Piyush, thanks for starting this discussion. > > > > >> > > > >> > > > > >> > > > >> I like the proposal. We can introduce an active execution > > > > version > > > > >> > for > > > > >> > > > >> "versioned bundles" and make scheduler/API resolve > through > > > it. > > > > >> The > > > > >> > > hard > > > > >> > > > >> part of this is making airflow able to distinguish the > > latest > > > > >> parsed > > > > >> > > > >> dagmodel's metadata from active scheduling metadata. I > will > > > > >> suggest > > > > >> > > you > > > > >> > > > >> draft this in a google docs and share for further > > > discussions. > > > > >> > > > >> > > > > >> > > > >> Regards > > > > >> > > > >> - Ephraim > > > > >> > > > >> > > > > >> > > > >> On Mon, 20 Apr 2026 at 01:31, Piyush Maheshwari < > > > > >> > > [email protected]> > > > > >> > > > >> wrote: > > > > >> > > > >> > > > > >> > > > >>> Thanks for sharing your thoughts Jens. > > > > >> > > > >>> > > > > >> > > > >>>> be able to test it? … a Q&A/Testing environment to be > > able > > > to > > > > >> > > sign-off > > > > >> > > > >>> changes. > > > > >> > > > >>> Yes, we’ve have built an isolated airflow environment to > > run > > > > >> > > regression > > > > >> > > > >>> checks before promoting to production. > > > > >> > > > >>> > > > > >> > > > >>> As you suggested, we’re already running both generic and > > > > >> DAG-custom > > > > >> > > > >> static > > > > >> > > > >>> checks in a CI job as a required step to merge to the > main > > > > >> branch. > > > > >> > > > >>> > > > > >> > > > >>>> But then the "main" branch might be best suited if > > > > >> > > > >>> implemented on the test system > > > > >> > > > >>> In this case, problematic commits on “main” can choke > > other > > > > >> > unrelated > > > > >> > > > >>> changes. > > > > >> > > > >>> So the other option would be to revert the problematic > > > commits > > > > >> and > > > > >> > > > deploy > > > > >> > > > >>> forward. > > > > >> > > > >>> > > > > >> > > > >>> However, a key limitation with this approach that > remains > > is > > > > >> that a > > > > >> > > > >> commit > > > > >> > > > >>> affecting multiple DAGs goes live for either all DAGs or > > > none. > > > > >> > > > >>> > > > > >> > > > >>> Second important feature we get with this is instant > > > DAG-level > > > > >> > > rollback > > > > >> > > > >>> without waiting for a revert commit to merge and be > picked > > > by > > > > >> > > airflow. > > > > >> > > > >>> > > > > >> > > > >>> I think DAG-level version pinning can also unlock a lot > of > > > > >> > > flexibility > > > > >> > > > >> for > > > > >> > > > >>> deployments including tiered rollouts, auto-rollback > > > triggers, > > > > >> > timed > > > > >> > > > >>> deployment windows and so on. > > > > >> > > > >>> > > > > >> > > > >>> Looking forward to hear your thoughts. > > > > >> > > > >>> Regards, > > > > >> > > > >>> Piyush > > > > >> > > > >>> > > > > >> > > > >>> On Sun, 19 Apr 2026 at 3:12 PM, Jens Scheffler < > > > > >> > [email protected]> > > > > >> > > > >>> wrote: > > > > >> > > > >>> > > > > >> > > > >>>> Thanks Piyush for dropping the discussion! > > > > >> > > > >>>> > > > > >> > > > >>>> I think in general QA processes are important and a > valid > > > use > > > > >> > case. > > > > >> > > So > > > > >> > > > >> a > > > > >> > > > >>>> kind of pinning Dag versions really is important. > > > > >> > > > >>>> > > > > >> > > > >>>> Thinking about it, if you pin the version ... how would > > you > > > > >> then > > > > >> > be > > > > >> > > > >> able > > > > >> > > > >>>> to test it? I assume you would need (and should have or > > > > invest > > > > >> > > into) a > > > > >> > > > >>>> Q&A/Testing environment to be able to sign-off changes. > > > Both > > > > in > > > > >> > > > >>>> infrastructure but also for Dag changes. > > > > >> > > > >>>> > > > > >> > > > >>>> If you are changing Dags first of all static checks on > > Dag > > > > code > > > > >> > are > > > > >> > > > >> very > > > > >> > > > >>>> much proposed as well as you can have tests implemented > > and > > > > >> test > > > > >> > > your > > > > >> > > > >>>> Dags and logic. Similar like software a CI/CD system > will > > > be > > > > a > > > > >> > good > > > > >> > > > >>>> setup. Alongside Dag changes also have logical changes > > that > > > > >> mostly > > > > >> > > can > > > > >> > > > >>>> only be tested in a live system and not as static > checks. > > > > >> > > > >>>> > > > > >> > > > >>>> Have you considered using Git and a set of branches for > > > > >> > implementing > > > > >> > > > >>>> such staging? E.g. you have a git repo and you plan to > > make > > > > >> > changes. > > > > >> > > > >>>> Then you would open a PR for the change and merge it to > > the > > > > >> "main" > > > > >> > > > >>>> branch - and there in your CI/CD you can check all > sorts > > of > > > > >> static > > > > >> > > > >>>> checks and tests. But then the "main" branch might be > > best > > > > >> suited > > > > >> > if > > > > >> > > > >>>> implemented on the test system. Once you validate the > > > changes > > > > >> > > > >> end-to-end > > > > >> > > > >>>> you could make another PR for example to a "prod" > branch. > > > And > > > > >> if > > > > >> > > your > > > > >> > > > >>>> production system is only pulling Dags from the "prod" > > > branch > > > > >> then > > > > >> > > you > > > > >> > > > >>>> can have this merging strategy as a staging setup. > > > > >> > > > >>>> > > > > >> > > > >>>> Would this resolve your PING problem? Or which other > > detail > > > > in > > > > >> the > > > > >> > > use > > > > >> > > > >>>> case would require a PIN on top of a staging strategy? > > > > >> > > > >>>> > > > > >> > > > >>>> Jens > > > > >> > > > >>>> > > > > >> > > > >>>> P.S.: Have enabled your confluence account after it was > > > > >> created in > > > > >> > > > >> order > > > > >> > > > >>>> to write to Confluence, sorry, typical pitfall after > > > account > > > > >> > > creation > > > > >> > > > >>>> permissions were not set. Now it should work. Let me > know > > > if > > > > >> not. > > > > >> > > > >>>> > > > > >> > > > >>>> On 19.04.26 01:40, Piyush Maheshwari wrote: > > > > >> > > > >>>>> Hi everyone, > > > > >> > > > >>>>> I'm a new contributor to Airflow. I'd like to propose > a > > > new > > > > >> > feature > > > > >> > > > >> for > > > > >> > > > >>>> Airflow: DAG Version Pinning. > > > > >> > > > >>>>> Building on the foundation introduced by AIP-63: DAG > > > > >> Versioning ( > > > > >> > > > >> > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > > > > > > > https://urldefense.com/v3/__https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-63*3A*DAG*Versioning__;JSsr!!Ci6f514n9QsL8ck!l3ZKTOw996h9qu4NR0VT4ouUryUdk_HmXUAVPbwCHwPwn0N2CCptVdx95-V0BoRFjws9huE_1Vy-THL8jw$ > > > > >> > > > >>> ), > > > > >> > > > >>>> this proposal aims to extend Airflow's capabilities to > > > > support > > > > >> > true > > > > >> > > > >>>> continuous deployment (CD) gating and safer release > > cycles. > > > > >> > > > >>>>> The Problem & Use Cases > > > > >> > > > >>>>> Currently, the scheduler always creates DagRuns using > > the > > > > >> latest > > > > >> > > > >> parsed > > > > >> > > > >>>> DagVersion. This means that the updated DAG code is > > > deployed > > > > >> > (takes > > > > >> > > > >>> effect) > > > > >> > > > >>>> right after the dag-processor processes it. While this > is > > > > great > > > > >> > for > > > > >> > > > >> rapid > > > > >> > > > >>>> development, teams running business-critical pipelines > > > often > > > > >> need > > > > >> > > > >>> stricter > > > > >> > > > >>>> deployment mechanisms. Specifically: > > > > >> > > > >>>>> * > > > > >> > > > >>>>> Safe Deployment Gating: The ability to pin a DAG to > its > > > last > > > > >> > known > > > > >> > > > >>>> stable version while new code is parsed in the > > background. > > > > This > > > > >> > > allows > > > > >> > > > >>> the > > > > >> > > > >>>> new version to be held back until it passes automated > > > > >> regression > > > > >> > > tests > > > > >> > > > >> or > > > > >> > > > >>>> receives explicit manual approval. > > > > >> > > > >>>>> * > > > > >> > > > >>>>> Instant Rollbacks: If an issue is detected in a newly > > > > promoted > > > > >> > DAG > > > > >> > > > >>>> version, users need the capability to instantly roll > back > > > to > > > > a > > > > >> > > > previous > > > > >> > > > >>>> version via the UI/API, without having to revert the > > > > underlying > > > > >> > code > > > > >> > > > >> and > > > > >> > > > >>>> wait for the repository sync and DAG processing cycle. > > > > >> > > > >>>>> High-Level Proposed Solution > > > > >> > > > >>>>> Introduce an optional active_dag_version_id to the > > > DagModel. > > > > >> This > > > > >> > > > >> field > > > > >> > > > >>>> can be used to pin a DAG version for scheduling and > > > > execution, > > > > >> > while > > > > >> > > > >> the > > > > >> > > > >>>> dag-processor can continue to parse and register newer > > DAG > > > > >> > versions. > > > > >> > > > >>>>> * > > > > >> > > > >>>>> When this pin is set, the scheduler and API will > respect > > > the > > > > >> > pinned > > > > >> > > > >>>> version for creating runs and executing tasks, > separating > > > the > > > > >> > > parsing > > > > >> > > > >> of > > > > >> > > > >>>> new code from the execution of new code. > > > > >> > > > >>>>> * > > > > >> > > > >>>>> If the pin is NULL, the system defaults to the current > > > > >> behavior > > > > >> > > > >> (always > > > > >> > > > >>>> executing the latest parsed version). This way, we can > > > > maintain > > > > >> > > > >> complete > > > > >> > > > >>>> backwards compatibility. > > > > >> > > > >>>>> I have put together some detailed notes covering the > > data > > > > >> model > > > > >> > > > >>> changes, > > > > >> > > > >>>> database migrations, and edge cases with this approach. > > If > > > > >> there > > > > >> > is > > > > >> > > > >>> general > > > > >> > > > >>>> alignment that this fits the vision for Airflow, I > would > > > like > > > > >> to > > > > >> > > take > > > > >> > > > >>> this > > > > >> > > > >>>> proposal through the formal AIP review process. > > > > >> > > > >>>>> But I would love to get the community's feedback on > the > > > > >> feature > > > > >> > and > > > > >> > > > >> the > > > > >> > > > >>>> high-level approach. > > > > >> > > > >>>>> I'll also need someone to grant me access to create > > > content > > > > on > > > > >> > the > > > > >> > > > >>>> Airflow Confluence wiki. > > > > >> > > > >>>>> Thanks for your time! > > > > >> > > > >>>>> Regards, > > > > >> > > > >>>>> Piyush > > > > >> > > > >>>>> > > > > >> > > > >>>> > > > > >> > > > > > > --------------------------------------------------------------------- > > > > >> > > > >>>> To unsubscribe, e-mail: > > [email protected] > > > > >> > > > >>>> For additional commands, e-mail: > > > [email protected] > > > > >> > > > >>>> > > > > >> > > > >>>> > > > > >> > > > > > > > >> > > > > > > > >> > > --------------------------------------------------------------------- > > > > >> > > > To unsubscribe, e-mail: [email protected] > > > > >> > > > For additional commands, e-mail: > [email protected] > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > > > > > > > >
