Hi Piyush, Thank you for this summary. It is accurate, and it matches where we landed. Thank you also for the many rounds of changes. The document is much clearer now than when we started.
I want to state my position plainly for the thread. My main concern was that a pin on a DagVersion could not hold, because the row could change under it and the bundle pointer could move. The pin guard fixes both. With that, plus the retention rule, the stale DAG change, the purity scope, and the divergence warning, I no longer object to the pin target. The AIP now says what it delivers and what it does not. That was my real goal. One thing I want to be clear about. My review was about whether the design holds, not about whether the feature is worth building. Those are two different questions, and the second one belongs to the community. The feature adds pin awareness in several core places: the version write path, db clean, the scheduler's eligibility query for import errors and stale DAGs, DagModel and asset field resolution in the DAG processor, and a synchronous update when a pin changes. Each change is small on its own. Together they mean these paths must consider pins from now on. What it gives back is per-DAG control over when a version takes effect, for pure DAGs that come from versioned bundles. That is real and useful for the teams who need it. Whether it is worth the added complexity in core paths is a judgment call, and voters should make it with both sides stated. I am not voting against it. I only do not want my resolved concerns to be read as a view on that question. I have left my remaining comments on the document itself. They are small and none of them changes the design, but I think they should be settled before the vote. On the vote, I agree with starting it again instead of continuing the old thread. The document has changed a lot. A fresh call, with a short note on what changed, would be fair to everyone who already voted. Thanks again for your patience through all of this. The proposal is in much better shape. Regards, Ephraim On Wed, 5 Aug 2026 at 10:55, Piyush Maheshwari <[email protected]> wrote: > Hi everyone, > > I'd like to thank Ephraim for the depth of this review. Following the > previous email, we had a series of detailed follow-up discussions, and I > believe we have converged. This email summarizes where we landed, answers > the concerns from the previous email for the benefit of the thread, and > describes the updates I have now made to the AIP. I would like the > community's review of the updated AIP ( > > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-109+DAG+Version+Pinning > ) > > == Precise scope of the proposal == > > The discussion sharpened what the AIP promises, and it now states this > explicitly: > > - The goal is deterministic, per-DAG control over when a new version of a > DAG takes effect, plus an instant, API-driven rollback to a prior version, > without affecting other DAGs sharing the same bundle. > > - A pin fixes the serialized DAG the scheduler enforces (task set, edges, > schedule, display fields) and the bundle (name and version) the worker > checks out. It provides graph and code-level determinism, not > behavior-level determinism, since a task that reads live external state at > execution time still does so under a pin. > > - Pinning guarantees are stated for DAGs that are "pure", meaning a given > bundle (name and version) always produces the same serialized DAG. The AIP > now defines this purity condition up front. > > - For impure DAGs, different pinning guarantees are now documented for > impure DAGs. A divergence between the pinned serialized DAG's snapshot and > the worker's fresh parse for execution-time fields is silent. The AIP now > explicitly documents this as "silent divergence," a limitation, and adds > detection for it. > > == Answers to the five concerns from the previous email == > > 1. "The latest DagVersion is mutable, so a pin does not freeze content." > Addressed by introducing a pin guard in the DAG Processor's > version-recording path. A pinned DagVersion is treated as referenced, so > its serialized content is never overwritten in place; a content change > creates a new DagVersion instead (the same path that already runs when task > instances reference the row). Note that the DAG processor only mutates the > latest DagVersion row, so a pin on an older version is already immutable > today; the guard changes behavior only when the pinned version is also the > latest. > > 2. "A pin would not pin the source code, because bundle_version is > refreshed in place on hash-identical parses." > Addressed by the same guard, which stops the in-place > bundle_version/version_data refresh from applying to a pinned row. When the > bundle advances without a serialized-DAG change while the pinned version is > the latest, the processor instead creates one new DagVersion carrying the > new bundle_version, which becomes the unpinned latest; subsequent no-change > bundle advances keep updating that new latest in place exactly as today. > This preserves the intent of the in-place refresh (the DAG's latest version > keeps tracking the bundle) while the pinned version's bundle reference > stays frozen. Since new DagRuns will also carry the pinned bundle, and > workers resolve their bundle checkout from the DagRun, the scheduler's view > and the worker's code stay on the same version. Note that this version > creation, without a hash change, occurs only per pin advance, not once per > commit. > > 3. "Version identity is often accidental (impure DAGs)." > Addressed by scoping (guarantees stated for pure DAGs, with the purity > condition defined in the AIP) and by divergence detection, where a > DagWarning is raised when a pinned DAG's fresh parse hash no longer matches > the pinned DagVersion's hash, naming the pinned version. The check is a > no-op for pure DAGs. > > 4. "Cleanup can delete a pinned version and silently unpin via ON DELETE > SET NULL." > Addressed: airflow db clean will retain any DagVersion referenced by an > active pin, the same way it retains task-instance-referenced versions. > > 5. "The deletion rule works against the main use case." > Revised: a dag_id that disappears from the latest parse would still be > marked stale on DagModel as today, but the scheduler's eligibility query > will admit a stale DAG while it has an active pin, so it keeps running on > its pinned version (parallel to the import-error exception already in the > AIP). Actual removal requires an explicit unpin first, after which the > normal stale-to-deleted path applies unchanged. This turns the defect > scenario from a silent removal into an explicit act. > > == On pinning the bundle instead of the DagVersion == > > We discussed this alternative at length, in two variants, and the AIP's > "Alternatives" section now covers both: > > - A global bundle pin (keep a history of bundle versions and freeze the > whole bundle on one) is simple and atomic, but it pins every DAG in the > bundle at once. It cannot keep a subset of DAGs on an older version while > the rest advance, and per-DAG control is the primary use case, since a > breaking change in a shared bundle typically affects only some of the DAGs > it emits. We see it as a potentially complementary, coarser-grained control > rather than a replacement. > > - A per-DAG pin keyed by the bundle (name and version) is not uniquely > resolvable to a serialized DAG for impure DAGs, since the pair can map to > several dag_version rows. It therefore cannot be expressed as a foreign key > to a specific version, and the latest matching row can resolve differently > across invocations, which is behavior drift while supposedly pinned. It is > also unstable over time, because a dag_version's bundle_version is > refreshed in place on hash-identical parses, so a previously valid bundle > version can stop resolving exactly when a rollback needs it. Freezing that > would require an immutable bundle-to-dag_version mapping, which amounts to > pinning the dag_version too and converges towards this proposal. > > == Operating models and rollback availability == > > Ephraim raised an important point about rollback availability. An on-demand > pin can only roll back to versions Airflow already recorded, so a change > that never altered the serialized DAG (e.g. one confined to an imported > helper module) has no distinct version to return to unless it was pinned at > the time. The AIP now states this honestly under a new "Usage Models and > Rollback Availability" section, which describes two models the primitive > serves: > > 1. On-demand pinning (incident response, planned cutovers): rollback > targets are the recorded DAG versions. Changes that altered the serialized > DAG (and had task instances) are reachable; changes that did not are a > stated limitation. > 2. Standing pins driven by an external release orchestrator: every DAG > always carries a pin and the orchestrator advances it per deployment. Every > pinned state remains an immutable rollback target, closing the gap above, > at the cost of operating the orchestrator. The consequences of the same are > also covered in the AIP. > > Airflow itself only ships the pin/unpin primitive; neither model is > mandatory to use this feature. > > == Updated AIP and proposed timeline == > > I have updated AIP-109 with all of the above: > > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-109+DAG+Version+Pinning > > I would appreciate the community's review of the updated proposal. If > multiple maintainers review it and if no major open questions or comments > remain by Wednesday, August 12, I plan to restart the vote phase. > > Thanks again to Ephraim for the multiple rounds of thorough review that got > the proposal to this shape. > > Regards, > Piyush > > On Tue, Jul 28, 2026 at 3:47 PM Ephraim Anierobi < > [email protected]> > wrote: > > > Thank you for this proposal, and for the careful work behind it. I took a > > second, deeper look at the document, and I want to share my concerns, > > because I believe they change the direction rather than the details. > > > > First, I want to recognize the real problem the AIP identifies. The parse > > loop controls when new code becomes active, and that timing is not > > deterministic. Teams need a way to control activation time, and they > need a > > fast rollback path. This is a genuine gap, and I am glad the AIP puts a > > spotlight on it. > > > > After the second review, however, I believe the proposal rests on a > > misunderstanding of what a DagVersion is. A DagVersion is a historical > > record of what Airflow observed after a parse. It is an output of the > > system, not an input to it. Dag versioning exists so that running and > past > > task instances keep a faithful record of the code they ran against. > *Bundle > > versioning* is the mechanism that decides which source code a run uses. > The > > AIP reads DagVersion as a release artifact that users can select and > > activate, and the current implementation does not support that reading: > > > > 1. *The latest DagVersion is mutable*. When no task instance references > it, > > write_dag() replaces its serialized content, hash, and Dag code in place, > > on the same version ID. A pin does not create a task instance reference. > So > > a user could pin the current version before a risky merge, and the next > > parse could replace the content behind that pin. The pin would not give > the > > determinism it promises. > > > > 2. *A pin on the current DagVersion would not pin the source code*. When > a > > new commit does not change the serialized content, `write_dag()` updates > > bundle_version on the latest DagVersion in place. A change to a helper > > module or an SQL file does not change the serialized hash, so workers > would > > run the new commit under the pinned version. > > > > 3. *Version identity is often accidental.* Parse-time variability and > > serializer defects can create versions that no one intended. Airflow > ships > > an inflation checker and a dedicated Dag warning type because this > problem > > is real and common. Dag factories at large scale, the AIP's own key > > example, are the most exposed. A pin would often select an accidental > > snapshot rather than a deliberate release. > > > > 4. *The proposal does not address cleanup*. Today, `airflow db clean` > keeps > > the latest version per Dag and skips only versions that task instances > > reference. An older pinned version with no task instances is eligible for > > deletion, and the proposed ON DELETE SET NULL behavior would then > silently > > return the Dag to the latest version, exactly when the pin matters most. > > The document does not cover this interaction. > > > > 5. *The deletion rule works against the main use case*. The AIP states > that > > removal from a new bundle version deletes the Dag, even when it is > pinned. > > A defect in a dag-factory file that changes or drops Dag IDs removes Dags > > from the parse results. In that case, Airflow would delete the Dag and > > discard the pin, so the protection would be absent in one of the most > > common failure modes the AIP is motivated by. > > > > These are not implementation gaps that more safeguards can close. They > all > > follow from the same source: the design asks a history table to act as a > > deployment mechanism. Making that work would mean redefining what > > DagVersion is, and I do not think we should. > > > > The goal itself is worth pursuing at the right layer. Deterministic > cutover > > and instant rollback are operations on the *source artifact*. Bundle > > versions already represent the source, and pinning or rolling back at the > > bundle layer would be atomic for all Dags and all support files at once. > I > > would be glad to help explore that direction. > > > > Thanks again for raising this discussion. The problem is real, and I hope > > we can solve it together on a foundation that supports it. > > > > - Ephraim > > > > On Mon, 27 Jul 2026 at 06:53, Piyush Maheshwari < > > [email protected]> > > wrote: > > > > > Thank you, Sumit. > > > > > > I've started a voting thread as suggested: > > > https://lists.apache.org/thread/bff758c5p68v1mtzhflytn7yzv0d8l1t > > > > > > Regards, > > > Piyush > > > > > > On Fri, Jul 24, 2026 at 12:08 PM Sumit Maheshwari < > > [email protected]> > > > wrote: > > > > > > > I've reviewed it again today and left some comments. Overall, it > looks > > > > great and would be a great feature add in Airflow. I think you can > > start > > > an > > > > official voting thread on this AIP. > > > > > > > > > > > > On Sun, Jul 19, 2026 at 7:32 PM Piyush Maheshwari < > > > > [email protected]> wrote: > > > > > > > > > Hi everyone, > > > > > Thanks for the reviews on AIP-109 ( > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-109+DAG+Version+Pinning > > > > > ). > > > > > Over the past few weeks, I implemented the feature in our fork > > (nearly > > > > > done). > > > > > Based on the insights and learnings from this process, I have > further > > > > > refined the AIP while addressing all comments. > > > > > > > > > > Please review the updated version. I'll wait to collect some > feedback > > > and > > > > > then take it to a vote. > > > > > I can start contributing PRs as soon as the AIP is approved. > > > > > > > > > > Thanks and regards, > > > > > Piyush > > > > > > > > > > On Wed, Jun 3, 2026 at 2:01 PM Ephraim Anierobi < > > > > [email protected] > > > > > > > > > > > wrote: > > > > > > > > > > > Hi Piyush, > > > > > > > > > > > > Thanks for the AIP. I’ve added a few comments to the doc. Please > > > take a > > > > > > look, as I think we should iron out those areas and fully > > understand > > > > the > > > > > > implications before moving forward. > > > > > > > > > > > > Regards > > > > > > - Ephraim > > > > > > > > > > > > On Tue, 26 May 2026 at 18:24, Przemysław Mirowski < > > [email protected] > > > > > > > > > > wrote: > > > > > > > > > > > > > I checked the API - +1. Thanks for writing this up! > > > > > > > > > > > > > > Thanks Nathan for mentioning the #63884 PR. It is nice addition > > > > > (already > > > > > > > merged), and I think that it will be really useful for users > who > > > got > > > > > used > > > > > > > to Airflow 2 retry mechanism. I think also that your PR and > > API-109 > > > > are > > > > > > > complementary as one is focusing on the versioning behaviour > for > > > > > retrying > > > > > > > task, the latter is focusing on versioning behaviour for new > Dag > > > > Runs. > > > > > > > ________________________________ > > > > > > > From: Christos Bisias <[email protected]> > > > > > > > Sent: 25 May 2026 09:46 > > > > > > > To: [email protected] <[email protected]> > > > > > > > Subject: Re: [DISCUSS] DAG Version Pinning for Deployment > Gating > > > > > > (Building > > > > > > > on AIP-63) > > > > > > > > > > > > > > Hello, > > > > > > > > > > > > > > I'm a little late on the discussion but I just came across the > > AIP > > > > and > > > > > I > > > > > > > like this idea. I've actually been thinking of working on > > something > > > > > > > similar, to allow people to handle a bad rollout by reverting > to > > > old > > > > > code > > > > > > > for that run without a full slow release pipeline. And this > > covers > > > > it. > > > > > > > > > > > > > > In my opinion, this seems more like a natural step towards what > > dag > > > > > > > versioning is supposed to do, than a new feature. > > > > > > > > > > > > > > Thank you, > > > > > > > Christos > > > > > > > > > > > > > > > > > > > > > On Mon, May 25, 2026 at 8:15 AM Piyush Maheshwari < > > > > > > > [email protected]> wrote: > > > > > > > > > > > > > > > Hi everyone, > > > > > > > > I wanted to send a gentle nudge to review AIP-109: DAG > Version > > > > > Pinning > > > > > > ( > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-109+DAG+Version+Pinning > > > > > > > > ). > > > > > > > > If there are no major concerns, I would like to take this to > a > > > vote > > > > > > soon. > > > > > > > > > > > > > > > > Thanks, > > > > > > > > Piyush > > > > > > > > > > > > > > > > On Fri, May 15, 2026 at 7:44 AM Piyush Maheshwari < > > > > > > > > [email protected]> wrote: > > > > > > > > > > > > > > > > > Thanks for the note, Sumit. Based on the feedback, I've > > drafted > > > > an > > > > > > AIP > > > > > > > > > that is now up for review. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-109+DAG+Version+Pinning > > > > > > > > > > > > > > > > > > Would like to get the community's feedback on the same. > > > > > > > > > > > > > > > > > > Nathan, I remember seeing your work (the issue and an older > > PR) > > > > > while > > > > > > > > > reviewing all ongoing work related to DAG versions. I agree > > > with > > > > > the > > > > > > > PR's > > > > > > > > > intent, although I haven't reviewed it yet. > > > > > > > > > I understand your PR makes the version-pinned execution > > > behavior > > > > of > > > > > > > > reruns > > > > > > > > > and backfills configurable. > > > > > > > > > However, this discussion revolves around the behavior for > new > > > > runs > > > > > > > only. > > > > > > > > > We need the capability to pin a DAG to a specific version > for > > > > > future > > > > > > > runs > > > > > > > > > instead. > > > > > > > > > Hope that clarifies. > > > > > > > > > > > > > > > > > > Regards, > > > > > > > > > Piyush > > > > > > > > > > > > > > > > > > On Thu, May 14, 2026 at 5:11 PM Nathan Hadfield < > > > > > > > > [email protected]> > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > >> Hello, > > > > > > > > >> > > > > > > > > >> I saw AIP-109 that was created in relation to this > > discussion > > > > and > > > > > > > > thought > > > > > > > > >> I’d better mention this PR that I’ve been working on for a > > > while > > > > > and > > > > > > > is > > > > > > > > >> close to being approved. > > > > > > > > >> > > > > > > > > >> https://github.com/apache/airflow/pull/63884 > > > > > > > > >> > > > > > > > > >> It is very much related to the motivations described and > > > > > implements > > > > > > > the > > > > > > > > >> desire for control over the behaviour when > > > clearing/backfilling > > > > > > runs. > > > > > > > > >> > > > > > > > > >> Happy to discuss best steps for this here or on the PR. > > > > > > > > >> > > > > > > > > >> Cheers, > > > > > > > > >> > > > > > > > > >> Nathan > > > > > > > > >> > > > > > > > > >> From: Przemysław Mirowski <[email protected]> > > > > > > > > >> Date: Tuesday, 28 April 2026 at 21:54 > > > > > > > > >> To: [email protected] <[email protected]> > > > > > > > > >> Subject: Re: [DISCUSS] DAG Version Pinning for Deployment > > > Gating > > > > > > > > >> (Building on AIP-63) > > > > > > > > >> > > > > > > > > >> This Message Is From an External Sender > > > > > > > > >> This message came from outside your organization. > > > > > > > > >> > > > > > > > > >> > > > > > > > > >> > P.S. In my opinion, what can be done in/around git, > should > > > be > > > > > done > > > > > > > > >> there. Recreation of CI/CD in any form inside of Airflow > > > itself > > > > is > > > > > > > > >> something which should not be done. > > > > > > > > >> > I'm glad we agree on this :) I suppose we just disagree > on > > > > what > > > > > is > > > > > > > > >> possible outside of Airflow :p > > > > > > > > >> > > > > > > > > >> I think that we just disagree on what the issue is, not on > > > what > > > > is > > > > > > > > >> possible/should be outside of Airflow. > > > > > > > > >> > > > > > > > > >> > I think we are trying to duplicate what we already have > in > > > > Git. > > > > > > > > >> > > > > > > > > >> Not really if we are only referring to version pinning. As > > far > > > > as > > > > > I > > > > > > am > > > > > > > > >> aware of how things are working, there is no possibility > to > > > > > > determine > > > > > > > > that > > > > > > > > >> Dag after e.g. deployment on 1:00:00 PM will be exactly > > parsed > > > > and > > > > > > > used > > > > > > > > >> since 1:01:00 PM forward. Basically, what version pinning > > > would > > > > > > > provide > > > > > > > > is > > > > > > > > >> the full control of the time since the given version will > be > > > > used > > > > > > > > >> (currently we can only have more-or-less timing which in > > some > > > > > cases, > > > > > > > is > > > > > > > > not > > > > > > > > >> sufficient). The "quick revert" is the consequence of > having > > > > above > > > > > > > > >> possibility. > > > > > > > > >> > > > > > > > > >> Looking at the general concerns, with having that feature > or > > > > not, > > > > > > > users > > > > > > > > >> can pretty easily test things on production, but it just > > > > requires > > > > > > more > > > > > > > > time > > > > > > > > >> between iterations without it. IMHO it will not change the > > > need > > > > > for > > > > > > > > >> Airflow-related platform teams which makes sure, by > > > > > > > standards/policies, > > > > > > > > >> that things are properly tested before production > > deployment. > > > I > > > > > > think > > > > > > > > that > > > > > > > > >> assumption that some users will misuse this feature is > true > > > > (like > > > > > > with > > > > > > > > most > > > > > > > > >> of the features really), but on the other hand it would > > > provide > > > > > more > > > > > > > > >> control for other users. The other solution possibly would > > be > > > to > > > > > > make > > > > > > > > Dag > > > > > > > > >> Processor work more on "events" instead of "simple" > parsing > > > loop > > > > > (I > > > > > > > > recall > > > > > > > > >> that there was some PR couple years ago with PoC of that, > > but > > > I > > > > > > > couldn't > > > > > > > > >> quickly find it). > > > > > > > > >> > > > > > > > > >> ________________________________ > > > > > > > > >> From: Jarek Potiuk <[email protected]> > > > > > > > > >> Sent: 28 April 2026 17:02 > > > > > > > > >> To: [email protected] <[email protected]> > > > > > > > > >> Subject: Re: [DISCUSS] DAG Version Pinning for Deployment > > > Gating > > > > > > > > >> (Building on AIP-63) > > > > > > > > >> > > > > > > > > >> Same concerns. I think we are trying to duplicate what we > > > > already > > > > > > have > > > > > > > > in > > > > > > > > >> Git—branches and reverts, for example—by moving what > should > > be > > > > > > managed > > > > > > > > as > > > > > > > > >> part of the development process to Airflow UI. > > > > > > > > >> > > > > > > > > >> Almost everything you describe can be done with: > > > > > > > > >> > > > > > > > > >> * having a dev/staging system configured properly to use > > > > > dev/staging > > > > > > > > >> branches > > > > > > > > >> * Having a process of managing development and a proper > > > > branching > > > > > > > > strategy > > > > > > > > >> * single git command (for example, `git revert XXXX` > > followed > > > by > > > > > > push > > > > > > > to > > > > > > > > >> the right branch) > > > > > > > > >> > > > > > > > > >> J. > > > > > > > > >> > > > > > > > > >> > > > > > > > > >> On Tue, Apr 28, 2026 at 10:34 AM Pierre Jeambrun < > > > > > > > [email protected] > > > > > > > > > > > > > > > > > >> wrote: > > > > > > > > >> > > > > > > > > >> > At first glance I tend to agree with Jens and Niko. > > > > > > > > >> > > > > > > > > > >> > I understand the request, but I agree that this resolves > > > CI/CD > > > > > and > > > > > > > > >> testing > > > > > > > > >> > issues that should probably be remain outside Airflow. > > > > > > > > >> > > > > > > > > > >> > On Mon, Apr 27, 2026 at 7:43 PM Oliveira, Niko < > > > > > > [email protected] > > > > > > > > > > > > > > > > >> > wrote: > > > > > > > > >> > > > > > > > > > >> > > Hey folks! > > > > > > > > >> > > > > > > > > > > >> > > > P.S. In my opinion, what can be done in/around git, > > > should > > > > > be > > > > > > > done > > > > > > > > >> > > there. Recreation of CI/CD in any form inside of > Airflow > > > > > itself > > > > > > is > > > > > > > > >> > > something which should not be done. > > > > > > > > >> > > > > > > > > > > >> > > I'm glad we agree on this :) I suppose we just > disagree > > on > > > > > what > > > > > > is > > > > > > > > >> > > possible outside of Airflow :p > > > > > > > > >> > > > > > > > > > > >> > > But at this point I will bow out of the conversation > and > > > let > > > > > > > others > > > > > > > > >> weigh > > > > > > > > >> > > in. I'm not fully convinced any of these requested > > > > behaviours > > > > > > > > require > > > > > > > > >> > > changes to Airflow (I think that's just masking some > dev > > > ops > > > > > > > work). > > > > > > > > >> But > > > > > > > > >> > > also I'm not completely opposed to the change either, > > I'm > > > > more > > > > > > on > > > > > > > > the > > > > > > > > >> > > fence, so if others love the feature by all means > > > implement > > > > > it! > > > > > > :) > > > > > > > > >> > > > > > > > > > > >> > > Cheers, > > > > > > > > >> > > Niko > > > > > > > > >> > > ________________________________ > > > > > > > > >> > > From: Przemysław Mirowski <[email protected]> > > > > > > > > >> > > Sent: Thursday, April 23, 2026 3:06 PM > > > > > > > > >> > > To: [email protected] <[email protected]> > > > > > > > > >> > > Subject: RE: [EXT] [DISCUSS] DAG Version Pinning for > > > > > Deployment > > > > > > > > Gating > > > > > > > > >> > > (Building on AIP-63) > > > > > > > > >> > > > > > > > > > > >> > > CAUTION: This email originated from outside of the > > > > > organization. > > > > > > > Do > > > > > > > > >> not > > > > > > > > >> > > click links or open attachments unless you can confirm > > the > > > > > > sender > > > > > > > > and > > > > > > > > >> > know > > > > > > > > >> > > the content is safe. > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > > >> > > AVERTISSEMENT: Ce courrier électronique provient d’un > > > > > expéditeur > > > > > > > > >> externe. > > > > > > > > >> > > Ne cliquez sur aucun lien et n’ouvrez aucune pièce > > jointe > > > si > > > > > > vous > > > > > > > ne > > > > > > > > >> > pouvez > > > > > > > > >> > > pas confirmer l’identité de l’expéditeur et si vous > > n’êtes > > > > pas > > > > > > > > certain > > > > > > > > >> > que > > > > > > > > >> > > le contenu ne présente aucun risque. > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > > >> > > Hi, > > > > > > > > >> > > > > > > > > > > >> > > I think that CI/CD and version pining are a little two > > > > > different > > > > > > > > >> things > > > > > > > > >> > > here. In a use cases with some critical systems > > involved, > > > > the > > > > > > > > >> situation > > > > > > > > >> > > when the Dag changes the version to the latest without > > > > > > possibility > > > > > > > > to > > > > > > > > >> > > determine when it will exactly happen (CI/CD will have > > > some > > > > > > > > >> more-or-less > > > > > > > > >> > > time to deploy the change, the same goes for Dag > > Processor > > > > > > parsing > > > > > > > > >> time) > > > > > > > > >> > is > > > > > > > > >> > > rather hard to do and in some systems it can make > change > > > > > > > deployment > > > > > > > > >> > harder > > > > > > > > >> > > and less safe. Of course, the ideal solution would be > to > > > > have > > > > > > > proper > > > > > > > > >> > > non-prod environment, which is fully representative in > > > > > > comparison > > > > > > > to > > > > > > > > >> > > production (in some cases exposing non-prod to prod > > > > > > > > data/traffic/etc. > > > > > > > > >> is, > > > > > > > > >> > > just, not an option - e.g. security), but it is not > > always > > > > > > > possible > > > > > > > > >> to do > > > > > > > > >> > > due to various reasons like costs, licenses, space > > and/or > > > > > > vendors. > > > > > > > > I'm > > > > > > > > >> > > agreeing especially with point 5 of Piyush latest > > message. > > > > > > Having > > > > > > > > >> above > > > > > > > > >> > in > > > > > > > > >> > > mind, I think that version pinning would be a nice > > > addition > > > > to > > > > > > the > > > > > > > > Dag > > > > > > > > >> > > Versioning feature with an assumption that it is for > > > > critical > > > > > > > > Airflow > > > > > > > > >> > Dags > > > > > > > > >> > > when full control of the Dags version change time is > > > > required > > > > > > > (maybe > > > > > > > > >> > there > > > > > > > > >> > > is also another way to achieve that). > > > > > > > > >> > > > > > > > > > > >> > > P.S. In my opinion, what can be done in/around git, > > should > > > > be > > > > > > done > > > > > > > > >> there. > > > > > > > > >> > > Recreation of CI/CD in any form inside of Airflow > itself > > > is > > > > > > > > something > > > > > > > > >> > which > > > > > > > > >> > > should not be done. > > > > > > > > >> > > ________________________________ > > > > > > > > >> > > From: Oliveira, Niko <[email protected]> > > > > > > > > >> > > Sent: 23 April 2026 01:50 > > > > > > > > >> > > To: [email protected] <[email protected]> > > > > > > > > >> > > Subject: Re: [DISCUSS] DAG Version Pinning for > > Deployment > > > > > Gating > > > > > > > > >> > (Building > > > > > > > > >> > > on AIP-63) > > > > > > > > >> > > > > > > > > > > >> > > Hey Piyush, > > > > > > > > >> > > > > > > > > > > >> > > Thanks for your reply, I do love how clearly it is > > written > > > > > and I > > > > > > > see > > > > > > > > >> > > exactly the problem you're trying to solve! > > > > > > > > >> > > > > > > > > > > >> > > I'm still just not convinced this needs to be done in > > > > Airflow, > > > > > > at > > > > > > > > >> least > > > > > > > > >> > > not with a first class feature. As interesting as I > > think > > > > your > > > > > > > > >> > microservice > > > > > > > > >> > > analogy is, Airflow is not a microservice component, > it > > > is a > > > > > > > (very, > > > > > > > > >> very) > > > > > > > > >> > > fancy cron scheduler. And I'm not sure the complexity > is > > > > worth > > > > > > the > > > > > > > > use > > > > > > > > >> > > case. Since any new code added to Airflow must be > > > maintained > > > > > by > > > > > > > this > > > > > > > > >> > > community and we must be cautious that any new pieces > > > serves > > > > > > > enough > > > > > > > > >> use > > > > > > > > >> > > cases/users to make it worth it. > > > > > > > > >> > > To me this should either be managed outside of an > > > individual > > > > > > > Airflow > > > > > > > > >> > > environment e.g. you have an entirely separate > > > > > staging/gamma/dev > > > > > > > > >> Airflow > > > > > > > > >> > > environment, which is exposed to some level of > > production > > > > > > traffic > > > > > > > > (to > > > > > > > > >> > > borrow your microservice analogy) until it can > graduate > > to > > > > the > > > > > > > > >> production > > > > > > > > >> > > environment. And if you really need on the fly > toggling > > > of a > > > > > > > > version, > > > > > > > > >> as > > > > > > > > >> > > you say, Airflow does this quite responsively, if you > > > > deploy a > > > > > > new > > > > > > > > >> > version > > > > > > > > >> > > of your dags it will parse and start using that new > > > version > > > > > > > > >> immediately > > > > > > > > >> > > (the problem you're trying to solve can be a benefit > > > here). > > > > > You > > > > > > > can > > > > > > > > >> even > > > > > > > > >> > > have multiple versions of your dags deployed at once > and > > > use > > > > > > > > >> > configuration > > > > > > > > >> > > to control which dag directory Airflow reads from (or > > > > > > move/symlink > > > > > > > > >> Dags > > > > > > > > >> > in > > > > > > > > >> > > and out of the Dags directory as needed from a known > > good > > > or > > > > > > > pinned > > > > > > > > >> > > source). Or use variables or some other parameter > store > > to > > > > > > control > > > > > > > > >> other > > > > > > > > >> > > pieces of runtime behaviour inside the Dags > themselves. > > > > > Between > > > > > > > > CI/CD, > > > > > > > > >> > dev > > > > > > > > >> > > ops and making use of existing Airflow primitives I > > think > > > > you > > > > > > can > > > > > > > > >> achieve > > > > > > > > >> > > what you're looking for. > > > > > > > > >> > > > > > > > > > > >> > > But as always, this is open and community based > > software, > > > so > > > > > I'm > > > > > > > > >> happy to > > > > > > > > >> > > disagree and commit if the rest of the community > thinks > > > this > > > > > is > > > > > > a > > > > > > > > >> > valuable > > > > > > > > >> > > feature :) > > > > > > > > >> > > > > > > > > > > >> > > Cheers, > > > > > > > > >> > > Niko > > > > > > > > >> > > ________________________________ > > > > > > > > >> > > From: Piyush Maheshwari <[email protected]> > > > > > > > > >> > > Sent: Tuesday, April 21, 2026 10:46 PM > > > > > > > > >> > > To: [email protected] <[email protected]> > > > > > > > > >> > > Subject: RE: [EXT] [DISCUSS] DAG Version Pinning for > > > > > Deployment > > > > > > > > Gating > > > > > > > > >> > > (Building on AIP-63) > > > > > > > > >> > > > > > > > > > > >> > > CAUTION: This email originated from outside of the > > > > > organization. > > > > > > > Do > > > > > > > > >> not > > > > > > > > >> > > click links or open attachments unless you can confirm > > the > > > > > > sender > > > > > > > > and > > > > > > > > >> > know > > > > > > > > >> > > the content is safe. > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > > >> > > AVERTISSEMENT: Ce courrier électronique provient d’un > > > > > expéditeur > > > > > > > > >> externe. > > > > > > > > >> > > Ne cliquez sur aucun lien et n’ouvrez aucune pièce > > jointe > > > si > > > > > > vous > > > > > > > ne > > > > > > > > >> > pouvez > > > > > > > > >> > > pas confirmer l’identité de l’expéditeur et si vous > > n’êtes > > > > pas > > > > > > > > certain > > > > > > > > >> > que > > > > > > > > >> > > le contenu ne présente aucun risque. > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > > >> > > Hi Ephraim, Jarek, Jens, and Niko, > > > > > > > > >> > > > > > > > > > > >> > > Thank you for the candid feedback. I want to clarify a > > few > > > > > > things, > > > > > > > > as > > > > > > > > >> I > > > > > > > > >> > > completely agree with Jens and Niko that "testing in > > > > > production" > > > > > > > is > > > > > > > > an > > > > > > > > >> > > anti-pattern. That is absolutely not the intention > here. > > > > > > > > >> > > > > > > > > > > >> > > 1. I view this as bringing standard microservice-like > > > > > deployment > > > > > > > > >> maturity > > > > > > > > >> > > to DAGs. > > > > > > > > >> > > Before service deployments in our org, code is tested > > > > locally, > > > > > > in > > > > > > > a > > > > > > > > >> dev > > > > > > > > >> > > environment, and via strict unit/e2e integration tests > > > > before > > > > > it > > > > > > > > ever > > > > > > > > >> > makes > > > > > > > > >> > > it to main. But even after merging and passing those > CI > > > > > > pipelines, > > > > > > > > we > > > > > > > > >> > still > > > > > > > > >> > > use load tests, pre-prod soak times, shadow traffic, > and > > > > gated > > > > > > > > >> production > > > > > > > > >> > > rollouts with automated rollback triggers. Having > > > deployment > > > > > > gates > > > > > > > > for > > > > > > > > >> > the > > > > > > > > >> > > production environment doesn't mean the pre-merge > checks > > > > > weren't > > > > > > > > >> strict > > > > > > > > >> > or > > > > > > > > >> > > that the change wasn't tested beforehand -- it just > > allows > > > > us > > > > > to > > > > > > > > place > > > > > > > > >> > > additional safety gates for the code to take effect, > > > exactly > > > > > > like > > > > > > > in > > > > > > > > >> the > > > > > > > > >> > > service world. > > > > > > > > >> > > > > > > > > > > >> > > 2. The core issue we are trying to solve is that > Airflow > > > > > > currently > > > > > > > > >> > > inseparably links Code Distribution (a file arriving > on > > > the > > > > > > > > >> dag-processor > > > > > > > > >> > > and being parsed) with Release Activation (the > scheduler > > > > > > executing > > > > > > > > >> that > > > > > > > > >> > > code). > > > > > > > > >> > > To extend the microservices analogy, I can think of > the > > > DAG > > > > > > > > processor > > > > > > > > >> > > parsing all files as "building the artifact(s)," while > > the > > > > > > > scheduler > > > > > > > > >> and > > > > > > > > >> > > executor acting on the DAG versions created thereafter > > as > > > > > > > > "deploying" > > > > > > > > >> or > > > > > > > > >> > > running the changed code. > > > > > > > > >> > > We simply want to decouple the build from the > > deployment. > > > > This > > > > > > > does > > > > > > > > >> not > > > > > > > > >> > > mean that the code arriving on the dag-processor will > be > > > > > tested > > > > > > > for > > > > > > > > >> the > > > > > > > > >> > > first time straight in production. It should've > already > > > > > passed a > > > > > > > set > > > > > > > > >> of > > > > > > > > >> > > checks in the CI pipeline. > > > > > > > > >> > > > > > > > > > > >> > > 3. It is also worth calling out that Airflow already > > > > supports > > > > > > this > > > > > > > > >> > > decoupled behavior at the run level for task re-runs > and > > > > > > > > mid-execution > > > > > > > > >> > DAG > > > > > > > > >> > > version bumps (by pinning the version for the rest of > > the > > > > > > > execution > > > > > > > > or > > > > > > > > >> > the > > > > > > > > >> > > rerun). We are simply trying to expose this existing > > > > > capability > > > > > > at > > > > > > > > the > > > > > > > > >> > DAG > > > > > > > > >> > > level so users can govern which version new scheduled > > runs > > > > are > > > > > > > > created > > > > > > > > >> > > with. > > > > > > > > >> > > > > > > > > > > >> > > 4. I also agree that Airflow itself should not be > aware > > of > > > > our > > > > > > > CI/CD > > > > > > > > >> > > pipeline, nor would it manage the deployment > > orchestration > > > > or > > > > > > > > testing. > > > > > > > > >> > > For our requirements, I just need Airflow to expose > APIs > > > to > > > > > > deploy > > > > > > > > >> (pin) > > > > > > > > >> > a > > > > > > > > >> > > DAG version, and to remove the pin (to restore/enable > > the > > > > > > default > > > > > > > > >> > > "auto-deploy latest" behavior). > > > > > > > > >> > > Beyond that, we intend to use an external release > > > > orchestrator > > > > > > > that > > > > > > > > >> can > > > > > > > > >> > > explicitly tell Airflow when a parsed version is > > actually > > > > > > allowed > > > > > > > to > > > > > > > > >> run. > > > > > > > > >> > > Until that API call is made, the previously pinned > > version > > > > > > remains > > > > > > > > >> > active. > > > > > > > > >> > > This ensures we don't introduce assumptions or > awareness > > > of > > > > > the > > > > > > > > >> presence > > > > > > > > >> > of > > > > > > > > >> > > any external gating mechanisms to Airflow. > > > > > > > > >> > > Also note that the intention is to keep the default > > > > > auto-deploy > > > > > > > > >> behavior > > > > > > > > >> > > unless a user (or a system on their behalf) explicitly > > > asks > > > > > > > Airflow > > > > > > > > to > > > > > > > > >> > pin > > > > > > > > >> > > a DAG to a specific version. > > > > > > > > >> > > > > > > > > > > >> > > 5. Most importantly, this feature provides an incident > > > > > response > > > > > > > > >> > "rollback" > > > > > > > > >> > > behavior. If a bad DAG version slips through CI/CD > into > > > > > > > production, > > > > > > > > >> > either > > > > > > > > >> > > an on-call engineer or a rollback-trigger > > > (airflow-external) > > > > > can > > > > > > > > >> > instantly > > > > > > > > >> > > roll back to the previous pinned version via the > API/UI > > to > > > > > > > mitigate. > > > > > > > > >> > > Without this, users have to revert the code in Git and > > > wait > > > > > for > > > > > > > the > > > > > > > > >> > entire > > > > > > > > >> > > CI/CD pipeline and file-sync process to run, which is > > > often > > > > > too > > > > > > > slow > > > > > > > > >> > during > > > > > > > > >> > > an outage. > > > > > > > > >> > > > > > > > > > > >> > > 6. Jarek - You are right, database schema changes can > be > > > > > > discussed > > > > > > > > >> later. > > > > > > > > >> > > My intention was only to share a very brief summary of > > > how I > > > > > > > deemed > > > > > > > > >> it to > > > > > > > > >> > > be technically feasible for early feedback. I did > > briefly > > > > > share > > > > > > > the > > > > > > > > >> > > high-level use cases ("Safe Deployment Gating" and > > > "Instant > > > > > > > > >> Rollbacks") > > > > > > > > >> > in > > > > > > > > >> > > the original mail, but I completely agree that > aligning > > on > > > > the > > > > > > UX > > > > > > > > >> first > > > > > > > > >> > > would be a good next step. > > > > > > > > >> > > > > > > > > > > >> > > If there are no major remaining concerns after this > > > > response, > > > > > I > > > > > > > can > > > > > > > > >> draft > > > > > > > > >> > > and share an AIP to detail the UX, followed by a > > > high-level > > > > > > > > proposal, > > > > > > > > >> > > caveats and next steps. > > > > > > > > >> > > > > > > > > > > >> > > Thanks for your time. > > > > > > > > >> > > Regards, > > > > > > > > >> > > Piyush > > > > > > > > >> > > > > > > > > > > >> > > On Tue, Apr 21, 2026 at 5:59 PM Oliveira, Niko < > > > > > > > [email protected] > > > > > > > > > > > > > > > > > >> > > wrote: > > > > > > > > >> > > > > > > > > > > >> > > > I am with Jens on this one. I think we're > complicating > > > > > Airflow > > > > > > > to > > > > > > > > >> get > > > > > > > > >> > > > around a bad practice. If stability of your Dags is > > > > critical > > > > > > and > > > > > > > > >> they > > > > > > > > >> > are > > > > > > > > >> > > > highly versioned then I think as Jens suggested > > running > > > > them > > > > > > > > >> through a > > > > > > > > >> > > > pipeline that first deploys them to a dev or gamma > > > > > environment > > > > > > > > which > > > > > > > > >> > > > verifies that quality of the Dags is what you > expect. > > If > > > > > > > something > > > > > > > > >> > slips > > > > > > > > >> > > > through, then it's just normal software practices of > > > > either > > > > > > > > >> reverting > > > > > > > > >> > and > > > > > > > > >> > > > rolling back or rolling forward with a fix pushed > > > through > > > > > the > > > > > > > > >> > pipeline. I > > > > > > > > >> > > > don't think Airflow should be aware of that process > or > > > > > > > opinionated > > > > > > > > >> > about > > > > > > > > >> > > it. > > > > > > > > >> > > > > > > > > > > > >> > > > Cheers, > > > > > > > > >> > > > Niko > > > > > > > > >> > > > ------------------------------ > > > > > > > > >> > > > *From:* Jens Scheffler <[email protected]> > > > > > > > > >> > > > *Sent:* Monday, April 20, 2026 11:17 AM > > > > > > > > >> > > > *To:* [email protected] < > [email protected]> > > > > > > > > >> > > > *Subject:* RE: [EXT] [DISCUSS] DAG Version Pinning > for > > > > > > > Deployment > > > > > > > > >> > Gating > > > > > > > > >> > > > (Building on AIP-63) > > > > > > > > >> > > > > > > > > > > > >> > > > CAUTION: This email originated from outside of the > > > > > > organization. > > > > > > > > Do > > > > > > > > >> not > > > > > > > > >> > > > click links or open attachments unless you can > confirm > > > the > > > > > > > sender > > > > > > > > >> and > > > > > > > > >> > > know > > > > > > > > >> > > > the content is safe. > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > > AVERTISSEMENT: Ce courrier électronique provient > d’un > > > > > > expéditeur > > > > > > > > >> > externe. > > > > > > > > >> > > > Ne cliquez sur aucun lien et n’ouvrez aucune pièce > > > jointe > > > > si > > > > > > > vous > > > > > > > > ne > > > > > > > > >> > > pouvez > > > > > > > > >> > > > pas confirmer l’identité de l’expéditeur et si vous > > > n’êtes > > > > > pas > > > > > > > > >> certain > > > > > > > > >> > > que > > > > > > > > >> > > > le contenu ne présente aucun risque. > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > > Hi, > > > > > > > > >> > > > > > > > > > > > >> > > > I am still quite sceptical. Yes, if such pinning is > > > made, > > > > > then > > > > > > > per > > > > > > > > >> Dag > > > > > > > > >> > a > > > > > > > > >> > > > change need to be possible via UI and API. But I > still > > > see > > > > > it > > > > > > as > > > > > > > > >> > > > checken-and-egg - so you want to run a pinned > version > > > but > > > > > then > > > > > > > how > > > > > > > > >> do > > > > > > > > >> > > > you test the changes (w/o moving a version pin)? > Then > > > > again > > > > > > some > > > > > > > > >> test > > > > > > > > >> > > > mode is needed or per run you need to make a "test > > run" > > > > with > > > > > > > > another > > > > > > > > >> > > > version. Smells a bit like mis-using a production > > system > > > > for > > > > > > > > >> testing. > > > > > > > > >> > > > > > > > > > > > >> > > > On the other hand, yes if all Dags share the same > Git > > > repo > > > > > > then > > > > > > > > >> merging > > > > > > > > >> > > > a branch to some other will switch all Dags at the > > same > > > > > time. > > > > > > > > Still > > > > > > > > >> you > > > > > > > > >> > > > could utilize standard Git tools and cherry-pick > > > > individual > > > > > > > > changes > > > > > > > > >> and > > > > > > > > >> > > > no force to always make a full rollout. At least 80% > > > > > possible > > > > > > > with > > > > > > > > >> > > > standard CI/CD tools and Git. > > > > > > > > >> > > > > > > > > > > > >> > > > TLDR I see the danger that instead of a proper CI/CD > > and > > > > > test > > > > > > > > system > > > > > > > > >> > > > such a feature might feel like you can easily test > on > > a > > > > > > > production > > > > > > > > >> > > > system. Effectively it would be needed allowing to > > > start a > > > > > Dag > > > > > > > > with > > > > > > > > >> any > > > > > > > > >> > > > version to also be able to jump back as a reversion. > > > Even > > > > > > > though, > > > > > > > > >> yes, > > > > > > > > >> > > > agree, all is technically possible. > > > > > > > > >> > > > > > > > > > > > >> > > > Jens > > > > > > > > >> > > > > > > > > > > > >> > > > On 20.04.26 16:40, Jarek Potiuk wrote: > > > > > > > > >> > > > > +1 to what Ephraim wrote. I think that was a > natural > > > > next > > > > > > step > > > > > > > > we > > > > > > > > >> > > > > discussed, but it needs significant refinement, > > > starting > > > > > > with > > > > > > > > the > > > > > > > > >> > > actual > > > > > > > > >> > > > > use cases it should serve and the UX for user > > > > > interaction. I > > > > > > > > think > > > > > > > > >> > > > related > > > > > > > > >> > > > > database changes are pretty secondary. Use cases > > cover > > > > > runs, > > > > > > > > >> re-runs, > > > > > > > > >> > > > > backfills, CI testing, rollbacks, etc. Following > the > > > > > > > > >> "documentation > > > > > > > > >> > > > first" > > > > > > > > >> > > > > approach discussed in separate thread, describing > > the > > > > > > context > > > > > > > > and > > > > > > > > >> > > > intention > > > > > > > > >> > > > > of what we want to achieve is much more important > > than > > > > DB > > > > > > > schema > > > > > > > > >> > > changes. > > > > > > > > >> > > > > Once we know which use cases we want to serve, the > > DB > > > > > schema > > > > > > > > >> changes > > > > > > > > >> > > and > > > > > > > > >> > > > > other related items will emerge naturally. > > > > > > > > >> > > > > > > > > > > > > >> > > > > On Mon, Apr 20, 2026 at 3:15 PM Ephraim Anierobi < > > > > > > > > >> > > > [email protected]> > > > > > > > > >> > > > > wrote: > > > > > > > > >> > > > > > > > > > > > > >> > > > >> Hi Piyush, thanks for starting this discussion. > > > > > > > > >> > > > >> > > > > > > > > >> > > > >> I like the proposal. We can introduce an active > > > > execution > > > > > > > > version > > > > > > > > >> > for > > > > > > > > >> > > > >> "versioned bundles" and make scheduler/API > resolve > > > > > through > > > > > > > it. > > > > > > > > >> The > > > > > > > > >> > > hard > > > > > > > > >> > > > >> part of this is making airflow able to > distinguish > > > the > > > > > > latest > > > > > > > > >> parsed > > > > > > > > >> > > > >> dagmodel's metadata from active scheduling > > metadata. > > > I > > > > > will > > > > > > > > >> suggest > > > > > > > > >> > > you > > > > > > > > >> > > > >> draft this in a google docs and share for further > > > > > > > discussions. > > > > > > > > >> > > > >> > > > > > > > > >> > > > >> Regards > > > > > > > > >> > > > >> - Ephraim > > > > > > > > >> > > > >> > > > > > > > > >> > > > >> On Mon, 20 Apr 2026 at 01:31, Piyush Maheshwari < > > > > > > > > >> > > [email protected]> > > > > > > > > >> > > > >> wrote: > > > > > > > > >> > > > >> > > > > > > > > >> > > > >>> Thanks for sharing your thoughts Jens. > > > > > > > > >> > > > >>> > > > > > > > > >> > > > >>>> be able to test it? … a Q&A/Testing environment > > to > > > be > > > > > > able > > > > > > > to > > > > > > > > >> > > sign-off > > > > > > > > >> > > > >>> changes. > > > > > > > > >> > > > >>> Yes, we’ve have built an isolated airflow > > > environment > > > > to > > > > > > run > > > > > > > > >> > > regression > > > > > > > > >> > > > >>> checks before promoting to production. > > > > > > > > >> > > > >>> > > > > > > > > >> > > > >>> As you suggested, we’re already running both > > generic > > > > and > > > > > > > > >> DAG-custom > > > > > > > > >> > > > >> static > > > > > > > > >> > > > >>> checks in a CI job as a required step to merge > to > > > the > > > > > main > > > > > > > > >> branch. > > > > > > > > >> > > > >>> > > > > > > > > >> > > > >>>> But then the "main" branch might be best suited > > if > > > > > > > > >> > > > >>> implemented on the test system > > > > > > > > >> > > > >>> In this case, problematic commits on “main” can > > > choke > > > > > > other > > > > > > > > >> > unrelated > > > > > > > > >> > > > >>> changes. > > > > > > > > >> > > > >>> So the other option would be to revert the > > > problematic > > > > > > > commits > > > > > > > > >> and > > > > > > > > >> > > > deploy > > > > > > > > >> > > > >>> forward. > > > > > > > > >> > > > >>> > > > > > > > > >> > > > >>> However, a key limitation with this approach > that > > > > > remains > > > > > > is > > > > > > > > >> that a > > > > > > > > >> > > > >> commit > > > > > > > > >> > > > >>> affecting multiple DAGs goes live for either all > > > DAGs > > > > or > > > > > > > none. > > > > > > > > >> > > > >>> > > > > > > > > >> > > > >>> Second important feature we get with this is > > instant > > > > > > > DAG-level > > > > > > > > >> > > rollback > > > > > > > > >> > > > >>> without waiting for a revert commit to merge and > > be > > > > > picked > > > > > > > by > > > > > > > > >> > > airflow. > > > > > > > > >> > > > >>> > > > > > > > > >> > > > >>> I think DAG-level version pinning can also > unlock > > a > > > > lot > > > > > of > > > > > > > > >> > > flexibility > > > > > > > > >> > > > >> for > > > > > > > > >> > > > >>> deployments including tiered rollouts, > > auto-rollback > > > > > > > triggers, > > > > > > > > >> > timed > > > > > > > > >> > > > >>> deployment windows and so on. > > > > > > > > >> > > > >>> > > > > > > > > >> > > > >>> Looking forward to hear your thoughts. > > > > > > > > >> > > > >>> Regards, > > > > > > > > >> > > > >>> Piyush > > > > > > > > >> > > > >>> > > > > > > > > >> > > > >>> On Sun, 19 Apr 2026 at 3:12 PM, Jens Scheffler < > > > > > > > > >> > [email protected]> > > > > > > > > >> > > > >>> wrote: > > > > > > > > >> > > > >>> > > > > > > > > >> > > > >>>> Thanks Piyush for dropping the discussion! > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > >>>> I think in general QA processes are important > > and a > > > > > valid > > > > > > > use > > > > > > > > >> > case. > > > > > > > > >> > > So > > > > > > > > >> > > > >> a > > > > > > > > >> > > > >>>> kind of pinning Dag versions really is > important. > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > >>>> Thinking about it, if you pin the version ... > how > > > > would > > > > > > you > > > > > > > > >> then > > > > > > > > >> > be > > > > > > > > >> > > > >> able > > > > > > > > >> > > > >>>> to test it? I assume you would need (and should > > > have > > > > or > > > > > > > > invest > > > > > > > > >> > > into) a > > > > > > > > >> > > > >>>> Q&A/Testing environment to be able to sign-off > > > > changes. > > > > > > > Both > > > > > > > > in > > > > > > > > >> > > > >>>> infrastructure but also for Dag changes. > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > >>>> If you are changing Dags first of all static > > checks > > > > on > > > > > > Dag > > > > > > > > code > > > > > > > > >> > are > > > > > > > > >> > > > >> very > > > > > > > > >> > > > >>>> much proposed as well as you can have tests > > > > implemented > > > > > > and > > > > > > > > >> test > > > > > > > > >> > > your > > > > > > > > >> > > > >>>> Dags and logic. Similar like software a CI/CD > > > system > > > > > will > > > > > > > be > > > > > > > > a > > > > > > > > >> > good > > > > > > > > >> > > > >>>> setup. Alongside Dag changes also have logical > > > > changes > > > > > > that > > > > > > > > >> mostly > > > > > > > > >> > > can > > > > > > > > >> > > > >>>> only be tested in a live system and not as > static > > > > > checks. > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > >>>> Have you considered using Git and a set of > > branches > > > > for > > > > > > > > >> > implementing > > > > > > > > >> > > > >>>> such staging? E.g. you have a git repo and you > > plan > > > > to > > > > > > make > > > > > > > > >> > changes. > > > > > > > > >> > > > >>>> Then you would open a PR for the change and > merge > > > it > > > > to > > > > > > the > > > > > > > > >> "main" > > > > > > > > >> > > > >>>> branch - and there in your CI/CD you can check > > all > > > > > sorts > > > > > > of > > > > > > > > >> static > > > > > > > > >> > > > >>>> checks and tests. But then the "main" branch > > might > > > be > > > > > > best > > > > > > > > >> suited > > > > > > > > >> > if > > > > > > > > >> > > > >>>> implemented on the test system. Once you > validate > > > the > > > > > > > changes > > > > > > > > >> > > > >> end-to-end > > > > > > > > >> > > > >>>> you could make another PR for example to a > "prod" > > > > > branch. > > > > > > > And > > > > > > > > >> if > > > > > > > > >> > > your > > > > > > > > >> > > > >>>> production system is only pulling Dags from the > > > > "prod" > > > > > > > branch > > > > > > > > >> then > > > > > > > > >> > > you > > > > > > > > >> > > > >>>> can have this merging strategy as a staging > > setup. > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > >>>> Would this resolve your PING problem? Or which > > > other > > > > > > detail > > > > > > > > in > > > > > > > > >> the > > > > > > > > >> > > use > > > > > > > > >> > > > >>>> case would require a PIN on top of a staging > > > > strategy? > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > >>>> Jens > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > >>>> P.S.: Have enabled your confluence account > after > > it > > > > was > > > > > > > > >> created in > > > > > > > > >> > > > >> order > > > > > > > > >> > > > >>>> to write to Confluence, sorry, typical pitfall > > > after > > > > > > > account > > > > > > > > >> > > creation > > > > > > > > >> > > > >>>> permissions were not set. Now it should work. > Let > > > me > > > > > know > > > > > > > if > > > > > > > > >> not. > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > >>>> On 19.04.26 01:40, Piyush Maheshwari wrote: > > > > > > > > >> > > > >>>>> Hi everyone, > > > > > > > > >> > > > >>>>> I'm a new contributor to Airflow. I'd like to > > > > propose > > > > > a > > > > > > > new > > > > > > > > >> > feature > > > > > > > > >> > > > >> for > > > > > > > > >> > > > >>>> Airflow: DAG Version Pinning. > > > > > > > > >> > > > >>>>> Building on the foundation introduced by > AIP-63: > > > DAG > > > > > > > > >> Versioning ( > > > > > > > > >> > > > >> > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://urldefense.com/v3/__https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-63*3A*DAG*Versioning__;JSsr!!Ci6f514n9QsL8ck!l3ZKTOw996h9qu4NR0VT4ouUryUdk_HmXUAVPbwCHwPwn0N2CCptVdx95-V0BoRFjws9huE_1Vy-THL8jw$ > > > > > > > > >> > > > >>> ), > > > > > > > > >> > > > >>>> this proposal aims to extend Airflow's > > capabilities > > > > to > > > > > > > > support > > > > > > > > >> > true > > > > > > > > >> > > > >>>> continuous deployment (CD) gating and safer > > release > > > > > > cycles. > > > > > > > > >> > > > >>>>> The Problem & Use Cases > > > > > > > > >> > > > >>>>> Currently, the scheduler always creates > DagRuns > > > > using > > > > > > the > > > > > > > > >> latest > > > > > > > > >> > > > >> parsed > > > > > > > > >> > > > >>>> DagVersion. This means that the updated DAG > code > > is > > > > > > > deployed > > > > > > > > >> > (takes > > > > > > > > >> > > > >>> effect) > > > > > > > > >> > > > >>>> right after the dag-processor processes it. > While > > > > this > > > > > is > > > > > > > > great > > > > > > > > >> > for > > > > > > > > >> > > > >> rapid > > > > > > > > >> > > > >>>> development, teams running business-critical > > > > pipelines > > > > > > > often > > > > > > > > >> need > > > > > > > > >> > > > >>> stricter > > > > > > > > >> > > > >>>> deployment mechanisms. Specifically: > > > > > > > > >> > > > >>>>> * > > > > > > > > >> > > > >>>>> Safe Deployment Gating: The ability to pin a > DAG > > > to > > > > > its > > > > > > > last > > > > > > > > >> > known > > > > > > > > >> > > > >>>> stable version while new code is parsed in the > > > > > > background. > > > > > > > > This > > > > > > > > >> > > allows > > > > > > > > >> > > > >>> the > > > > > > > > >> > > > >>>> new version to be held back until it passes > > > automated > > > > > > > > >> regression > > > > > > > > >> > > tests > > > > > > > > >> > > > >> or > > > > > > > > >> > > > >>>> receives explicit manual approval. > > > > > > > > >> > > > >>>>> * > > > > > > > > >> > > > >>>>> Instant Rollbacks: If an issue is detected in > a > > > > newly > > > > > > > > promoted > > > > > > > > >> > DAG > > > > > > > > >> > > > >>>> version, users need the capability to instantly > > > roll > > > > > back > > > > > > > to > > > > > > > > a > > > > > > > > >> > > > previous > > > > > > > > >> > > > >>>> version via the UI/API, without having to > revert > > > the > > > > > > > > underlying > > > > > > > > >> > code > > > > > > > > >> > > > >> and > > > > > > > > >> > > > >>>> wait for the repository sync and DAG processing > > > > cycle. > > > > > > > > >> > > > >>>>> High-Level Proposed Solution > > > > > > > > >> > > > >>>>> Introduce an optional active_dag_version_id to > > the > > > > > > > DagModel. > > > > > > > > >> This > > > > > > > > >> > > > >> field > > > > > > > > >> > > > >>>> can be used to pin a DAG version for scheduling > > and > > > > > > > > execution, > > > > > > > > >> > while > > > > > > > > >> > > > >> the > > > > > > > > >> > > > >>>> dag-processor can continue to parse and > register > > > > newer > > > > > > DAG > > > > > > > > >> > versions. > > > > > > > > >> > > > >>>>> * > > > > > > > > >> > > > >>>>> When this pin is set, the scheduler and API > will > > > > > respect > > > > > > > the > > > > > > > > >> > pinned > > > > > > > > >> > > > >>>> version for creating runs and executing tasks, > > > > > separating > > > > > > > the > > > > > > > > >> > > parsing > > > > > > > > >> > > > >> of > > > > > > > > >> > > > >>>> new code from the execution of new code. > > > > > > > > >> > > > >>>>> * > > > > > > > > >> > > > >>>>> If the pin is NULL, the system defaults to the > > > > current > > > > > > > > >> behavior > > > > > > > > >> > > > >> (always > > > > > > > > >> > > > >>>> executing the latest parsed version). This way, > > we > > > > can > > > > > > > > maintain > > > > > > > > >> > > > >> complete > > > > > > > > >> > > > >>>> backwards compatibility. > > > > > > > > >> > > > >>>>> I have put together some detailed notes > covering > > > the > > > > > > data > > > > > > > > >> model > > > > > > > > >> > > > >>> changes, > > > > > > > > >> > > > >>>> database migrations, and edge cases with this > > > > approach. > > > > > > If > > > > > > > > >> there > > > > > > > > >> > is > > > > > > > > >> > > > >>> general > > > > > > > > >> > > > >>>> alignment that this fits the vision for > Airflow, > > I > > > > > would > > > > > > > like > > > > > > > > >> to > > > > > > > > >> > > take > > > > > > > > >> > > > >>> this > > > > > > > > >> > > > >>>> proposal through the formal AIP review process. > > > > > > > > >> > > > >>>>> But I would love to get the community's > feedback > > > on > > > > > the > > > > > > > > >> feature > > > > > > > > >> > and > > > > > > > > >> > > > >> the > > > > > > > > >> > > > >>>> high-level approach. > > > > > > > > >> > > > >>>>> I'll also need someone to grant me access to > > > create > > > > > > > content > > > > > > > > on > > > > > > > > >> > the > > > > > > > > >> > > > >>>> Airflow Confluence wiki. > > > > > > > > >> > > > >>>>> Thanks for your time! > > > > > > > > >> > > > >>>>> Regards, > > > > > > > > >> > > > >>>>> Piyush > > > > > > > > >> > > > >>>>> > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > > > > > > > --------------------------------------------------------------------- > > > > > > > > >> > > > >>>> To unsubscribe, e-mail: > > > > > > [email protected] > > > > > > > > >> > > > >>>> For additional commands, e-mail: > > > > > > > [email protected] > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > --------------------------------------------------------------------- > > > > > > > > >> > > > To unsubscribe, e-mail: > > > > [email protected] > > > > > > > > >> > > > For additional commands, e-mail: > > > > > [email protected] > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
