Hi everyone, I'd like to request the community's review of the AIP https://cwiki.apache.org/confluence/spaces/AIRFLOW/pages/421958312/AIP-109+DAG+Version+Pinning . If there are no differing opinions I'd like to take it to a vote soon.
Also, thank you Ephraim for the re-review. I've addressed all comments from the latest review and updated the AIP accordingly. Thanks and regards, Piyush On Fri, Aug 7, 2026 at 4:10 PM Ephraim Anierobi <[email protected]> wrote: > Hi Piyush, > > Thank you for this summary. It is accurate, and it matches where we landed. > Thank you also for the many rounds of changes. The document is much clearer > now than when we started. > > I want to state my position plainly for the thread. My main concern was > that a pin on a DagVersion could not hold, because the row could change > under it and the bundle pointer could move. The pin guard fixes both. With > that, plus the retention rule, the stale DAG change, the purity scope, and > the divergence warning, I no longer object to the pin target. The AIP now > says what it delivers and what it does not. That was my real goal. > > One thing I want to be clear about. My review was about whether the design > holds, not about whether the feature is worth building. Those are two > different questions, and the second one belongs to the community. > > The feature adds pin awareness in several core places: the version write > path, db clean, the scheduler's eligibility query for import errors and > stale DAGs, DagModel and asset field resolution in the DAG processor, and a > synchronous update when a pin changes. Each change is small on its own. > Together they mean these paths must consider pins from now on. > > What it gives back is per-DAG control over when a version takes effect, for > pure DAGs that come from versioned bundles. That is real and useful for the > teams who need it. Whether it is worth the added complexity in core paths > is a judgment call, and voters should make it with both sides stated. I am > not voting against it. I only do not want my resolved concerns to be read > as a view on that question. > > I have left my remaining comments on the document itself. They are small > and none of them changes the design, but I think they should be settled > before the vote. > > On the vote, I agree with starting it again instead of continuing the old > thread. The document has changed a lot. A fresh call, with a short note on > what changed, would be fair to everyone who already voted. > > Thanks again for your patience through all of this. The proposal is in much > better shape. > > Regards, > Ephraim > > On Wed, 5 Aug 2026 at 10:55, Piyush Maheshwari <[email protected] > > > wrote: > > > Hi everyone, > > > > I'd like to thank Ephraim for the depth of this review. Following the > > previous email, we had a series of detailed follow-up discussions, and I > > believe we have converged. This email summarizes where we landed, answers > > the concerns from the previous email for the benefit of the thread, and > > describes the updates I have now made to the AIP. I would like the > > community's review of the updated AIP ( > > > > > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-109+DAG+Version+Pinning > > ) > > > > == Precise scope of the proposal == > > > > The discussion sharpened what the AIP promises, and it now states this > > explicitly: > > > > - The goal is deterministic, per-DAG control over when a new version of a > > DAG takes effect, plus an instant, API-driven rollback to a prior > version, > > without affecting other DAGs sharing the same bundle. > > > > - A pin fixes the serialized DAG the scheduler enforces (task set, edges, > > schedule, display fields) and the bundle (name and version) the worker > > checks out. It provides graph and code-level determinism, not > > behavior-level determinism, since a task that reads live external state > at > > execution time still does so under a pin. > > > > - Pinning guarantees are stated for DAGs that are "pure", meaning a given > > bundle (name and version) always produces the same serialized DAG. The > AIP > > now defines this purity condition up front. > > > > - For impure DAGs, different pinning guarantees are now documented for > > impure DAGs. A divergence between the pinned serialized DAG's snapshot > and > > the worker's fresh parse for execution-time fields is silent. The AIP now > > explicitly documents this as "silent divergence," a limitation, and adds > > detection for it. > > > > == Answers to the five concerns from the previous email == > > > > 1. "The latest DagVersion is mutable, so a pin does not freeze content." > > Addressed by introducing a pin guard in the DAG Processor's > > version-recording path. A pinned DagVersion is treated as referenced, so > > its serialized content is never overwritten in place; a content change > > creates a new DagVersion instead (the same path that already runs when > task > > instances reference the row). Note that the DAG processor only mutates > the > > latest DagVersion row, so a pin on an older version is already immutable > > today; the guard changes behavior only when the pinned version is also > the > > latest. > > > > 2. "A pin would not pin the source code, because bundle_version is > > refreshed in place on hash-identical parses." > > Addressed by the same guard, which stops the in-place > > bundle_version/version_data refresh from applying to a pinned row. When > the > > bundle advances without a serialized-DAG change while the pinned version > is > > the latest, the processor instead creates one new DagVersion carrying the > > new bundle_version, which becomes the unpinned latest; subsequent > no-change > > bundle advances keep updating that new latest in place exactly as today. > > This preserves the intent of the in-place refresh (the DAG's latest > version > > keeps tracking the bundle) while the pinned version's bundle reference > > stays frozen. Since new DagRuns will also carry the pinned bundle, and > > workers resolve their bundle checkout from the DagRun, the scheduler's > view > > and the worker's code stay on the same version. Note that this version > > creation, without a hash change, occurs only per pin advance, not once > per > > commit. > > > > 3. "Version identity is often accidental (impure DAGs)." > > Addressed by scoping (guarantees stated for pure DAGs, with the purity > > condition defined in the AIP) and by divergence detection, where a > > DagWarning is raised when a pinned DAG's fresh parse hash no longer > matches > > the pinned DagVersion's hash, naming the pinned version. The check is a > > no-op for pure DAGs. > > > > 4. "Cleanup can delete a pinned version and silently unpin via ON DELETE > > SET NULL." > > Addressed: airflow db clean will retain any DagVersion referenced by an > > active pin, the same way it retains task-instance-referenced versions. > > > > 5. "The deletion rule works against the main use case." > > Revised: a dag_id that disappears from the latest parse would still be > > marked stale on DagModel as today, but the scheduler's eligibility query > > will admit a stale DAG while it has an active pin, so it keeps running on > > its pinned version (parallel to the import-error exception already in the > > AIP). Actual removal requires an explicit unpin first, after which the > > normal stale-to-deleted path applies unchanged. This turns the defect > > scenario from a silent removal into an explicit act. > > > > == On pinning the bundle instead of the DagVersion == > > > > We discussed this alternative at length, in two variants, and the AIP's > > "Alternatives" section now covers both: > > > > - A global bundle pin (keep a history of bundle versions and freeze the > > whole bundle on one) is simple and atomic, but it pins every DAG in the > > bundle at once. It cannot keep a subset of DAGs on an older version while > > the rest advance, and per-DAG control is the primary use case, since a > > breaking change in a shared bundle typically affects only some of the > DAGs > > it emits. We see it as a potentially complementary, coarser-grained > control > > rather than a replacement. > > > > - A per-DAG pin keyed by the bundle (name and version) is not uniquely > > resolvable to a serialized DAG for impure DAGs, since the pair can map to > > several dag_version rows. It therefore cannot be expressed as a foreign > key > > to a specific version, and the latest matching row can resolve > differently > > across invocations, which is behavior drift while supposedly pinned. It > is > > also unstable over time, because a dag_version's bundle_version is > > refreshed in place on hash-identical parses, so a previously valid bundle > > version can stop resolving exactly when a rollback needs it. Freezing > that > > would require an immutable bundle-to-dag_version mapping, which amounts > to > > pinning the dag_version too and converges towards this proposal. > > > > == Operating models and rollback availability == > > > > Ephraim raised an important point about rollback availability. An > on-demand > > pin can only roll back to versions Airflow already recorded, so a change > > that never altered the serialized DAG (e.g. one confined to an imported > > helper module) has no distinct version to return to unless it was pinned > at > > the time. The AIP now states this honestly under a new "Usage Models and > > Rollback Availability" section, which describes two models the primitive > > serves: > > > > 1. On-demand pinning (incident response, planned cutovers): rollback > > targets are the recorded DAG versions. Changes that altered the > serialized > > DAG (and had task instances) are reachable; changes that did not are a > > stated limitation. > > 2. Standing pins driven by an external release orchestrator: every DAG > > always carries a pin and the orchestrator advances it per deployment. > Every > > pinned state remains an immutable rollback target, closing the gap above, > > at the cost of operating the orchestrator. The consequences of the same > are > > also covered in the AIP. > > > > Airflow itself only ships the pin/unpin primitive; neither model is > > mandatory to use this feature. > > > > == Updated AIP and proposed timeline == > > > > I have updated AIP-109 with all of the above: > > > > > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-109+DAG+Version+Pinning > > > > I would appreciate the community's review of the updated proposal. If > > multiple maintainers review it and if no major open questions or comments > > remain by Wednesday, August 12, I plan to restart the vote phase. > > > > Thanks again to Ephraim for the multiple rounds of thorough review that > got > > the proposal to this shape. > > > > Regards, > > Piyush > > > > On Tue, Jul 28, 2026 at 3:47 PM Ephraim Anierobi < > > [email protected]> > > wrote: > > > > > Thank you for this proposal, and for the careful work behind it. I > took a > > > second, deeper look at the document, and I want to share my concerns, > > > because I believe they change the direction rather than the details. > > > > > > First, I want to recognize the real problem the AIP identifies. The > parse > > > loop controls when new code becomes active, and that timing is not > > > deterministic. Teams need a way to control activation time, and they > > need a > > > fast rollback path. This is a genuine gap, and I am glad the AIP puts a > > > spotlight on it. > > > > > > After the second review, however, I believe the proposal rests on a > > > misunderstanding of what a DagVersion is. A DagVersion is a historical > > > record of what Airflow observed after a parse. It is an output of the > > > system, not an input to it. Dag versioning exists so that running and > > past > > > task instances keep a faithful record of the code they ran against. > > *Bundle > > > versioning* is the mechanism that decides which source code a run uses. > > The > > > AIP reads DagVersion as a release artifact that users can select and > > > activate, and the current implementation does not support that reading: > > > > > > 1. *The latest DagVersion is mutable*. When no task instance references > > it, > > > write_dag() replaces its serialized content, hash, and Dag code in > place, > > > on the same version ID. A pin does not create a task instance > reference. > > So > > > a user could pin the current version before a risky merge, and the next > > > parse could replace the content behind that pin. The pin would not give > > the > > > determinism it promises. > > > > > > 2. *A pin on the current DagVersion would not pin the source code*. > When > > a > > > new commit does not change the serialized content, `write_dag()` > updates > > > bundle_version on the latest DagVersion in place. A change to a helper > > > module or an SQL file does not change the serialized hash, so workers > > would > > > run the new commit under the pinned version. > > > > > > 3. *Version identity is often accidental.* Parse-time variability and > > > serializer defects can create versions that no one intended. Airflow > > ships > > > an inflation checker and a dedicated Dag warning type because this > > problem > > > is real and common. Dag factories at large scale, the AIP's own key > > > example, are the most exposed. A pin would often select an accidental > > > snapshot rather than a deliberate release. > > > > > > 4. *The proposal does not address cleanup*. Today, `airflow db clean` > > keeps > > > the latest version per Dag and skips only versions that task instances > > > reference. An older pinned version with no task instances is eligible > for > > > deletion, and the proposed ON DELETE SET NULL behavior would then > > silently > > > return the Dag to the latest version, exactly when the pin matters > most. > > > The document does not cover this interaction. > > > > > > 5. *The deletion rule works against the main use case*. The AIP states > > that > > > removal from a new bundle version deletes the Dag, even when it is > > pinned. > > > A defect in a dag-factory file that changes or drops Dag IDs removes > Dags > > > from the parse results. In that case, Airflow would delete the Dag and > > > discard the pin, so the protection would be absent in one of the most > > > common failure modes the AIP is motivated by. > > > > > > These are not implementation gaps that more safeguards can close. They > > all > > > follow from the same source: the design asks a history table to act as > a > > > deployment mechanism. Making that work would mean redefining what > > > DagVersion is, and I do not think we should. > > > > > > The goal itself is worth pursuing at the right layer. Deterministic > > cutover > > > and instant rollback are operations on the *source artifact*. Bundle > > > versions already represent the source, and pinning or rolling back at > the > > > bundle layer would be atomic for all Dags and all support files at > once. > > I > > > would be glad to help explore that direction. > > > > > > Thanks again for raising this discussion. The problem is real, and I > hope > > > we can solve it together on a foundation that supports it. > > > > > > - Ephraim > > > > > > On Mon, 27 Jul 2026 at 06:53, Piyush Maheshwari < > > > [email protected]> > > > wrote: > > > > > > > Thank you, Sumit. > > > > > > > > I've started a voting thread as suggested: > > > > https://lists.apache.org/thread/bff758c5p68v1mtzhflytn7yzv0d8l1t > > > > > > > > Regards, > > > > Piyush > > > > > > > > On Fri, Jul 24, 2026 at 12:08 PM Sumit Maheshwari < > > > [email protected]> > > > > wrote: > > > > > > > > > I've reviewed it again today and left some comments. Overall, it > > looks > > > > > great and would be a great feature add in Airflow. I think you can > > > start > > > > an > > > > > official voting thread on this AIP. > > > > > > > > > > > > > > > On Sun, Jul 19, 2026 at 7:32 PM Piyush Maheshwari < > > > > > [email protected]> wrote: > > > > > > > > > > > Hi everyone, > > > > > > Thanks for the reviews on AIP-109 ( > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-109+DAG+Version+Pinning > > > > > > ). > > > > > > Over the past few weeks, I implemented the feature in our fork > > > (nearly > > > > > > done). > > > > > > Based on the insights and learnings from this process, I have > > further > > > > > > refined the AIP while addressing all comments. > > > > > > > > > > > > Please review the updated version. I'll wait to collect some > > feedback > > > > and > > > > > > then take it to a vote. > > > > > > I can start contributing PRs as soon as the AIP is approved. > > > > > > > > > > > > Thanks and regards, > > > > > > Piyush > > > > > > > > > > > > On Wed, Jun 3, 2026 at 2:01 PM Ephraim Anierobi < > > > > > [email protected] > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > Hi Piyush, > > > > > > > > > > > > > > Thanks for the AIP. I’ve added a few comments to the doc. > Please > > > > take a > > > > > > > look, as I think we should iron out those areas and fully > > > understand > > > > > the > > > > > > > implications before moving forward. > > > > > > > > > > > > > > Regards > > > > > > > - Ephraim > > > > > > > > > > > > > > On Tue, 26 May 2026 at 18:24, Przemysław Mirowski < > > > [email protected] > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > I checked the API - +1. Thanks for writing this up! > > > > > > > > > > > > > > > > Thanks Nathan for mentioning the #63884 PR. It is nice > addition > > > > > > (already > > > > > > > > merged), and I think that it will be really useful for users > > who > > > > got > > > > > > used > > > > > > > > to Airflow 2 retry mechanism. I think also that your PR and > > > API-109 > > > > > are > > > > > > > > complementary as one is focusing on the versioning behaviour > > for > > > > > > retrying > > > > > > > > task, the latter is focusing on versioning behaviour for new > > Dag > > > > > Runs. > > > > > > > > ________________________________ > > > > > > > > From: Christos Bisias <[email protected]> > > > > > > > > Sent: 25 May 2026 09:46 > > > > > > > > To: [email protected] <[email protected]> > > > > > > > > Subject: Re: [DISCUSS] DAG Version Pinning for Deployment > > Gating > > > > > > > (Building > > > > > > > > on AIP-63) > > > > > > > > > > > > > > > > Hello, > > > > > > > > > > > > > > > > I'm a little late on the discussion but I just came across > the > > > AIP > > > > > and > > > > > > I > > > > > > > > like this idea. I've actually been thinking of working on > > > something > > > > > > > > similar, to allow people to handle a bad rollout by reverting > > to > > > > old > > > > > > code > > > > > > > > for that run without a full slow release pipeline. And this > > > covers > > > > > it. > > > > > > > > > > > > > > > > In my opinion, this seems more like a natural step towards > what > > > dag > > > > > > > > versioning is supposed to do, than a new feature. > > > > > > > > > > > > > > > > Thank you, > > > > > > > > Christos > > > > > > > > > > > > > > > > > > > > > > > > On Mon, May 25, 2026 at 8:15 AM Piyush Maheshwari < > > > > > > > > [email protected]> wrote: > > > > > > > > > > > > > > > > > Hi everyone, > > > > > > > > > I wanted to send a gentle nudge to review AIP-109: DAG > > Version > > > > > > Pinning > > > > > > > ( > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-109+DAG+Version+Pinning > > > > > > > > > ). > > > > > > > > > If there are no major concerns, I would like to take this > to > > a > > > > vote > > > > > > > soon. > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > Piyush > > > > > > > > > > > > > > > > > > On Fri, May 15, 2026 at 7:44 AM Piyush Maheshwari < > > > > > > > > > [email protected]> wrote: > > > > > > > > > > > > > > > > > > > Thanks for the note, Sumit. Based on the feedback, I've > > > drafted > > > > > an > > > > > > > AIP > > > > > > > > > > that is now up for review. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-109+DAG+Version+Pinning > > > > > > > > > > > > > > > > > > > > Would like to get the community's feedback on the same. > > > > > > > > > > > > > > > > > > > > Nathan, I remember seeing your work (the issue and an > older > > > PR) > > > > > > while > > > > > > > > > > reviewing all ongoing work related to DAG versions. I > agree > > > > with > > > > > > the > > > > > > > > PR's > > > > > > > > > > intent, although I haven't reviewed it yet. > > > > > > > > > > I understand your PR makes the version-pinned execution > > > > behavior > > > > > of > > > > > > > > > reruns > > > > > > > > > > and backfills configurable. > > > > > > > > > > However, this discussion revolves around the behavior for > > new > > > > > runs > > > > > > > > only. > > > > > > > > > > We need the capability to pin a DAG to a specific version > > for > > > > > > future > > > > > > > > runs > > > > > > > > > > instead. > > > > > > > > > > Hope that clarifies. > > > > > > > > > > > > > > > > > > > > Regards, > > > > > > > > > > Piyush > > > > > > > > > > > > > > > > > > > > On Thu, May 14, 2026 at 5:11 PM Nathan Hadfield < > > > > > > > > > [email protected]> > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > >> Hello, > > > > > > > > > >> > > > > > > > > > >> I saw AIP-109 that was created in relation to this > > > discussion > > > > > and > > > > > > > > > thought > > > > > > > > > >> I’d better mention this PR that I’ve been working on > for a > > > > while > > > > > > and > > > > > > > > is > > > > > > > > > >> close to being approved. > > > > > > > > > >> > > > > > > > > > >> https://github.com/apache/airflow/pull/63884 > > > > > > > > > >> > > > > > > > > > >> It is very much related to the motivations described and > > > > > > implements > > > > > > > > the > > > > > > > > > >> desire for control over the behaviour when > > > > clearing/backfilling > > > > > > > runs. > > > > > > > > > >> > > > > > > > > > >> Happy to discuss best steps for this here or on the PR. > > > > > > > > > >> > > > > > > > > > >> Cheers, > > > > > > > > > >> > > > > > > > > > >> Nathan > > > > > > > > > >> > > > > > > > > > >> From: Przemysław Mirowski <[email protected]> > > > > > > > > > >> Date: Tuesday, 28 April 2026 at 21:54 > > > > > > > > > >> To: [email protected] <[email protected]> > > > > > > > > > >> Subject: Re: [DISCUSS] DAG Version Pinning for > Deployment > > > > Gating > > > > > > > > > >> (Building on AIP-63) > > > > > > > > > >> > > > > > > > > > >> This Message Is From an External Sender > > > > > > > > > >> This message came from outside your organization. > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > >> > P.S. In my opinion, what can be done in/around git, > > should > > > > be > > > > > > done > > > > > > > > > >> there. Recreation of CI/CD in any form inside of Airflow > > > > itself > > > > > is > > > > > > > > > >> something which should not be done. > > > > > > > > > >> > I'm glad we agree on this :) I suppose we just > disagree > > on > > > > > what > > > > > > is > > > > > > > > > >> possible outside of Airflow :p > > > > > > > > > >> > > > > > > > > > >> I think that we just disagree on what the issue is, not > on > > > > what > > > > > is > > > > > > > > > >> possible/should be outside of Airflow. > > > > > > > > > >> > > > > > > > > > >> > I think we are trying to duplicate what we already > have > > in > > > > > Git. > > > > > > > > > >> > > > > > > > > > >> Not really if we are only referring to version pinning. > As > > > far > > > > > as > > > > > > I > > > > > > > am > > > > > > > > > >> aware of how things are working, there is no possibility > > to > > > > > > > determine > > > > > > > > > that > > > > > > > > > >> Dag after e.g. deployment on 1:00:00 PM will be exactly > > > parsed > > > > > and > > > > > > > > used > > > > > > > > > >> since 1:01:00 PM forward. Basically, what version > pinning > > > > would > > > > > > > > provide > > > > > > > > > is > > > > > > > > > >> the full control of the time since the given version > will > > be > > > > > used > > > > > > > > > >> (currently we can only have more-or-less timing which in > > > some > > > > > > cases, > > > > > > > > is > > > > > > > > > not > > > > > > > > > >> sufficient). The "quick revert" is the consequence of > > having > > > > > above > > > > > > > > > >> possibility. > > > > > > > > > >> > > > > > > > > > >> Looking at the general concerns, with having that > feature > > or > > > > > not, > > > > > > > > users > > > > > > > > > >> can pretty easily test things on production, but it just > > > > > requires > > > > > > > more > > > > > > > > > time > > > > > > > > > >> between iterations without it. IMHO it will not change > the > > > > need > > > > > > for > > > > > > > > > >> Airflow-related platform teams which makes sure, by > > > > > > > > standards/policies, > > > > > > > > > >> that things are properly tested before production > > > deployment. > > > > I > > > > > > > think > > > > > > > > > that > > > > > > > > > >> assumption that some users will misuse this feature is > > true > > > > > (like > > > > > > > with > > > > > > > > > most > > > > > > > > > >> of the features really), but on the other hand it would > > > > provide > > > > > > more > > > > > > > > > >> control for other users. The other solution possibly > would > > > be > > > > to > > > > > > > make > > > > > > > > > Dag > > > > > > > > > >> Processor work more on "events" instead of "simple" > > parsing > > > > loop > > > > > > (I > > > > > > > > > recall > > > > > > > > > >> that there was some PR couple years ago with PoC of > that, > > > but > > > > I > > > > > > > > couldn't > > > > > > > > > >> quickly find it). > > > > > > > > > >> > > > > > > > > > >> ________________________________ > > > > > > > > > >> From: Jarek Potiuk <[email protected]> > > > > > > > > > >> Sent: 28 April 2026 17:02 > > > > > > > > > >> To: [email protected] <[email protected]> > > > > > > > > > >> Subject: Re: [DISCUSS] DAG Version Pinning for > Deployment > > > > Gating > > > > > > > > > >> (Building on AIP-63) > > > > > > > > > >> > > > > > > > > > >> Same concerns. I think we are trying to duplicate what > we > > > > > already > > > > > > > have > > > > > > > > > in > > > > > > > > > >> Git—branches and reverts, for example—by moving what > > should > > > be > > > > > > > managed > > > > > > > > > as > > > > > > > > > >> part of the development process to Airflow UI. > > > > > > > > > >> > > > > > > > > > >> Almost everything you describe can be done with: > > > > > > > > > >> > > > > > > > > > >> * having a dev/staging system configured properly to use > > > > > > dev/staging > > > > > > > > > >> branches > > > > > > > > > >> * Having a process of managing development and a proper > > > > > branching > > > > > > > > > strategy > > > > > > > > > >> * single git command (for example, `git revert XXXX` > > > followed > > > > by > > > > > > > push > > > > > > > > to > > > > > > > > > >> the right branch) > > > > > > > > > >> > > > > > > > > > >> J. > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > >> On Tue, Apr 28, 2026 at 10:34 AM Pierre Jeambrun < > > > > > > > > [email protected] > > > > > > > > > > > > > > > > > > > >> wrote: > > > > > > > > > >> > > > > > > > > > >> > At first glance I tend to agree with Jens and Niko. > > > > > > > > > >> > > > > > > > > > > >> > I understand the request, but I agree that this > resolves > > > > CI/CD > > > > > > and > > > > > > > > > >> testing > > > > > > > > > >> > issues that should probably be remain outside Airflow. > > > > > > > > > >> > > > > > > > > > > >> > On Mon, Apr 27, 2026 at 7:43 PM Oliveira, Niko < > > > > > > > [email protected] > > > > > > > > > > > > > > > > > > >> > wrote: > > > > > > > > > >> > > > > > > > > > > >> > > Hey folks! > > > > > > > > > >> > > > > > > > > > > > >> > > > P.S. In my opinion, what can be done in/around > git, > > > > should > > > > > > be > > > > > > > > done > > > > > > > > > >> > > there. Recreation of CI/CD in any form inside of > > Airflow > > > > > > itself > > > > > > > is > > > > > > > > > >> > > something which should not be done. > > > > > > > > > >> > > > > > > > > > > > >> > > I'm glad we agree on this :) I suppose we just > > disagree > > > on > > > > > > what > > > > > > > is > > > > > > > > > >> > > possible outside of Airflow :p > > > > > > > > > >> > > > > > > > > > > > >> > > But at this point I will bow out of the conversation > > and > > > > let > > > > > > > > others > > > > > > > > > >> weigh > > > > > > > > > >> > > in. I'm not fully convinced any of these requested > > > > > behaviours > > > > > > > > > require > > > > > > > > > >> > > changes to Airflow (I think that's just masking some > > dev > > > > ops > > > > > > > > work). > > > > > > > > > >> But > > > > > > > > > >> > > also I'm not completely opposed to the change > either, > > > I'm > > > > > more > > > > > > > on > > > > > > > > > the > > > > > > > > > >> > > fence, so if others love the feature by all means > > > > implement > > > > > > it! > > > > > > > :) > > > > > > > > > >> > > > > > > > > > > > >> > > Cheers, > > > > > > > > > >> > > Niko > > > > > > > > > >> > > ________________________________ > > > > > > > > > >> > > From: Przemysław Mirowski <[email protected]> > > > > > > > > > >> > > Sent: Thursday, April 23, 2026 3:06 PM > > > > > > > > > >> > > To: [email protected] <[email protected]> > > > > > > > > > >> > > Subject: RE: [EXT] [DISCUSS] DAG Version Pinning for > > > > > > Deployment > > > > > > > > > Gating > > > > > > > > > >> > > (Building on AIP-63) > > > > > > > > > >> > > > > > > > > > > > >> > > CAUTION: This email originated from outside of the > > > > > > organization. > > > > > > > > Do > > > > > > > > > >> not > > > > > > > > > >> > > click links or open attachments unless you can > confirm > > > the > > > > > > > sender > > > > > > > > > and > > > > > > > > > >> > know > > > > > > > > > >> > > the content is safe. > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > AVERTISSEMENT: Ce courrier électronique provient > d’un > > > > > > expéditeur > > > > > > > > > >> externe. > > > > > > > > > >> > > Ne cliquez sur aucun lien et n’ouvrez aucune pièce > > > jointe > > > > si > > > > > > > vous > > > > > > > > ne > > > > > > > > > >> > pouvez > > > > > > > > > >> > > pas confirmer l’identité de l’expéditeur et si vous > > > n’êtes > > > > > pas > > > > > > > > > certain > > > > > > > > > >> > que > > > > > > > > > >> > > le contenu ne présente aucun risque. > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > Hi, > > > > > > > > > >> > > > > > > > > > > > >> > > I think that CI/CD and version pining are a little > two > > > > > > different > > > > > > > > > >> things > > > > > > > > > >> > > here. In a use cases with some critical systems > > > involved, > > > > > the > > > > > > > > > >> situation > > > > > > > > > >> > > when the Dag changes the version to the latest > without > > > > > > > possibility > > > > > > > > > to > > > > > > > > > >> > > determine when it will exactly happen (CI/CD will > have > > > > some > > > > > > > > > >> more-or-less > > > > > > > > > >> > > time to deploy the change, the same goes for Dag > > > Processor > > > > > > > parsing > > > > > > > > > >> time) > > > > > > > > > >> > is > > > > > > > > > >> > > rather hard to do and in some systems it can make > > change > > > > > > > > deployment > > > > > > > > > >> > harder > > > > > > > > > >> > > and less safe. Of course, the ideal solution would > be > > to > > > > > have > > > > > > > > proper > > > > > > > > > >> > > non-prod environment, which is fully representative > in > > > > > > > comparison > > > > > > > > to > > > > > > > > > >> > > production (in some cases exposing non-prod to prod > > > > > > > > > data/traffic/etc. > > > > > > > > > >> is, > > > > > > > > > >> > > just, not an option - e.g. security), but it is not > > > always > > > > > > > > possible > > > > > > > > > >> to do > > > > > > > > > >> > > due to various reasons like costs, licenses, space > > > and/or > > > > > > > vendors. > > > > > > > > > I'm > > > > > > > > > >> > > agreeing especially with point 5 of Piyush latest > > > message. > > > > > > > Having > > > > > > > > > >> above > > > > > > > > > >> > in > > > > > > > > > >> > > mind, I think that version pinning would be a nice > > > > addition > > > > > to > > > > > > > the > > > > > > > > > Dag > > > > > > > > > >> > > Versioning feature with an assumption that it is for > > > > > critical > > > > > > > > > Airflow > > > > > > > > > >> > Dags > > > > > > > > > >> > > when full control of the Dags version change time is > > > > > required > > > > > > > > (maybe > > > > > > > > > >> > there > > > > > > > > > >> > > is also another way to achieve that). > > > > > > > > > >> > > > > > > > > > > > >> > > P.S. In my opinion, what can be done in/around git, > > > should > > > > > be > > > > > > > done > > > > > > > > > >> there. > > > > > > > > > >> > > Recreation of CI/CD in any form inside of Airflow > > itself > > > > is > > > > > > > > > something > > > > > > > > > >> > which > > > > > > > > > >> > > should not be done. > > > > > > > > > >> > > ________________________________ > > > > > > > > > >> > > From: Oliveira, Niko <[email protected]> > > > > > > > > > >> > > Sent: 23 April 2026 01:50 > > > > > > > > > >> > > To: [email protected] <[email protected]> > > > > > > > > > >> > > Subject: Re: [DISCUSS] DAG Version Pinning for > > > Deployment > > > > > > Gating > > > > > > > > > >> > (Building > > > > > > > > > >> > > on AIP-63) > > > > > > > > > >> > > > > > > > > > > > >> > > Hey Piyush, > > > > > > > > > >> > > > > > > > > > > > >> > > Thanks for your reply, I do love how clearly it is > > > written > > > > > > and I > > > > > > > > see > > > > > > > > > >> > > exactly the problem you're trying to solve! > > > > > > > > > >> > > > > > > > > > > > >> > > I'm still just not convinced this needs to be done > in > > > > > Airflow, > > > > > > > at > > > > > > > > > >> least > > > > > > > > > >> > > not with a first class feature. As interesting as I > > > think > > > > > your > > > > > > > > > >> > microservice > > > > > > > > > >> > > analogy is, Airflow is not a microservice component, > > it > > > > is a > > > > > > > > (very, > > > > > > > > > >> very) > > > > > > > > > >> > > fancy cron scheduler. And I'm not sure the > complexity > > is > > > > > worth > > > > > > > the > > > > > > > > > use > > > > > > > > > >> > > case. Since any new code added to Airflow must be > > > > maintained > > > > > > by > > > > > > > > this > > > > > > > > > >> > > community and we must be cautious that any new > pieces > > > > serves > > > > > > > > enough > > > > > > > > > >> use > > > > > > > > > >> > > cases/users to make it worth it. > > > > > > > > > >> > > To me this should either be managed outside of an > > > > individual > > > > > > > > Airflow > > > > > > > > > >> > > environment e.g. you have an entirely separate > > > > > > staging/gamma/dev > > > > > > > > > >> Airflow > > > > > > > > > >> > > environment, which is exposed to some level of > > > production > > > > > > > traffic > > > > > > > > > (to > > > > > > > > > >> > > borrow your microservice analogy) until it can > > graduate > > > to > > > > > the > > > > > > > > > >> production > > > > > > > > > >> > > environment. And if you really need on the fly > > toggling > > > > of a > > > > > > > > > version, > > > > > > > > > >> as > > > > > > > > > >> > > you say, Airflow does this quite responsively, if > you > > > > > deploy a > > > > > > > new > > > > > > > > > >> > version > > > > > > > > > >> > > of your dags it will parse and start using that new > > > > version > > > > > > > > > >> immediately > > > > > > > > > >> > > (the problem you're trying to solve can be a benefit > > > > here). > > > > > > You > > > > > > > > can > > > > > > > > > >> even > > > > > > > > > >> > > have multiple versions of your dags deployed at once > > and > > > > use > > > > > > > > > >> > configuration > > > > > > > > > >> > > to control which dag directory Airflow reads from > (or > > > > > > > move/symlink > > > > > > > > > >> Dags > > > > > > > > > >> > in > > > > > > > > > >> > > and out of the Dags directory as needed from a known > > > good > > > > or > > > > > > > > pinned > > > > > > > > > >> > > source). Or use variables or some other parameter > > store > > > to > > > > > > > control > > > > > > > > > >> other > > > > > > > > > >> > > pieces of runtime behaviour inside the Dags > > themselves. > > > > > > Between > > > > > > > > > CI/CD, > > > > > > > > > >> > dev > > > > > > > > > >> > > ops and making use of existing Airflow primitives I > > > think > > > > > you > > > > > > > can > > > > > > > > > >> achieve > > > > > > > > > >> > > what you're looking for. > > > > > > > > > >> > > > > > > > > > > > >> > > But as always, this is open and community based > > > software, > > > > so > > > > > > I'm > > > > > > > > > >> happy to > > > > > > > > > >> > > disagree and commit if the rest of the community > > thinks > > > > this > > > > > > is > > > > > > > a > > > > > > > > > >> > valuable > > > > > > > > > >> > > feature :) > > > > > > > > > >> > > > > > > > > > > > >> > > Cheers, > > > > > > > > > >> > > Niko > > > > > > > > > >> > > ________________________________ > > > > > > > > > >> > > From: Piyush Maheshwari <[email protected]> > > > > > > > > > >> > > Sent: Tuesday, April 21, 2026 10:46 PM > > > > > > > > > >> > > To: [email protected] <[email protected]> > > > > > > > > > >> > > Subject: RE: [EXT] [DISCUSS] DAG Version Pinning for > > > > > > Deployment > > > > > > > > > Gating > > > > > > > > > >> > > (Building on AIP-63) > > > > > > > > > >> > > > > > > > > > > > >> > > CAUTION: This email originated from outside of the > > > > > > organization. > > > > > > > > Do > > > > > > > > > >> not > > > > > > > > > >> > > click links or open attachments unless you can > confirm > > > the > > > > > > > sender > > > > > > > > > and > > > > > > > > > >> > know > > > > > > > > > >> > > the content is safe. > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > AVERTISSEMENT: Ce courrier électronique provient > d’un > > > > > > expéditeur > > > > > > > > > >> externe. > > > > > > > > > >> > > Ne cliquez sur aucun lien et n’ouvrez aucune pièce > > > jointe > > > > si > > > > > > > vous > > > > > > > > ne > > > > > > > > > >> > pouvez > > > > > > > > > >> > > pas confirmer l’identité de l’expéditeur et si vous > > > n’êtes > > > > > pas > > > > > > > > > certain > > > > > > > > > >> > que > > > > > > > > > >> > > le contenu ne présente aucun risque. > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > Hi Ephraim, Jarek, Jens, and Niko, > > > > > > > > > >> > > > > > > > > > > > >> > > Thank you for the candid feedback. I want to > clarify a > > > few > > > > > > > things, > > > > > > > > > as > > > > > > > > > >> I > > > > > > > > > >> > > completely agree with Jens and Niko that "testing in > > > > > > production" > > > > > > > > is > > > > > > > > > an > > > > > > > > > >> > > anti-pattern. That is absolutely not the intention > > here. > > > > > > > > > >> > > > > > > > > > > > >> > > 1. I view this as bringing standard > microservice-like > > > > > > deployment > > > > > > > > > >> maturity > > > > > > > > > >> > > to DAGs. > > > > > > > > > >> > > Before service deployments in our org, code is > tested > > > > > locally, > > > > > > > in > > > > > > > > a > > > > > > > > > >> dev > > > > > > > > > >> > > environment, and via strict unit/e2e integration > tests > > > > > before > > > > > > it > > > > > > > > > ever > > > > > > > > > >> > makes > > > > > > > > > >> > > it to main. But even after merging and passing those > > CI > > > > > > > pipelines, > > > > > > > > > we > > > > > > > > > >> > still > > > > > > > > > >> > > use load tests, pre-prod soak times, shadow traffic, > > and > > > > > gated > > > > > > > > > >> production > > > > > > > > > >> > > rollouts with automated rollback triggers. Having > > > > deployment > > > > > > > gates > > > > > > > > > for > > > > > > > > > >> > the > > > > > > > > > >> > > production environment doesn't mean the pre-merge > > checks > > > > > > weren't > > > > > > > > > >> strict > > > > > > > > > >> > or > > > > > > > > > >> > > that the change wasn't tested beforehand -- it just > > > allows > > > > > us > > > > > > to > > > > > > > > > place > > > > > > > > > >> > > additional safety gates for the code to take effect, > > > > exactly > > > > > > > like > > > > > > > > in > > > > > > > > > >> the > > > > > > > > > >> > > service world. > > > > > > > > > >> > > > > > > > > > > > >> > > 2. The core issue we are trying to solve is that > > Airflow > > > > > > > currently > > > > > > > > > >> > > inseparably links Code Distribution (a file arriving > > on > > > > the > > > > > > > > > >> dag-processor > > > > > > > > > >> > > and being parsed) with Release Activation (the > > scheduler > > > > > > > executing > > > > > > > > > >> that > > > > > > > > > >> > > code). > > > > > > > > > >> > > To extend the microservices analogy, I can think of > > the > > > > DAG > > > > > > > > > processor > > > > > > > > > >> > > parsing all files as "building the artifact(s)," > while > > > the > > > > > > > > scheduler > > > > > > > > > >> and > > > > > > > > > >> > > executor acting on the DAG versions created > thereafter > > > as > > > > > > > > > "deploying" > > > > > > > > > >> or > > > > > > > > > >> > > running the changed code. > > > > > > > > > >> > > We simply want to decouple the build from the > > > deployment. > > > > > This > > > > > > > > does > > > > > > > > > >> not > > > > > > > > > >> > > mean that the code arriving on the dag-processor > will > > be > > > > > > tested > > > > > > > > for > > > > > > > > > >> the > > > > > > > > > >> > > first time straight in production. It should've > > already > > > > > > passed a > > > > > > > > set > > > > > > > > > >> of > > > > > > > > > >> > > checks in the CI pipeline. > > > > > > > > > >> > > > > > > > > > > > >> > > 3. It is also worth calling out that Airflow already > > > > > supports > > > > > > > this > > > > > > > > > >> > > decoupled behavior at the run level for task re-runs > > and > > > > > > > > > mid-execution > > > > > > > > > >> > DAG > > > > > > > > > >> > > version bumps (by pinning the version for the rest > of > > > the > > > > > > > > execution > > > > > > > > > or > > > > > > > > > >> > the > > > > > > > > > >> > > rerun). We are simply trying to expose this existing > > > > > > capability > > > > > > > at > > > > > > > > > the > > > > > > > > > >> > DAG > > > > > > > > > >> > > level so users can govern which version new > scheduled > > > runs > > > > > are > > > > > > > > > created > > > > > > > > > >> > > with. > > > > > > > > > >> > > > > > > > > > > > >> > > 4. I also agree that Airflow itself should not be > > aware > > > of > > > > > our > > > > > > > > CI/CD > > > > > > > > > >> > > pipeline, nor would it manage the deployment > > > orchestration > > > > > or > > > > > > > > > testing. > > > > > > > > > >> > > For our requirements, I just need Airflow to expose > > APIs > > > > to > > > > > > > deploy > > > > > > > > > >> (pin) > > > > > > > > > >> > a > > > > > > > > > >> > > DAG version, and to remove the pin (to > restore/enable > > > the > > > > > > > default > > > > > > > > > >> > > "auto-deploy latest" behavior). > > > > > > > > > >> > > Beyond that, we intend to use an external release > > > > > orchestrator > > > > > > > > that > > > > > > > > > >> can > > > > > > > > > >> > > explicitly tell Airflow when a parsed version is > > > actually > > > > > > > allowed > > > > > > > > to > > > > > > > > > >> run. > > > > > > > > > >> > > Until that API call is made, the previously pinned > > > version > > > > > > > remains > > > > > > > > > >> > active. > > > > > > > > > >> > > This ensures we don't introduce assumptions or > > awareness > > > > of > > > > > > the > > > > > > > > > >> presence > > > > > > > > > >> > of > > > > > > > > > >> > > any external gating mechanisms to Airflow. > > > > > > > > > >> > > Also note that the intention is to keep the default > > > > > > auto-deploy > > > > > > > > > >> behavior > > > > > > > > > >> > > unless a user (or a system on their behalf) > explicitly > > > > asks > > > > > > > > Airflow > > > > > > > > > to > > > > > > > > > >> > pin > > > > > > > > > >> > > a DAG to a specific version. > > > > > > > > > >> > > > > > > > > > > > >> > > 5. Most importantly, this feature provides an > incident > > > > > > response > > > > > > > > > >> > "rollback" > > > > > > > > > >> > > behavior. If a bad DAG version slips through CI/CD > > into > > > > > > > > production, > > > > > > > > > >> > either > > > > > > > > > >> > > an on-call engineer or a rollback-trigger > > > > (airflow-external) > > > > > > can > > > > > > > > > >> > instantly > > > > > > > > > >> > > roll back to the previous pinned version via the > > API/UI > > > to > > > > > > > > mitigate. > > > > > > > > > >> > > Without this, users have to revert the code in Git > and > > > > wait > > > > > > for > > > > > > > > the > > > > > > > > > >> > entire > > > > > > > > > >> > > CI/CD pipeline and file-sync process to run, which > is > > > > often > > > > > > too > > > > > > > > slow > > > > > > > > > >> > during > > > > > > > > > >> > > an outage. > > > > > > > > > >> > > > > > > > > > > > >> > > 6. Jarek - You are right, database schema changes > can > > be > > > > > > > discussed > > > > > > > > > >> later. > > > > > > > > > >> > > My intention was only to share a very brief summary > of > > > > how I > > > > > > > > deemed > > > > > > > > > >> it to > > > > > > > > > >> > > be technically feasible for early feedback. I did > > > briefly > > > > > > share > > > > > > > > the > > > > > > > > > >> > > high-level use cases ("Safe Deployment Gating" and > > > > "Instant > > > > > > > > > >> Rollbacks") > > > > > > > > > >> > in > > > > > > > > > >> > > the original mail, but I completely agree that > > aligning > > > on > > > > > the > > > > > > > UX > > > > > > > > > >> first > > > > > > > > > >> > > would be a good next step. > > > > > > > > > >> > > > > > > > > > > > >> > > If there are no major remaining concerns after this > > > > > response, > > > > > > I > > > > > > > > can > > > > > > > > > >> draft > > > > > > > > > >> > > and share an AIP to detail the UX, followed by a > > > > high-level > > > > > > > > > proposal, > > > > > > > > > >> > > caveats and next steps. > > > > > > > > > >> > > > > > > > > > > > >> > > Thanks for your time. > > > > > > > > > >> > > Regards, > > > > > > > > > >> > > Piyush > > > > > > > > > >> > > > > > > > > > > > >> > > On Tue, Apr 21, 2026 at 5:59 PM Oliveira, Niko < > > > > > > > > [email protected] > > > > > > > > > > > > > > > > > > > >> > > wrote: > > > > > > > > > >> > > > > > > > > > > > >> > > > I am with Jens on this one. I think we're > > complicating > > > > > > Airflow > > > > > > > > to > > > > > > > > > >> get > > > > > > > > > >> > > > around a bad practice. If stability of your Dags > is > > > > > critical > > > > > > > and > > > > > > > > > >> they > > > > > > > > > >> > are > > > > > > > > > >> > > > highly versioned then I think as Jens suggested > > > running > > > > > them > > > > > > > > > >> through a > > > > > > > > > >> > > > pipeline that first deploys them to a dev or gamma > > > > > > environment > > > > > > > > > which > > > > > > > > > >> > > > verifies that quality of the Dags is what you > > expect. > > > If > > > > > > > > something > > > > > > > > > >> > slips > > > > > > > > > >> > > > through, then it's just normal software practices > of > > > > > either > > > > > > > > > >> reverting > > > > > > > > > >> > and > > > > > > > > > >> > > > rolling back or rolling forward with a fix pushed > > > > through > > > > > > the > > > > > > > > > >> > pipeline. I > > > > > > > > > >> > > > don't think Airflow should be aware of that > process > > or > > > > > > > > opinionated > > > > > > > > > >> > about > > > > > > > > > >> > > it. > > > > > > > > > >> > > > > > > > > > > > > >> > > > Cheers, > > > > > > > > > >> > > > Niko > > > > > > > > > >> > > > ------------------------------ > > > > > > > > > >> > > > *From:* Jens Scheffler <[email protected]> > > > > > > > > > >> > > > *Sent:* Monday, April 20, 2026 11:17 AM > > > > > > > > > >> > > > *To:* [email protected] < > > [email protected]> > > > > > > > > > >> > > > *Subject:* RE: [EXT] [DISCUSS] DAG Version Pinning > > for > > > > > > > > Deployment > > > > > > > > > >> > Gating > > > > > > > > > >> > > > (Building on AIP-63) > > > > > > > > > >> > > > > > > > > > > > > >> > > > CAUTION: This email originated from outside of the > > > > > > > organization. > > > > > > > > > Do > > > > > > > > > >> not > > > > > > > > > >> > > > click links or open attachments unless you can > > confirm > > > > the > > > > > > > > sender > > > > > > > > > >> and > > > > > > > > > >> > > know > > > > > > > > > >> > > > the content is safe. > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > AVERTISSEMENT: Ce courrier électronique provient > > d’un > > > > > > > expéditeur > > > > > > > > > >> > externe. > > > > > > > > > >> > > > Ne cliquez sur aucun lien et n’ouvrez aucune pièce > > > > jointe > > > > > si > > > > > > > > vous > > > > > > > > > ne > > > > > > > > > >> > > pouvez > > > > > > > > > >> > > > pas confirmer l’identité de l’expéditeur et si > vous > > > > n’êtes > > > > > > pas > > > > > > > > > >> certain > > > > > > > > > >> > > que > > > > > > > > > >> > > > le contenu ne présente aucun risque. > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > Hi, > > > > > > > > > >> > > > > > > > > > > > > >> > > > I am still quite sceptical. Yes, if such pinning > is > > > > made, > > > > > > then > > > > > > > > per > > > > > > > > > >> Dag > > > > > > > > > >> > a > > > > > > > > > >> > > > change need to be possible via UI and API. But I > > still > > > > see > > > > > > it > > > > > > > as > > > > > > > > > >> > > > checken-and-egg - so you want to run a pinned > > version > > > > but > > > > > > then > > > > > > > > how > > > > > > > > > >> do > > > > > > > > > >> > > > you test the changes (w/o moving a version pin)? > > Then > > > > > again > > > > > > > some > > > > > > > > > >> test > > > > > > > > > >> > > > mode is needed or per run you need to make a "test > > > run" > > > > > with > > > > > > > > > another > > > > > > > > > >> > > > version. Smells a bit like mis-using a production > > > system > > > > > for > > > > > > > > > >> testing. > > > > > > > > > >> > > > > > > > > > > > > >> > > > On the other hand, yes if all Dags share the same > > Git > > > > repo > > > > > > > then > > > > > > > > > >> merging > > > > > > > > > >> > > > a branch to some other will switch all Dags at the > > > same > > > > > > time. > > > > > > > > > Still > > > > > > > > > >> you > > > > > > > > > >> > > > could utilize standard Git tools and cherry-pick > > > > > individual > > > > > > > > > changes > > > > > > > > > >> and > > > > > > > > > >> > > > no force to always make a full rollout. At least > 80% > > > > > > possible > > > > > > > > with > > > > > > > > > >> > > > standard CI/CD tools and Git. > > > > > > > > > >> > > > > > > > > > > > > >> > > > TLDR I see the danger that instead of a proper > CI/CD > > > and > > > > > > test > > > > > > > > > system > > > > > > > > > >> > > > such a feature might feel like you can easily test > > on > > > a > > > > > > > > production > > > > > > > > > >> > > > system. Effectively it would be needed allowing to > > > > start a > > > > > > Dag > > > > > > > > > with > > > > > > > > > >> any > > > > > > > > > >> > > > version to also be able to jump back as a > reversion. > > > > Even > > > > > > > > though, > > > > > > > > > >> yes, > > > > > > > > > >> > > > agree, all is technically possible. > > > > > > > > > >> > > > > > > > > > > > > >> > > > Jens > > > > > > > > > >> > > > > > > > > > > > > >> > > > On 20.04.26 16:40, Jarek Potiuk wrote: > > > > > > > > > >> > > > > +1 to what Ephraim wrote. I think that was a > > natural > > > > > next > > > > > > > step > > > > > > > > > we > > > > > > > > > >> > > > > discussed, but it needs significant refinement, > > > > starting > > > > > > > with > > > > > > > > > the > > > > > > > > > >> > > actual > > > > > > > > > >> > > > > use cases it should serve and the UX for user > > > > > > interaction. I > > > > > > > > > think > > > > > > > > > >> > > > related > > > > > > > > > >> > > > > database changes are pretty secondary. Use cases > > > cover > > > > > > runs, > > > > > > > > > >> re-runs, > > > > > > > > > >> > > > > backfills, CI testing, rollbacks, etc. Following > > the > > > > > > > > > >> "documentation > > > > > > > > > >> > > > first" > > > > > > > > > >> > > > > approach discussed in separate thread, > describing > > > the > > > > > > > context > > > > > > > > > and > > > > > > > > > >> > > > intention > > > > > > > > > >> > > > > of what we want to achieve is much more > important > > > than > > > > > DB > > > > > > > > schema > > > > > > > > > >> > > changes. > > > > > > > > > >> > > > > Once we know which use cases we want to serve, > the > > > DB > > > > > > schema > > > > > > > > > >> changes > > > > > > > > > >> > > and > > > > > > > > > >> > > > > other related items will emerge naturally. > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > On Mon, Apr 20, 2026 at 3:15 PM Ephraim > Anierobi < > > > > > > > > > >> > > > [email protected]> > > > > > > > > > >> > > > > wrote: > > > > > > > > > >> > > > > > > > > > > > > > >> > > > >> Hi Piyush, thanks for starting this discussion. > > > > > > > > > >> > > > >> > > > > > > > > > >> > > > >> I like the proposal. We can introduce an active > > > > > execution > > > > > > > > > version > > > > > > > > > >> > for > > > > > > > > > >> > > > >> "versioned bundles" and make scheduler/API > > resolve > > > > > > through > > > > > > > > it. > > > > > > > > > >> The > > > > > > > > > >> > > hard > > > > > > > > > >> > > > >> part of this is making airflow able to > > distinguish > > > > the > > > > > > > latest > > > > > > > > > >> parsed > > > > > > > > > >> > > > >> dagmodel's metadata from active scheduling > > > metadata. > > > > I > > > > > > will > > > > > > > > > >> suggest > > > > > > > > > >> > > you > > > > > > > > > >> > > > >> draft this in a google docs and share for > further > > > > > > > > discussions. > > > > > > > > > >> > > > >> > > > > > > > > > >> > > > >> Regards > > > > > > > > > >> > > > >> - Ephraim > > > > > > > > > >> > > > >> > > > > > > > > > >> > > > >> On Mon, 20 Apr 2026 at 01:31, Piyush > Maheshwari < > > > > > > > > > >> > > [email protected]> > > > > > > > > > >> > > > >> wrote: > > > > > > > > > >> > > > >> > > > > > > > > > >> > > > >>> Thanks for sharing your thoughts Jens. > > > > > > > > > >> > > > >>> > > > > > > > > > >> > > > >>>> be able to test it? … a Q&A/Testing > environment > > > to > > > > be > > > > > > > able > > > > > > > > to > > > > > > > > > >> > > sign-off > > > > > > > > > >> > > > >>> changes. > > > > > > > > > >> > > > >>> Yes, we’ve have built an isolated airflow > > > > environment > > > > > to > > > > > > > run > > > > > > > > > >> > > regression > > > > > > > > > >> > > > >>> checks before promoting to production. > > > > > > > > > >> > > > >>> > > > > > > > > > >> > > > >>> As you suggested, we’re already running both > > > generic > > > > > and > > > > > > > > > >> DAG-custom > > > > > > > > > >> > > > >> static > > > > > > > > > >> > > > >>> checks in a CI job as a required step to merge > > to > > > > the > > > > > > main > > > > > > > > > >> branch. > > > > > > > > > >> > > > >>> > > > > > > > > > >> > > > >>>> But then the "main" branch might be best > suited > > > if > > > > > > > > > >> > > > >>> implemented on the test system > > > > > > > > > >> > > > >>> In this case, problematic commits on “main” > can > > > > choke > > > > > > > other > > > > > > > > > >> > unrelated > > > > > > > > > >> > > > >>> changes. > > > > > > > > > >> > > > >>> So the other option would be to revert the > > > > problematic > > > > > > > > commits > > > > > > > > > >> and > > > > > > > > > >> > > > deploy > > > > > > > > > >> > > > >>> forward. > > > > > > > > > >> > > > >>> > > > > > > > > > >> > > > >>> However, a key limitation with this approach > > that > > > > > > remains > > > > > > > is > > > > > > > > > >> that a > > > > > > > > > >> > > > >> commit > > > > > > > > > >> > > > >>> affecting multiple DAGs goes live for either > all > > > > DAGs > > > > > or > > > > > > > > none. > > > > > > > > > >> > > > >>> > > > > > > > > > >> > > > >>> Second important feature we get with this is > > > instant > > > > > > > > DAG-level > > > > > > > > > >> > > rollback > > > > > > > > > >> > > > >>> without waiting for a revert commit to merge > and > > > be > > > > > > picked > > > > > > > > by > > > > > > > > > >> > > airflow. > > > > > > > > > >> > > > >>> > > > > > > > > > >> > > > >>> I think DAG-level version pinning can also > > unlock > > > a > > > > > lot > > > > > > of > > > > > > > > > >> > > flexibility > > > > > > > > > >> > > > >> for > > > > > > > > > >> > > > >>> deployments including tiered rollouts, > > > auto-rollback > > > > > > > > triggers, > > > > > > > > > >> > timed > > > > > > > > > >> > > > >>> deployment windows and so on. > > > > > > > > > >> > > > >>> > > > > > > > > > >> > > > >>> Looking forward to hear your thoughts. > > > > > > > > > >> > > > >>> Regards, > > > > > > > > > >> > > > >>> Piyush > > > > > > > > > >> > > > >>> > > > > > > > > > >> > > > >>> On Sun, 19 Apr 2026 at 3:12 PM, Jens > Scheffler < > > > > > > > > > >> > [email protected]> > > > > > > > > > >> > > > >>> wrote: > > > > > > > > > >> > > > >>> > > > > > > > > > >> > > > >>>> Thanks Piyush for dropping the discussion! > > > > > > > > > >> > > > >>>> > > > > > > > > > >> > > > >>>> I think in general QA processes are important > > > and a > > > > > > valid > > > > > > > > use > > > > > > > > > >> > case. > > > > > > > > > >> > > So > > > > > > > > > >> > > > >> a > > > > > > > > > >> > > > >>>> kind of pinning Dag versions really is > > important. > > > > > > > > > >> > > > >>>> > > > > > > > > > >> > > > >>>> Thinking about it, if you pin the version ... > > how > > > > > would > > > > > > > you > > > > > > > > > >> then > > > > > > > > > >> > be > > > > > > > > > >> > > > >> able > > > > > > > > > >> > > > >>>> to test it? I assume you would need (and > should > > > > have > > > > > or > > > > > > > > > invest > > > > > > > > > >> > > into) a > > > > > > > > > >> > > > >>>> Q&A/Testing environment to be able to > sign-off > > > > > changes. > > > > > > > > Both > > > > > > > > > in > > > > > > > > > >> > > > >>>> infrastructure but also for Dag changes. > > > > > > > > > >> > > > >>>> > > > > > > > > > >> > > > >>>> If you are changing Dags first of all static > > > checks > > > > > on > > > > > > > Dag > > > > > > > > > code > > > > > > > > > >> > are > > > > > > > > > >> > > > >> very > > > > > > > > > >> > > > >>>> much proposed as well as you can have tests > > > > > implemented > > > > > > > and > > > > > > > > > >> test > > > > > > > > > >> > > your > > > > > > > > > >> > > > >>>> Dags and logic. Similar like software a CI/CD > > > > system > > > > > > will > > > > > > > > be > > > > > > > > > a > > > > > > > > > >> > good > > > > > > > > > >> > > > >>>> setup. Alongside Dag changes also have > logical > > > > > changes > > > > > > > that > > > > > > > > > >> mostly > > > > > > > > > >> > > can > > > > > > > > > >> > > > >>>> only be tested in a live system and not as > > static > > > > > > checks. > > > > > > > > > >> > > > >>>> > > > > > > > > > >> > > > >>>> Have you considered using Git and a set of > > > branches > > > > > for > > > > > > > > > >> > implementing > > > > > > > > > >> > > > >>>> such staging? E.g. you have a git repo and > you > > > plan > > > > > to > > > > > > > make > > > > > > > > > >> > changes. > > > > > > > > > >> > > > >>>> Then you would open a PR for the change and > > merge > > > > it > > > > > to > > > > > > > the > > > > > > > > > >> "main" > > > > > > > > > >> > > > >>>> branch - and there in your CI/CD you can > check > > > all > > > > > > sorts > > > > > > > of > > > > > > > > > >> static > > > > > > > > > >> > > > >>>> checks and tests. But then the "main" branch > > > might > > > > be > > > > > > > best > > > > > > > > > >> suited > > > > > > > > > >> > if > > > > > > > > > >> > > > >>>> implemented on the test system. Once you > > validate > > > > the > > > > > > > > changes > > > > > > > > > >> > > > >> end-to-end > > > > > > > > > >> > > > >>>> you could make another PR for example to a > > "prod" > > > > > > branch. > > > > > > > > And > > > > > > > > > >> if > > > > > > > > > >> > > your > > > > > > > > > >> > > > >>>> production system is only pulling Dags from > the > > > > > "prod" > > > > > > > > branch > > > > > > > > > >> then > > > > > > > > > >> > > you > > > > > > > > > >> > > > >>>> can have this merging strategy as a staging > > > setup. > > > > > > > > > >> > > > >>>> > > > > > > > > > >> > > > >>>> Would this resolve your PING problem? Or > which > > > > other > > > > > > > detail > > > > > > > > > in > > > > > > > > > >> the > > > > > > > > > >> > > use > > > > > > > > > >> > > > >>>> case would require a PIN on top of a staging > > > > > strategy? > > > > > > > > > >> > > > >>>> > > > > > > > > > >> > > > >>>> Jens > > > > > > > > > >> > > > >>>> > > > > > > > > > >> > > > >>>> P.S.: Have enabled your confluence account > > after > > > it > > > > > was > > > > > > > > > >> created in > > > > > > > > > >> > > > >> order > > > > > > > > > >> > > > >>>> to write to Confluence, sorry, typical > pitfall > > > > after > > > > > > > > account > > > > > > > > > >> > > creation > > > > > > > > > >> > > > >>>> permissions were not set. Now it should work. > > Let > > > > me > > > > > > know > > > > > > > > if > > > > > > > > > >> not. > > > > > > > > > >> > > > >>>> > > > > > > > > > >> > > > >>>> On 19.04.26 01:40, Piyush Maheshwari wrote: > > > > > > > > > >> > > > >>>>> Hi everyone, > > > > > > > > > >> > > > >>>>> I'm a new contributor to Airflow. I'd like > to > > > > > propose > > > > > > a > > > > > > > > new > > > > > > > > > >> > feature > > > > > > > > > >> > > > >> for > > > > > > > > > >> > > > >>>> Airflow: DAG Version Pinning. > > > > > > > > > >> > > > >>>>> Building on the foundation introduced by > > AIP-63: > > > > DAG > > > > > > > > > >> Versioning ( > > > > > > > > > >> > > > >> > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://urldefense.com/v3/__https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-63*3A*DAG*Versioning__;JSsr!!Ci6f514n9QsL8ck!l3ZKTOw996h9qu4NR0VT4ouUryUdk_HmXUAVPbwCHwPwn0N2CCptVdx95-V0BoRFjws9huE_1Vy-THL8jw$ > > > > > > > > > >> > > > >>> ), > > > > > > > > > >> > > > >>>> this proposal aims to extend Airflow's > > > capabilities > > > > > to > > > > > > > > > support > > > > > > > > > >> > true > > > > > > > > > >> > > > >>>> continuous deployment (CD) gating and safer > > > release > > > > > > > cycles. > > > > > > > > > >> > > > >>>>> The Problem & Use Cases > > > > > > > > > >> > > > >>>>> Currently, the scheduler always creates > > DagRuns > > > > > using > > > > > > > the > > > > > > > > > >> latest > > > > > > > > > >> > > > >> parsed > > > > > > > > > >> > > > >>>> DagVersion. This means that the updated DAG > > code > > > is > > > > > > > > deployed > > > > > > > > > >> > (takes > > > > > > > > > >> > > > >>> effect) > > > > > > > > > >> > > > >>>> right after the dag-processor processes it. > > While > > > > > this > > > > > > is > > > > > > > > > great > > > > > > > > > >> > for > > > > > > > > > >> > > > >> rapid > > > > > > > > > >> > > > >>>> development, teams running business-critical > > > > > pipelines > > > > > > > > often > > > > > > > > > >> need > > > > > > > > > >> > > > >>> stricter > > > > > > > > > >> > > > >>>> deployment mechanisms. Specifically: > > > > > > > > > >> > > > >>>>> * > > > > > > > > > >> > > > >>>>> Safe Deployment Gating: The ability to pin a > > DAG > > > > to > > > > > > its > > > > > > > > last > > > > > > > > > >> > known > > > > > > > > > >> > > > >>>> stable version while new code is parsed in > the > > > > > > > background. > > > > > > > > > This > > > > > > > > > >> > > allows > > > > > > > > > >> > > > >>> the > > > > > > > > > >> > > > >>>> new version to be held back until it passes > > > > automated > > > > > > > > > >> regression > > > > > > > > > >> > > tests > > > > > > > > > >> > > > >> or > > > > > > > > > >> > > > >>>> receives explicit manual approval. > > > > > > > > > >> > > > >>>>> * > > > > > > > > > >> > > > >>>>> Instant Rollbacks: If an issue is detected > in > > a > > > > > newly > > > > > > > > > promoted > > > > > > > > > >> > DAG > > > > > > > > > >> > > > >>>> version, users need the capability to > instantly > > > > roll > > > > > > back > > > > > > > > to > > > > > > > > > a > > > > > > > > > >> > > > previous > > > > > > > > > >> > > > >>>> version via the UI/API, without having to > > revert > > > > the > > > > > > > > > underlying > > > > > > > > > >> > code > > > > > > > > > >> > > > >> and > > > > > > > > > >> > > > >>>> wait for the repository sync and DAG > processing > > > > > cycle. > > > > > > > > > >> > > > >>>>> High-Level Proposed Solution > > > > > > > > > >> > > > >>>>> Introduce an optional active_dag_version_id > to > > > the > > > > > > > > DagModel. > > > > > > > > > >> This > > > > > > > > > >> > > > >> field > > > > > > > > > >> > > > >>>> can be used to pin a DAG version for > scheduling > > > and > > > > > > > > > execution, > > > > > > > > > >> > while > > > > > > > > > >> > > > >> the > > > > > > > > > >> > > > >>>> dag-processor can continue to parse and > > register > > > > > newer > > > > > > > DAG > > > > > > > > > >> > versions. > > > > > > > > > >> > > > >>>>> * > > > > > > > > > >> > > > >>>>> When this pin is set, the scheduler and API > > will > > > > > > respect > > > > > > > > the > > > > > > > > > >> > pinned > > > > > > > > > >> > > > >>>> version for creating runs and executing > tasks, > > > > > > separating > > > > > > > > the > > > > > > > > > >> > > parsing > > > > > > > > > >> > > > >> of > > > > > > > > > >> > > > >>>> new code from the execution of new code. > > > > > > > > > >> > > > >>>>> * > > > > > > > > > >> > > > >>>>> If the pin is NULL, the system defaults to > the > > > > > current > > > > > > > > > >> behavior > > > > > > > > > >> > > > >> (always > > > > > > > > > >> > > > >>>> executing the latest parsed version). This > way, > > > we > > > > > can > > > > > > > > > maintain > > > > > > > > > >> > > > >> complete > > > > > > > > > >> > > > >>>> backwards compatibility. > > > > > > > > > >> > > > >>>>> I have put together some detailed notes > > covering > > > > the > > > > > > > data > > > > > > > > > >> model > > > > > > > > > >> > > > >>> changes, > > > > > > > > > >> > > > >>>> database migrations, and edge cases with this > > > > > approach. > > > > > > > If > > > > > > > > > >> there > > > > > > > > > >> > is > > > > > > > > > >> > > > >>> general > > > > > > > > > >> > > > >>>> alignment that this fits the vision for > > Airflow, > > > I > > > > > > would > > > > > > > > like > > > > > > > > > >> to > > > > > > > > > >> > > take > > > > > > > > > >> > > > >>> this > > > > > > > > > >> > > > >>>> proposal through the formal AIP review > process. > > > > > > > > > >> > > > >>>>> But I would love to get the community's > > feedback > > > > on > > > > > > the > > > > > > > > > >> feature > > > > > > > > > >> > and > > > > > > > > > >> > > > >> the > > > > > > > > > >> > > > >>>> high-level approach. > > > > > > > > > >> > > > >>>>> I'll also need someone to grant me access to > > > > create > > > > > > > > content > > > > > > > > > on > > > > > > > > > >> > the > > > > > > > > > >> > > > >>>> Airflow Confluence wiki. > > > > > > > > > >> > > > >>>>> Thanks for your time! > > > > > > > > > >> > > > >>>>> Regards, > > > > > > > > > >> > > > >>>>> Piyush > > > > > > > > > >> > > > >>>>> > > > > > > > > > >> > > > >>>> > > > > > > > > > >> > > > > > > > > > > > > > > > > > --------------------------------------------------------------------- > > > > > > > > > >> > > > >>>> To unsubscribe, e-mail: > > > > > > > [email protected] > > > > > > > > > >> > > > >>>> For additional commands, e-mail: > > > > > > > > [email protected] > > > > > > > > > >> > > > >>>> > > > > > > > > > >> > > > >>>> > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > --------------------------------------------------------------------- > > > > > > > > > >> > > > To unsubscribe, e-mail: > > > > > [email protected] > > > > > > > > > >> > > > For additional commands, e-mail: > > > > > > [email protected] > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
