Hi everyone
Gentle reminder to review the AIP
https://cwiki.apache.org/confluence/spaces/AIRFLOW/pages/421958312/AIP-109+DAG+Version+Pinning
.

Thank you.

On Thu, 13 Aug 2026 at 4:05 PM, Piyush Maheshwari <
[email protected]> wrote:

> Hi everyone,
> I'd like to request the community's review of the AIP
> https://cwiki.apache.org/confluence/spaces/AIRFLOW/pages/421958312/AIP-109+DAG+Version+Pinning
> .
> If there are no differing opinions I'd like to take it to a vote soon.
>
> Also, thank you Ephraim for the re-review. I've addressed all comments
> from the latest review and updated the AIP accordingly.
>
> Thanks and regards,
> Piyush
>
> On Fri, Aug 7, 2026 at 4:10 PM Ephraim Anierobi <
> [email protected]> wrote:
>
>> Hi Piyush,
>>
>> Thank you for this summary. It is accurate, and it matches where we
>> landed.
>> Thank you also for the many rounds of changes. The document is much
>> clearer
>> now than when we started.
>>
>> I want to state my position plainly for the thread. My main concern was
>> that a pin on a DagVersion could not hold, because the row could change
>> under it and the bundle pointer could move. The pin guard fixes both. With
>> that, plus the retention rule, the stale DAG change, the purity scope, and
>> the divergence warning, I no longer object to the pin target. The AIP now
>> says what it delivers and what it does not. That was my real goal.
>>
>> One thing I want to be clear about. My review was about whether the design
>> holds, not about whether the feature is worth building. Those are two
>> different questions, and the second one belongs to the community.
>>
>> The feature adds pin awareness in several core places: the version write
>> path, db clean, the scheduler's eligibility query for import errors and
>> stale DAGs, DagModel and asset field resolution in the DAG processor, and
>> a
>> synchronous update when a pin changes. Each change is small on its own.
>> Together they mean these paths must consider pins from now on.
>>
>> What it gives back is per-DAG control over when a version takes effect,
>> for
>> pure DAGs that come from versioned bundles. That is real and useful for
>> the
>> teams who need it. Whether it is worth the added complexity in core paths
>> is a judgment call, and voters should make it with both sides stated. I am
>> not voting against it. I only do not want my resolved concerns to be read
>> as a view on that question.
>>
>> I have left my remaining comments on the document itself. They are small
>> and none of them changes the design, but I think they should be settled
>> before the vote.
>>
>> On the vote, I agree with starting it again instead of continuing the old
>> thread. The document has changed a lot. A fresh call, with a short note on
>> what changed, would be fair to everyone who already voted.
>>
>> Thanks again for your patience through all of this. The proposal is in
>> much
>> better shape.
>>
>> Regards,
>> Ephraim
>>
>> On Wed, 5 Aug 2026 at 10:55, Piyush Maheshwari <
>> [email protected]>
>> wrote:
>>
>> > Hi everyone,
>> >
>> > I'd like to thank Ephraim for the depth of this review. Following the
>> > previous email, we had a series of detailed follow-up discussions, and I
>> > believe we have converged. This email summarizes where we landed,
>> answers
>> > the concerns from the previous email for the benefit of the thread, and
>> > describes the updates I have now made to the AIP. I would like the
>> > community's review of the updated AIP (
>> >
>> >
>> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-109+DAG+Version+Pinning
>> > )
>> >
>> > == Precise scope of the proposal ==
>> >
>> > The discussion sharpened what the AIP promises, and it now states this
>> > explicitly:
>> >
>> > - The goal is deterministic, per-DAG control over when a new version of
>> a
>> > DAG takes effect, plus an instant, API-driven rollback to a prior
>> version,
>> > without affecting other DAGs sharing the same bundle.
>> >
>> > - A pin fixes the serialized DAG the scheduler enforces (task set,
>> edges,
>> > schedule, display fields) and the bundle (name and version) the worker
>> > checks out. It provides graph and code-level determinism, not
>> > behavior-level determinism, since a task that reads live external state
>> at
>> > execution time still does so under a pin.
>> >
>> > - Pinning guarantees are stated for DAGs that are "pure", meaning a
>> given
>> > bundle (name and version) always produces the same serialized DAG. The
>> AIP
>> > now defines this purity condition up front.
>> >
>> > - For impure DAGs, different pinning guarantees are now documented for
>> > impure DAGs. A divergence between the pinned serialized DAG's snapshot
>> and
>> > the worker's fresh parse for execution-time fields is silent. The AIP
>> now
>> > explicitly documents this as "silent divergence," a limitation, and adds
>> > detection for it.
>> >
>> > == Answers to the five concerns from the previous email ==
>> >
>> > 1. "The latest DagVersion is mutable, so a pin does not freeze content."
>> > Addressed by introducing a pin guard in the DAG Processor's
>> > version-recording path. A pinned DagVersion is treated as referenced, so
>> > its serialized content is never overwritten in place; a content change
>> > creates a new DagVersion instead (the same path that already runs when
>> task
>> > instances reference the row). Note that the DAG processor only mutates
>> the
>> > latest DagVersion row, so a pin on an older version is already immutable
>> > today; the guard changes behavior only when the pinned version is also
>> the
>> > latest.
>> >
>> > 2. "A pin would not pin the source code, because bundle_version is
>> > refreshed in place on hash-identical parses."
>> > Addressed by the same guard, which stops the in-place
>> > bundle_version/version_data refresh from applying to a pinned row. When
>> the
>> > bundle advances without a serialized-DAG change while the pinned
>> version is
>> > the latest, the processor instead creates one new DagVersion carrying
>> the
>> > new bundle_version, which becomes the unpinned latest; subsequent
>> no-change
>> > bundle advances keep updating that new latest in place exactly as today.
>> > This preserves the intent of the in-place refresh (the DAG's latest
>> version
>> > keeps tracking the bundle) while the pinned version's bundle reference
>> > stays frozen. Since new DagRuns will also carry the pinned bundle, and
>> > workers resolve their bundle checkout from the DagRun, the scheduler's
>> view
>> > and the worker's code stay on the same version. Note that this version
>> > creation, without a hash change, occurs only per pin advance, not once
>> per
>> > commit.
>> >
>> > 3. "Version identity is often accidental (impure DAGs)."
>> > Addressed by scoping (guarantees stated for pure DAGs, with the purity
>> > condition defined in the AIP) and by divergence detection, where a
>> > DagWarning is raised when a pinned DAG's fresh parse hash no longer
>> matches
>> > the pinned DagVersion's hash, naming the pinned version. The check is a
>> > no-op for pure DAGs.
>> >
>> > 4. "Cleanup can delete a pinned version and silently unpin via ON DELETE
>> > SET NULL."
>> > Addressed: airflow db clean will retain any DagVersion referenced by an
>> > active pin, the same way it retains task-instance-referenced versions.
>> >
>> > 5. "The deletion rule works against the main use case."
>> > Revised: a dag_id that disappears from the latest parse would still be
>> > marked stale on DagModel as today, but the scheduler's eligibility query
>> > will admit a stale DAG while it has an active pin, so it keeps running
>> on
>> > its pinned version (parallel to the import-error exception already in
>> the
>> > AIP). Actual removal requires an explicit unpin first, after which the
>> > normal stale-to-deleted path applies unchanged. This turns the defect
>> > scenario from a silent removal into an explicit act.
>> >
>> > == On pinning the bundle instead of the DagVersion ==
>> >
>> > We discussed this alternative at length, in two variants, and the AIP's
>> > "Alternatives" section now covers both:
>> >
>> > - A global bundle pin (keep a history of bundle versions and freeze the
>> > whole bundle on one) is simple and atomic, but it pins every DAG in the
>> > bundle at once. It cannot keep a subset of DAGs on an older version
>> while
>> > the rest advance, and per-DAG control is the primary use case, since a
>> > breaking change in a shared bundle typically affects only some of the
>> DAGs
>> > it emits. We see it as a potentially complementary, coarser-grained
>> control
>> > rather than a replacement.
>> >
>> > - A per-DAG pin keyed by the bundle (name and version) is not uniquely
>> > resolvable to a serialized DAG for impure DAGs, since the pair can map
>> to
>> > several dag_version rows. It therefore cannot be expressed as a foreign
>> key
>> > to a specific version, and the latest matching row can resolve
>> differently
>> > across invocations, which is behavior drift while supposedly pinned. It
>> is
>> > also unstable over time, because a dag_version's bundle_version is
>> > refreshed in place on hash-identical parses, so a previously valid
>> bundle
>> > version can stop resolving exactly when a rollback needs it. Freezing
>> that
>> > would require an immutable bundle-to-dag_version mapping, which amounts
>> to
>> > pinning the dag_version too and converges towards this proposal.
>> >
>> > == Operating models and rollback availability ==
>> >
>> > Ephraim raised an important point about rollback availability. An
>> on-demand
>> > pin can only roll back to versions Airflow already recorded, so a change
>> > that never altered the serialized DAG (e.g. one confined to an imported
>> > helper module) has no distinct version to return to unless it was
>> pinned at
>> > the time. The AIP now states this honestly under a new "Usage Models and
>> > Rollback Availability" section, which describes two models the primitive
>> > serves:
>> >
>> > 1. On-demand pinning (incident response, planned cutovers): rollback
>> > targets are the recorded DAG versions. Changes that altered the
>> serialized
>> > DAG (and had task instances) are reachable; changes that did not are a
>> > stated limitation.
>> > 2. Standing pins driven by an external release orchestrator: every DAG
>> > always carries a pin and the orchestrator advances it per deployment.
>> Every
>> > pinned state remains an immutable rollback target, closing the gap
>> above,
>> > at the cost of operating the orchestrator. The consequences of the same
>> are
>> > also covered in the AIP.
>> >
>> > Airflow itself only ships the pin/unpin primitive; neither model is
>> > mandatory to use this feature.
>> >
>> > == Updated AIP and proposed timeline ==
>> >
>> > I have updated AIP-109 with all of the above:
>> >
>> >
>> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-109+DAG+Version+Pinning
>> >
>> > I would appreciate the community's review of the updated proposal. If
>> > multiple maintainers review it and if no major open questions or
>> comments
>> > remain by Wednesday, August 12, I plan to restart the vote phase.
>> >
>> > Thanks again to Ephraim for the multiple rounds of thorough review that
>> got
>> > the proposal to this shape.
>> >
>> > Regards,
>> > Piyush
>> >
>> > On Tue, Jul 28, 2026 at 3:47 PM Ephraim Anierobi <
>> > [email protected]>
>> > wrote:
>> >
>> > > Thank you for this proposal, and for the careful work behind it. I
>> took a
>> > > second, deeper look at the document, and I want to share my concerns,
>> > > because I believe they change the direction rather than the details.
>> > >
>> > > First, I want to recognize the real problem the AIP identifies. The
>> parse
>> > > loop controls when new code becomes active, and that timing is not
>> > > deterministic. Teams need a way to control activation time, and they
>> > need a
>> > > fast rollback path. This is a genuine gap, and I am glad the AIP puts
>> a
>> > > spotlight on it.
>> > >
>> > > After the second review, however, I believe the proposal rests on a
>> > > misunderstanding of what a DagVersion is. A DagVersion is a historical
>> > > record of what Airflow observed after a parse. It is an output of the
>> > > system, not an input to it. Dag versioning exists so that running and
>> > past
>> > > task instances keep a faithful record of the code they ran against.
>> > *Bundle
>> > > versioning* is the mechanism that decides which source code a run
>> uses.
>> > The
>> > > AIP reads DagVersion as a release artifact that users can select and
>> > > activate, and the current implementation does not support that
>> reading:
>> > >
>> > > 1. *The latest DagVersion is mutable*. When no task instance
>> references
>> > it,
>> > > write_dag() replaces its serialized content, hash, and Dag code in
>> place,
>> > > on the same version ID. A pin does not create a task instance
>> reference.
>> > So
>> > > a user could pin the current version before a risky merge, and the
>> next
>> > > parse could replace the content behind that pin. The pin would not
>> give
>> > the
>> > > determinism it promises.
>> > >
>> > > 2. *A pin on the current DagVersion would not pin the source code*.
>> When
>> > a
>> > > new commit does not change the serialized content, `write_dag()`
>> updates
>> > > bundle_version on the latest DagVersion in place. A change to a helper
>> > > module or an SQL file does not change the serialized hash, so workers
>> > would
>> > > run the new commit under the pinned version.
>> > >
>> > > 3. *Version identity is often accidental.* Parse-time variability and
>> > > serializer defects can create versions that no one intended. Airflow
>> > ships
>> > > an inflation checker and a dedicated Dag warning type because this
>> > problem
>> > > is real and common. Dag factories at large scale, the AIP's own key
>> > > example, are the most exposed. A pin would often select an accidental
>> > > snapshot rather than a deliberate release.
>> > >
>> > > 4. *The proposal does not address cleanup*. Today, `airflow db clean`
>> > keeps
>> > > the latest version per Dag and skips only versions that task instances
>> > > reference. An older pinned version with no task instances is eligible
>> for
>> > > deletion, and the proposed ON DELETE SET NULL behavior would then
>> > silently
>> > > return the Dag to the latest version, exactly when the pin matters
>> most.
>> > > The document does not cover this interaction.
>> > >
>> > > 5. *The deletion rule works against the main use case*. The AIP states
>> > that
>> > > removal from a new bundle version deletes the Dag, even when it is
>> > pinned.
>> > > A defect in a dag-factory file that changes or drops Dag IDs removes
>> Dags
>> > > from the parse results. In that case, Airflow would delete the Dag and
>> > > discard the pin, so the protection would be absent in one of the most
>> > > common failure modes the AIP is motivated by.
>> > >
>> > > These are not implementation gaps that more safeguards can close. They
>> > all
>> > > follow from the same source: the design asks a history table to act
>> as a
>> > > deployment mechanism. Making that work would mean redefining what
>> > > DagVersion is, and I do not think we should.
>> > >
>> > > The goal itself is worth pursuing at the right layer. Deterministic
>> > cutover
>> > > and instant rollback are operations on the *source artifact*. Bundle
>> > > versions already represent the source, and pinning or rolling back at
>> the
>> > > bundle layer would be atomic for all Dags and all support files at
>> once.
>> > I
>> > > would be glad to help explore that direction.
>> > >
>> > > Thanks again for raising this discussion. The problem is real, and I
>> hope
>> > > we can solve it together on a foundation that supports it.
>> > >
>> > > - Ephraim
>> > >
>> > > On Mon, 27 Jul 2026 at 06:53, Piyush Maheshwari <
>> > > [email protected]>
>> > > wrote:
>> > >
>> > > > Thank you, Sumit.
>> > > >
>> > > > I've started a voting thread as suggested:
>> > > > https://lists.apache.org/thread/bff758c5p68v1mtzhflytn7yzv0d8l1t
>> > > >
>> > > > Regards,
>> > > > Piyush
>> > > >
>> > > > On Fri, Jul 24, 2026 at 12:08 PM Sumit Maheshwari <
>> > > [email protected]>
>> > > > wrote:
>> > > >
>> > > > > I've reviewed it again today and left some comments. Overall, it
>> > looks
>> > > > > great and would be a great feature add in Airflow. I think you can
>> > > start
>> > > > an
>> > > > > official voting thread on this AIP.
>> > > > >
>> > > > >
>> > > > > On Sun, Jul 19, 2026 at 7:32 PM Piyush Maheshwari <
>> > > > > [email protected]> wrote:
>> > > > >
>> > > > > > Hi everyone,
>> > > > > > Thanks for the reviews on AIP-109 (
>> > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-109+DAG+Version+Pinning
>> > > > > > ).
>> > > > > > Over the past few weeks, I implemented the feature in our fork
>> > > (nearly
>> > > > > > done).
>> > > > > > Based on the insights and learnings from this process, I have
>> > further
>> > > > > > refined the AIP while addressing all comments.
>> > > > > >
>> > > > > > Please review the updated version. I'll wait to collect some
>> > feedback
>> > > > and
>> > > > > > then take it to a vote.
>> > > > > > I can start contributing PRs as soon as the AIP is approved.
>> > > > > >
>> > > > > > Thanks and regards,
>> > > > > > Piyush
>> > > > > >
>> > > > > > On Wed, Jun 3, 2026 at 2:01 PM Ephraim Anierobi <
>> > > > > [email protected]
>> > > > > > >
>> > > > > > wrote:
>> > > > > >
>> > > > > > > Hi Piyush,
>> > > > > > >
>> > > > > > > Thanks for the AIP. I’ve added a few comments to the doc.
>> Please
>> > > > take a
>> > > > > > > look, as I think we should iron out those areas and fully
>> > > understand
>> > > > > the
>> > > > > > > implications before moving forward.
>> > > > > > >
>> > > > > > > Regards
>> > > > > > > - Ephraim
>> > > > > > >
>> > > > > > > On Tue, 26 May 2026 at 18:24, Przemysław Mirowski <
>> > > [email protected]
>> > > > >
>> > > > > > > wrote:
>> > > > > > >
>> > > > > > > > I checked the API - +1. Thanks for writing this up!
>> > > > > > > >
>> > > > > > > > Thanks Nathan for mentioning the #63884 PR. It is nice
>> addition
>> > > > > > (already
>> > > > > > > > merged), and I think that it will be really useful for users
>> > who
>> > > > got
>> > > > > > used
>> > > > > > > > to Airflow 2 retry mechanism. I think also that your PR and
>> > > API-109
>> > > > > are
>> > > > > > > > complementary as one is focusing on the versioning behaviour
>> > for
>> > > > > > retrying
>> > > > > > > > task, the latter is focusing on versioning behaviour for new
>> > Dag
>> > > > > Runs.
>> > > > > > > > ________________________________
>> > > > > > > > From: Christos Bisias <[email protected]>
>> > > > > > > > Sent: 25 May 2026 09:46
>> > > > > > > > To: [email protected] <[email protected]>
>> > > > > > > > Subject: Re: [DISCUSS] DAG Version Pinning for Deployment
>> > Gating
>> > > > > > > (Building
>> > > > > > > > on AIP-63)
>> > > > > > > >
>> > > > > > > > Hello,
>> > > > > > > >
>> > > > > > > > I'm a little late on the discussion but I just came across
>> the
>> > > AIP
>> > > > > and
>> > > > > > I
>> > > > > > > > like this idea. I've actually been thinking of working on
>> > > something
>> > > > > > > > similar, to allow people to handle a bad rollout by
>> reverting
>> > to
>> > > > old
>> > > > > > code
>> > > > > > > > for that run without a full slow release pipeline. And this
>> > > covers
>> > > > > it.
>> > > > > > > >
>> > > > > > > > In my opinion, this seems more like a natural step towards
>> what
>> > > dag
>> > > > > > > > versioning is supposed to do, than a new feature.
>> > > > > > > >
>> > > > > > > > Thank you,
>> > > > > > > > Christos
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > On Mon, May 25, 2026 at 8:15 AM Piyush Maheshwari <
>> > > > > > > > [email protected]> wrote:
>> > > > > > > >
>> > > > > > > > > Hi everyone,
>> > > > > > > > > I wanted to send a gentle nudge to review AIP-109: DAG
>> > Version
>> > > > > > Pinning
>> > > > > > > (
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-109+DAG+Version+Pinning
>> > > > > > > > > ).
>> > > > > > > > > If there are no major concerns, I would like to take this
>> to
>> > a
>> > > > vote
>> > > > > > > soon.
>> > > > > > > > >
>> > > > > > > > > Thanks,
>> > > > > > > > > Piyush
>> > > > > > > > >
>> > > > > > > > > On Fri, May 15, 2026 at 7:44 AM Piyush Maheshwari <
>> > > > > > > > > [email protected]> wrote:
>> > > > > > > > >
>> > > > > > > > > > Thanks for the note, Sumit. Based on the feedback, I've
>> > > drafted
>> > > > > an
>> > > > > > > AIP
>> > > > > > > > > > that is now up for review.
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-109+DAG+Version+Pinning
>> > > > > > > > > >
>> > > > > > > > > > Would like to get the community's feedback on the same.
>> > > > > > > > > >
>> > > > > > > > > > Nathan, I remember seeing your work (the issue and an
>> older
>> > > PR)
>> > > > > > while
>> > > > > > > > > > reviewing all ongoing work related to DAG versions. I
>> agree
>> > > > with
>> > > > > > the
>> > > > > > > > PR's
>> > > > > > > > > > intent, although I haven't reviewed it yet.
>> > > > > > > > > > I understand your PR makes the version-pinned execution
>> > > > behavior
>> > > > > of
>> > > > > > > > > reruns
>> > > > > > > > > > and backfills configurable.
>> > > > > > > > > > However, this discussion revolves around the behavior
>> for
>> > new
>> > > > > runs
>> > > > > > > > only.
>> > > > > > > > > > We need the capability to pin a DAG to a specific
>> version
>> > for
>> > > > > > future
>> > > > > > > > runs
>> > > > > > > > > > instead.
>> > > > > > > > > > Hope that clarifies.
>> > > > > > > > > >
>> > > > > > > > > > Regards,
>> > > > > > > > > > Piyush
>> > > > > > > > > >
>> > > > > > > > > > On Thu, May 14, 2026 at 5:11 PM Nathan Hadfield <
>> > > > > > > > > [email protected]>
>> > > > > > > > > > wrote:
>> > > > > > > > > >
>> > > > > > > > > >> Hello,
>> > > > > > > > > >>
>> > > > > > > > > >> I saw AIP-109 that was created in relation to this
>> > > discussion
>> > > > > and
>> > > > > > > > > thought
>> > > > > > > > > >> I’d better mention this PR that I’ve been working on
>> for a
>> > > > while
>> > > > > > and
>> > > > > > > > is
>> > > > > > > > > >> close to being approved.
>> > > > > > > > > >>
>> > > > > > > > > >> https://github.com/apache/airflow/pull/63884
>> > > > > > > > > >>
>> > > > > > > > > >> It is very much related to the motivations described
>> and
>> > > > > > implements
>> > > > > > > > the
>> > > > > > > > > >> desire for control over the behaviour when
>> > > > clearing/backfilling
>> > > > > > > runs.
>> > > > > > > > > >>
>> > > > > > > > > >> Happy to discuss best steps for this here or on the PR.
>> > > > > > > > > >>
>> > > > > > > > > >> Cheers,
>> > > > > > > > > >>
>> > > > > > > > > >> Nathan
>> > > > > > > > > >>
>> > > > > > > > > >> From: Przemysław Mirowski <[email protected]>
>> > > > > > > > > >> Date: Tuesday, 28 April 2026 at 21:54
>> > > > > > > > > >> To: [email protected] <[email protected]>
>> > > > > > > > > >> Subject: Re: [DISCUSS] DAG Version Pinning for
>> Deployment
>> > > > Gating
>> > > > > > > > > >> (Building on AIP-63)
>> > > > > > > > > >>
>> > > > > > > > > >> This Message Is From an External Sender
>> > > > > > > > > >> This message came from outside your organization.
>> > > > > > > > > >>
>> > > > > > > > > >>
>> > > > > > > > > >> > P.S. In my opinion, what can be done in/around git,
>> > should
>> > > > be
>> > > > > > done
>> > > > > > > > > >> there. Recreation of CI/CD in any form inside of
>> Airflow
>> > > > itself
>> > > > > is
>> > > > > > > > > >> something which should not be done.
>> > > > > > > > > >> > I'm glad we agree on this :) I suppose we just
>> disagree
>> > on
>> > > > > what
>> > > > > > is
>> > > > > > > > > >> possible outside of Airflow :p
>> > > > > > > > > >>
>> > > > > > > > > >> I think that we just disagree on what the issue is,
>> not on
>> > > > what
>> > > > > is
>> > > > > > > > > >> possible/should be outside of Airflow.
>> > > > > > > > > >>
>> > > > > > > > > >> > I think we are trying to duplicate what we already
>> have
>> > in
>> > > > > Git.
>> > > > > > > > > >>
>> > > > > > > > > >> Not really if we are only referring to version
>> pinning. As
>> > > far
>> > > > > as
>> > > > > > I
>> > > > > > > am
>> > > > > > > > > >> aware of how things are working, there is no
>> possibility
>> > to
>> > > > > > > determine
>> > > > > > > > > that
>> > > > > > > > > >> Dag after e.g. deployment on 1:00:00 PM will be exactly
>> > > parsed
>> > > > > and
>> > > > > > > > used
>> > > > > > > > > >> since 1:01:00 PM forward. Basically, what version
>> pinning
>> > > > would
>> > > > > > > > provide
>> > > > > > > > > is
>> > > > > > > > > >> the full control of the time since the given version
>> will
>> > be
>> > > > > used
>> > > > > > > > > >> (currently we can only have more-or-less timing which
>> in
>> > > some
>> > > > > > cases,
>> > > > > > > > is
>> > > > > > > > > not
>> > > > > > > > > >> sufficient). The "quick revert" is the consequence of
>> > having
>> > > > > above
>> > > > > > > > > >> possibility.
>> > > > > > > > > >>
>> > > > > > > > > >> Looking at the general concerns, with having that
>> feature
>> > or
>> > > > > not,
>> > > > > > > > users
>> > > > > > > > > >> can pretty easily test things on production, but it
>> just
>> > > > > requires
>> > > > > > > more
>> > > > > > > > > time
>> > > > > > > > > >> between iterations without it. IMHO it will not change
>> the
>> > > > need
>> > > > > > for
>> > > > > > > > > >> Airflow-related platform teams which makes sure, by
>> > > > > > > > standards/policies,
>> > > > > > > > > >> that things are properly tested before production
>> > > deployment.
>> > > > I
>> > > > > > > think
>> > > > > > > > > that
>> > > > > > > > > >> assumption that some users will misuse this feature is
>> > true
>> > > > > (like
>> > > > > > > with
>> > > > > > > > > most
>> > > > > > > > > >> of the features really), but on the other hand it would
>> > > > provide
>> > > > > > more
>> > > > > > > > > >> control for other users. The other solution possibly
>> would
>> > > be
>> > > > to
>> > > > > > > make
>> > > > > > > > > Dag
>> > > > > > > > > >> Processor work more on "events" instead of "simple"
>> > parsing
>> > > > loop
>> > > > > > (I
>> > > > > > > > > recall
>> > > > > > > > > >> that there was some PR couple years ago with PoC of
>> that,
>> > > but
>> > > > I
>> > > > > > > > couldn't
>> > > > > > > > > >> quickly find it).
>> > > > > > > > > >>
>> > > > > > > > > >> ________________________________
>> > > > > > > > > >> From: Jarek Potiuk <[email protected]>
>> > > > > > > > > >> Sent: 28 April 2026 17:02
>> > > > > > > > > >> To: [email protected] <[email protected]>
>> > > > > > > > > >> Subject: Re: [DISCUSS] DAG Version Pinning for
>> Deployment
>> > > > Gating
>> > > > > > > > > >> (Building on AIP-63)
>> > > > > > > > > >>
>> > > > > > > > > >> Same concerns. I think we are trying to duplicate what
>> we
>> > > > > already
>> > > > > > > have
>> > > > > > > > > in
>> > > > > > > > > >> Git—branches and reverts, for example—by moving what
>> > should
>> > > be
>> > > > > > > managed
>> > > > > > > > > as
>> > > > > > > > > >> part of the development process to Airflow UI.
>> > > > > > > > > >>
>> > > > > > > > > >> Almost everything you describe can be done with:
>> > > > > > > > > >>
>> > > > > > > > > >> * having a dev/staging system configured properly to
>> use
>> > > > > > dev/staging
>> > > > > > > > > >> branches
>> > > > > > > > > >> * Having a process of managing development and a proper
>> > > > > branching
>> > > > > > > > > strategy
>> > > > > > > > > >> * single git command (for example, `git revert XXXX`
>> > > followed
>> > > > by
>> > > > > > > push
>> > > > > > > > to
>> > > > > > > > > >> the right branch)
>> > > > > > > > > >>
>> > > > > > > > > >> J.
>> > > > > > > > > >>
>> > > > > > > > > >>
>> > > > > > > > > >> On Tue, Apr 28, 2026 at 10:34 AM Pierre Jeambrun <
>> > > > > > > > [email protected]
>> > > > > > > > > >
>> > > > > > > > > >> wrote:
>> > > > > > > > > >>
>> > > > > > > > > >> > At first glance I tend to agree with Jens and Niko.
>> > > > > > > > > >> >
>> > > > > > > > > >> > I understand the request, but I agree that this
>> resolves
>> > > > CI/CD
>> > > > > > and
>> > > > > > > > > >> testing
>> > > > > > > > > >> > issues that should probably be remain outside
>> Airflow.
>> > > > > > > > > >> >
>> > > > > > > > > >> > On Mon, Apr 27, 2026 at 7:43 PM Oliveira, Niko <
>> > > > > > > [email protected]
>> > > > > > > > >
>> > > > > > > > > >> > wrote:
>> > > > > > > > > >> >
>> > > > > > > > > >> > > Hey folks!
>> > > > > > > > > >> > >
>> > > > > > > > > >> > > > P.S. In my opinion, what can be done in/around
>> git,
>> > > > should
>> > > > > > be
>> > > > > > > > done
>> > > > > > > > > >> > > there. Recreation of CI/CD in any form inside of
>> > Airflow
>> > > > > > itself
>> > > > > > > is
>> > > > > > > > > >> > > something which should not be done.
>> > > > > > > > > >> > >
>> > > > > > > > > >> > > I'm glad we agree on this :) I suppose we just
>> > disagree
>> > > on
>> > > > > > what
>> > > > > > > is
>> > > > > > > > > >> > > possible outside of Airflow :p
>> > > > > > > > > >> > >
>> > > > > > > > > >> > > But at this point I will bow out of the
>> conversation
>> > and
>> > > > let
>> > > > > > > > others
>> > > > > > > > > >> weigh
>> > > > > > > > > >> > > in. I'm not fully convinced any of these requested
>> > > > > behaviours
>> > > > > > > > > require
>> > > > > > > > > >> > > changes to Airflow (I think that's just masking
>> some
>> > dev
>> > > > ops
>> > > > > > > > work).
>> > > > > > > > > >> But
>> > > > > > > > > >> > > also I'm not completely opposed to the change
>> either,
>> > > I'm
>> > > > > more
>> > > > > > > on
>> > > > > > > > > the
>> > > > > > > > > >> > > fence, so if others love the feature by all means
>> > > > implement
>> > > > > > it!
>> > > > > > > :)
>> > > > > > > > > >> > >
>> > > > > > > > > >> > > Cheers,
>> > > > > > > > > >> > > Niko
>> > > > > > > > > >> > > ________________________________
>> > > > > > > > > >> > > From: Przemysław Mirowski <[email protected]>
>> > > > > > > > > >> > > Sent: Thursday, April 23, 2026 3:06 PM
>> > > > > > > > > >> > > To: [email protected] <[email protected]
>> >
>> > > > > > > > > >> > > Subject: RE: [EXT] [DISCUSS] DAG Version Pinning
>> for
>> > > > > > Deployment
>> > > > > > > > > Gating
>> > > > > > > > > >> > > (Building on AIP-63)
>> > > > > > > > > >> > >
>> > > > > > > > > >> > > CAUTION: This email originated from outside of the
>> > > > > > organization.
>> > > > > > > > Do
>> > > > > > > > > >> not
>> > > > > > > > > >> > > click links or open attachments unless you can
>> confirm
>> > > the
>> > > > > > > sender
>> > > > > > > > > and
>> > > > > > > > > >> > know
>> > > > > > > > > >> > > the content is safe.
>> > > > > > > > > >> > >
>> > > > > > > > > >> > >
>> > > > > > > > > >> > >
>> > > > > > > > > >> > > AVERTISSEMENT: Ce courrier électronique provient
>> d’un
>> > > > > > expéditeur
>> > > > > > > > > >> externe.
>> > > > > > > > > >> > > Ne cliquez sur aucun lien et n’ouvrez aucune pièce
>> > > jointe
>> > > > si
>> > > > > > > vous
>> > > > > > > > ne
>> > > > > > > > > >> > pouvez
>> > > > > > > > > >> > > pas confirmer l’identité de l’expéditeur et si vous
>> > > n’êtes
>> > > > > pas
>> > > > > > > > > certain
>> > > > > > > > > >> > que
>> > > > > > > > > >> > > le contenu ne présente aucun risque.
>> > > > > > > > > >> > >
>> > > > > > > > > >> > >
>> > > > > > > > > >> > >
>> > > > > > > > > >> > > Hi,
>> > > > > > > > > >> > >
>> > > > > > > > > >> > > I think that CI/CD and version pining are a little
>> two
>> > > > > > different
>> > > > > > > > > >> things
>> > > > > > > > > >> > > here. In a use cases with some critical systems
>> > > involved,
>> > > > > the
>> > > > > > > > > >> situation
>> > > > > > > > > >> > > when the Dag changes the version to the latest
>> without
>> > > > > > > possibility
>> > > > > > > > > to
>> > > > > > > > > >> > > determine when it will exactly happen (CI/CD will
>> have
>> > > > some
>> > > > > > > > > >> more-or-less
>> > > > > > > > > >> > > time to deploy the change, the same goes for Dag
>> > > Processor
>> > > > > > > parsing
>> > > > > > > > > >> time)
>> > > > > > > > > >> > is
>> > > > > > > > > >> > > rather hard to do and in some systems it can make
>> > change
>> > > > > > > > deployment
>> > > > > > > > > >> > harder
>> > > > > > > > > >> > > and less safe. Of course, the ideal solution would
>> be
>> > to
>> > > > > have
>> > > > > > > > proper
>> > > > > > > > > >> > > non-prod environment, which is fully
>> representative in
>> > > > > > > comparison
>> > > > > > > > to
>> > > > > > > > > >> > > production (in some cases exposing non-prod to prod
>> > > > > > > > > data/traffic/etc.
>> > > > > > > > > >> is,
>> > > > > > > > > >> > > just, not an option - e.g. security), but it is not
>> > > always
>> > > > > > > > possible
>> > > > > > > > > >> to do
>> > > > > > > > > >> > > due to various reasons like costs, licenses, space
>> > > and/or
>> > > > > > > vendors.
>> > > > > > > > > I'm
>> > > > > > > > > >> > > agreeing especially with point 5 of Piyush latest
>> > > message.
>> > > > > > > Having
>> > > > > > > > > >> above
>> > > > > > > > > >> > in
>> > > > > > > > > >> > > mind, I think that version pinning would be a nice
>> > > > addition
>> > > > > to
>> > > > > > > the
>> > > > > > > > > Dag
>> > > > > > > > > >> > > Versioning feature with an assumption that it is
>> for
>> > > > > critical
>> > > > > > > > > Airflow
>> > > > > > > > > >> > Dags
>> > > > > > > > > >> > > when full control of the Dags version change time
>> is
>> > > > > required
>> > > > > > > > (maybe
>> > > > > > > > > >> > there
>> > > > > > > > > >> > > is also another way to achieve that).
>> > > > > > > > > >> > >
>> > > > > > > > > >> > > P.S. In my opinion, what can be done in/around git,
>> > > should
>> > > > > be
>> > > > > > > done
>> > > > > > > > > >> there.
>> > > > > > > > > >> > > Recreation of CI/CD in any form inside of Airflow
>> > itself
>> > > > is
>> > > > > > > > > something
>> > > > > > > > > >> > which
>> > > > > > > > > >> > > should not be done.
>> > > > > > > > > >> > > ________________________________
>> > > > > > > > > >> > > From: Oliveira, Niko <[email protected]>
>> > > > > > > > > >> > > Sent: 23 April 2026 01:50
>> > > > > > > > > >> > > To: [email protected] <[email protected]
>> >
>> > > > > > > > > >> > > Subject: Re: [DISCUSS] DAG Version Pinning for
>> > > Deployment
>> > > > > > Gating
>> > > > > > > > > >> > (Building
>> > > > > > > > > >> > > on AIP-63)
>> > > > > > > > > >> > >
>> > > > > > > > > >> > > Hey Piyush,
>> > > > > > > > > >> > >
>> > > > > > > > > >> > > Thanks for your reply, I do love how clearly it is
>> > > written
>> > > > > > and I
>> > > > > > > > see
>> > > > > > > > > >> > > exactly the problem you're trying to solve!
>> > > > > > > > > >> > >
>> > > > > > > > > >> > > I'm still just not convinced this needs to be done
>> in
>> > > > > Airflow,
>> > > > > > > at
>> > > > > > > > > >> least
>> > > > > > > > > >> > > not with a first class feature. As interesting as I
>> > > think
>> > > > > your
>> > > > > > > > > >> > microservice
>> > > > > > > > > >> > > analogy is, Airflow is not a microservice
>> component,
>> > it
>> > > > is a
>> > > > > > > > (very,
>> > > > > > > > > >> very)
>> > > > > > > > > >> > > fancy cron scheduler. And I'm not sure the
>> complexity
>> > is
>> > > > > worth
>> > > > > > > the
>> > > > > > > > > use
>> > > > > > > > > >> > > case. Since any new code added to Airflow must be
>> > > > maintained
>> > > > > > by
>> > > > > > > > this
>> > > > > > > > > >> > > community and we must be cautious that any new
>> pieces
>> > > > serves
>> > > > > > > > enough
>> > > > > > > > > >> use
>> > > > > > > > > >> > > cases/users to make it worth it.
>> > > > > > > > > >> > > To me this should either be managed outside of an
>> > > > individual
>> > > > > > > > Airflow
>> > > > > > > > > >> > > environment e.g. you have an entirely separate
>> > > > > > staging/gamma/dev
>> > > > > > > > > >> Airflow
>> > > > > > > > > >> > > environment, which is exposed to some level of
>> > > production
>> > > > > > > traffic
>> > > > > > > > > (to
>> > > > > > > > > >> > > borrow your microservice analogy) until it can
>> > graduate
>> > > to
>> > > > > the
>> > > > > > > > > >> production
>> > > > > > > > > >> > > environment. And if you really need on the fly
>> > toggling
>> > > > of a
>> > > > > > > > > version,
>> > > > > > > > > >> as
>> > > > > > > > > >> > > you say, Airflow does this quite responsively, if
>> you
>> > > > > deploy a
>> > > > > > > new
>> > > > > > > > > >> > version
>> > > > > > > > > >> > > of your dags it will parse and start using that new
>> > > > version
>> > > > > > > > > >> immediately
>> > > > > > > > > >> > > (the problem you're trying to solve can be a
>> benefit
>> > > > here).
>> > > > > > You
>> > > > > > > > can
>> > > > > > > > > >> even
>> > > > > > > > > >> > > have multiple versions of your dags deployed at
>> once
>> > and
>> > > > use
>> > > > > > > > > >> > configuration
>> > > > > > > > > >> > > to control which dag directory Airflow reads from
>> (or
>> > > > > > > move/symlink
>> > > > > > > > > >> Dags
>> > > > > > > > > >> > in
>> > > > > > > > > >> > > and out of the Dags directory as needed from a
>> known
>> > > good
>> > > > or
>> > > > > > > > pinned
>> > > > > > > > > >> > > source). Or use variables or some other parameter
>> > store
>> > > to
>> > > > > > > control
>> > > > > > > > > >> other
>> > > > > > > > > >> > > pieces of runtime behaviour inside the Dags
>> > themselves.
>> > > > > > Between
>> > > > > > > > > CI/CD,
>> > > > > > > > > >> > dev
>> > > > > > > > > >> > > ops and making use of existing Airflow primitives I
>> > > think
>> > > > > you
>> > > > > > > can
>> > > > > > > > > >> achieve
>> > > > > > > > > >> > > what you're looking for.
>> > > > > > > > > >> > >
>> > > > > > > > > >> > > But as always, this is open and community based
>> > > software,
>> > > > so
>> > > > > > I'm
>> > > > > > > > > >> happy to
>> > > > > > > > > >> > > disagree and commit if the rest of the community
>> > thinks
>> > > > this
>> > > > > > is
>> > > > > > > a
>> > > > > > > > > >> > valuable
>> > > > > > > > > >> > > feature :)
>> > > > > > > > > >> > >
>> > > > > > > > > >> > > Cheers,
>> > > > > > > > > >> > > Niko
>> > > > > > > > > >> > > ________________________________
>> > > > > > > > > >> > > From: Piyush Maheshwari <[email protected]>
>> > > > > > > > > >> > > Sent: Tuesday, April 21, 2026 10:46 PM
>> > > > > > > > > >> > > To: [email protected] <[email protected]
>> >
>> > > > > > > > > >> > > Subject: RE: [EXT] [DISCUSS] DAG Version Pinning
>> for
>> > > > > > Deployment
>> > > > > > > > > Gating
>> > > > > > > > > >> > > (Building on AIP-63)
>> > > > > > > > > >> > >
>> > > > > > > > > >> > > CAUTION: This email originated from outside of the
>> > > > > > organization.
>> > > > > > > > Do
>> > > > > > > > > >> not
>> > > > > > > > > >> > > click links or open attachments unless you can
>> confirm
>> > > the
>> > > > > > > sender
>> > > > > > > > > and
>> > > > > > > > > >> > know
>> > > > > > > > > >> > > the content is safe.
>> > > > > > > > > >> > >
>> > > > > > > > > >> > >
>> > > > > > > > > >> > >
>> > > > > > > > > >> > > AVERTISSEMENT: Ce courrier électronique provient
>> d’un
>> > > > > > expéditeur
>> > > > > > > > > >> externe.
>> > > > > > > > > >> > > Ne cliquez sur aucun lien et n’ouvrez aucune pièce
>> > > jointe
>> > > > si
>> > > > > > > vous
>> > > > > > > > ne
>> > > > > > > > > >> > pouvez
>> > > > > > > > > >> > > pas confirmer l’identité de l’expéditeur et si vous
>> > > n’êtes
>> > > > > pas
>> > > > > > > > > certain
>> > > > > > > > > >> > que
>> > > > > > > > > >> > > le contenu ne présente aucun risque.
>> > > > > > > > > >> > >
>> > > > > > > > > >> > >
>> > > > > > > > > >> > >
>> > > > > > > > > >> > > Hi Ephraim, Jarek, Jens, and Niko,
>> > > > > > > > > >> > >
>> > > > > > > > > >> > > Thank you for the candid feedback. I want to
>> clarify a
>> > > few
>> > > > > > > things,
>> > > > > > > > > as
>> > > > > > > > > >> I
>> > > > > > > > > >> > > completely agree with Jens and Niko that "testing
>> in
>> > > > > > production"
>> > > > > > > > is
>> > > > > > > > > an
>> > > > > > > > > >> > > anti-pattern. That is absolutely not the intention
>> > here.
>> > > > > > > > > >> > >
>> > > > > > > > > >> > > 1. I view this as bringing standard
>> microservice-like
>> > > > > > deployment
>> > > > > > > > > >> maturity
>> > > > > > > > > >> > > to DAGs.
>> > > > > > > > > >> > > Before service deployments in our org, code is
>> tested
>> > > > > locally,
>> > > > > > > in
>> > > > > > > > a
>> > > > > > > > > >> dev
>> > > > > > > > > >> > > environment, and via strict unit/e2e integration
>> tests
>> > > > > before
>> > > > > > it
>> > > > > > > > > ever
>> > > > > > > > > >> > makes
>> > > > > > > > > >> > > it to main. But even after merging and passing
>> those
>> > CI
>> > > > > > > pipelines,
>> > > > > > > > > we
>> > > > > > > > > >> > still
>> > > > > > > > > >> > > use load tests, pre-prod soak times, shadow
>> traffic,
>> > and
>> > > > > gated
>> > > > > > > > > >> production
>> > > > > > > > > >> > > rollouts with automated rollback triggers. Having
>> > > > deployment
>> > > > > > > gates
>> > > > > > > > > for
>> > > > > > > > > >> > the
>> > > > > > > > > >> > > production environment doesn't mean the pre-merge
>> > checks
>> > > > > > weren't
>> > > > > > > > > >> strict
>> > > > > > > > > >> > or
>> > > > > > > > > >> > > that the change wasn't tested beforehand -- it just
>> > > allows
>> > > > > us
>> > > > > > to
>> > > > > > > > > place
>> > > > > > > > > >> > > additional safety gates for the code to take
>> effect,
>> > > > exactly
>> > > > > > > like
>> > > > > > > > in
>> > > > > > > > > >> the
>> > > > > > > > > >> > > service world.
>> > > > > > > > > >> > >
>> > > > > > > > > >> > > 2. The core issue we are trying to solve is that
>> > Airflow
>> > > > > > > currently
>> > > > > > > > > >> > > inseparably links Code Distribution (a file
>> arriving
>> > on
>> > > > the
>> > > > > > > > > >> dag-processor
>> > > > > > > > > >> > > and being parsed) with Release Activation (the
>> > scheduler
>> > > > > > > executing
>> > > > > > > > > >> that
>> > > > > > > > > >> > > code).
>> > > > > > > > > >> > > To extend the microservices analogy, I can think of
>> > the
>> > > > DAG
>> > > > > > > > > processor
>> > > > > > > > > >> > > parsing all files as "building the artifact(s),"
>> while
>> > > the
>> > > > > > > > scheduler
>> > > > > > > > > >> and
>> > > > > > > > > >> > > executor acting on the DAG versions created
>> thereafter
>> > > as
>> > > > > > > > > "deploying"
>> > > > > > > > > >> or
>> > > > > > > > > >> > > running the changed code.
>> > > > > > > > > >> > > We simply want to decouple the build from the
>> > > deployment.
>> > > > > This
>> > > > > > > > does
>> > > > > > > > > >> not
>> > > > > > > > > >> > > mean that the code arriving on the dag-processor
>> will
>> > be
>> > > > > > tested
>> > > > > > > > for
>> > > > > > > > > >> the
>> > > > > > > > > >> > > first time straight in production. It should've
>> > already
>> > > > > > passed a
>> > > > > > > > set
>> > > > > > > > > >> of
>> > > > > > > > > >> > > checks in the CI pipeline.
>> > > > > > > > > >> > >
>> > > > > > > > > >> > > 3. It is also worth calling out that Airflow
>> already
>> > > > > supports
>> > > > > > > this
>> > > > > > > > > >> > > decoupled behavior at the run level for task
>> re-runs
>> > and
>> > > > > > > > > mid-execution
>> > > > > > > > > >> > DAG
>> > > > > > > > > >> > > version bumps (by pinning the version for the rest
>> of
>> > > the
>> > > > > > > > execution
>> > > > > > > > > or
>> > > > > > > > > >> > the
>> > > > > > > > > >> > > rerun). We are simply trying to expose this
>> existing
>> > > > > > capability
>> > > > > > > at
>> > > > > > > > > the
>> > > > > > > > > >> > DAG
>> > > > > > > > > >> > > level so users can govern which version new
>> scheduled
>> > > runs
>> > > > > are
>> > > > > > > > > created
>> > > > > > > > > >> > > with.
>> > > > > > > > > >> > >
>> > > > > > > > > >> > > 4. I also agree that Airflow itself should not be
>> > aware
>> > > of
>> > > > > our
>> > > > > > > > CI/CD
>> > > > > > > > > >> > > pipeline, nor would it manage the deployment
>> > > orchestration
>> > > > > or
>> > > > > > > > > testing.
>> > > > > > > > > >> > > For our requirements, I just need Airflow to expose
>> > APIs
>> > > > to
>> > > > > > > deploy
>> > > > > > > > > >> (pin)
>> > > > > > > > > >> > a
>> > > > > > > > > >> > > DAG version, and to remove the pin (to
>> restore/enable
>> > > the
>> > > > > > > default
>> > > > > > > > > >> > > "auto-deploy latest" behavior).
>> > > > > > > > > >> > > Beyond that, we intend to use an external release
>> > > > > orchestrator
>> > > > > > > > that
>> > > > > > > > > >> can
>> > > > > > > > > >> > > explicitly tell Airflow when a parsed version is
>> > > actually
>> > > > > > > allowed
>> > > > > > > > to
>> > > > > > > > > >> run.
>> > > > > > > > > >> > > Until that API call is made, the previously pinned
>> > > version
>> > > > > > > remains
>> > > > > > > > > >> > active.
>> > > > > > > > > >> > > This ensures we don't introduce assumptions or
>> > awareness
>> > > > of
>> > > > > > the
>> > > > > > > > > >> presence
>> > > > > > > > > >> > of
>> > > > > > > > > >> > > any external gating mechanisms to Airflow.
>> > > > > > > > > >> > > Also note that the intention is to keep the default
>> > > > > > auto-deploy
>> > > > > > > > > >> behavior
>> > > > > > > > > >> > > unless a user (or a system on their behalf)
>> explicitly
>> > > > asks
>> > > > > > > > Airflow
>> > > > > > > > > to
>> > > > > > > > > >> > pin
>> > > > > > > > > >> > > a DAG to a specific version.
>> > > > > > > > > >> > >
>> > > > > > > > > >> > > 5. Most importantly, this feature provides an
>> incident
>> > > > > > response
>> > > > > > > > > >> > "rollback"
>> > > > > > > > > >> > > behavior. If a bad DAG version slips through CI/CD
>> > into
>> > > > > > > > production,
>> > > > > > > > > >> > either
>> > > > > > > > > >> > > an on-call engineer or a rollback-trigger
>> > > > (airflow-external)
>> > > > > > can
>> > > > > > > > > >> > instantly
>> > > > > > > > > >> > > roll back to the previous pinned version via the
>> > API/UI
>> > > to
>> > > > > > > > mitigate.
>> > > > > > > > > >> > > Without this, users have to revert the code in Git
>> and
>> > > > wait
>> > > > > > for
>> > > > > > > > the
>> > > > > > > > > >> > entire
>> > > > > > > > > >> > > CI/CD pipeline and file-sync process to run, which
>> is
>> > > > often
>> > > > > > too
>> > > > > > > > slow
>> > > > > > > > > >> > during
>> > > > > > > > > >> > > an outage.
>> > > > > > > > > >> > >
>> > > > > > > > > >> > > 6. Jarek - You are right, database schema changes
>> can
>> > be
>> > > > > > > discussed
>> > > > > > > > > >> later.
>> > > > > > > > > >> > > My intention was only to share a very brief
>> summary of
>> > > > how I
>> > > > > > > > deemed
>> > > > > > > > > >> it to
>> > > > > > > > > >> > > be technically feasible for early feedback. I did
>> > > briefly
>> > > > > > share
>> > > > > > > > the
>> > > > > > > > > >> > > high-level use cases ("Safe Deployment Gating" and
>> > > > "Instant
>> > > > > > > > > >> Rollbacks")
>> > > > > > > > > >> > in
>> > > > > > > > > >> > > the original mail, but I completely agree that
>> > aligning
>> > > on
>> > > > > the
>> > > > > > > UX
>> > > > > > > > > >> first
>> > > > > > > > > >> > > would be a good next step.
>> > > > > > > > > >> > >
>> > > > > > > > > >> > > If there are no major remaining concerns after this
>> > > > > response,
>> > > > > > I
>> > > > > > > > can
>> > > > > > > > > >> draft
>> > > > > > > > > >> > > and share an AIP to detail the UX, followed by a
>> > > > high-level
>> > > > > > > > > proposal,
>> > > > > > > > > >> > > caveats and next steps.
>> > > > > > > > > >> > >
>> > > > > > > > > >> > > Thanks for your time.
>> > > > > > > > > >> > > Regards,
>> > > > > > > > > >> > > Piyush
>> > > > > > > > > >> > >
>> > > > > > > > > >> > > On Tue, Apr 21, 2026 at 5:59 PM Oliveira, Niko <
>> > > > > > > > [email protected]
>> > > > > > > > > >
>> > > > > > > > > >> > > wrote:
>> > > > > > > > > >> > >
>> > > > > > > > > >> > > > I am with Jens on this one. I think we're
>> > complicating
>> > > > > > Airflow
>> > > > > > > > to
>> > > > > > > > > >> get
>> > > > > > > > > >> > > > around a bad practice. If stability of your Dags
>> is
>> > > > > critical
>> > > > > > > and
>> > > > > > > > > >> they
>> > > > > > > > > >> > are
>> > > > > > > > > >> > > > highly versioned then I think as Jens suggested
>> > > running
>> > > > > them
>> > > > > > > > > >> through a
>> > > > > > > > > >> > > > pipeline that first deploys them to a dev or
>> gamma
>> > > > > > environment
>> > > > > > > > > which
>> > > > > > > > > >> > > > verifies that quality of the Dags is what you
>> > expect.
>> > > If
>> > > > > > > > something
>> > > > > > > > > >> > slips
>> > > > > > > > > >> > > > through, then it's just normal software
>> practices of
>> > > > > either
>> > > > > > > > > >> reverting
>> > > > > > > > > >> > and
>> > > > > > > > > >> > > > rolling back or rolling forward with a fix pushed
>> > > > through
>> > > > > > the
>> > > > > > > > > >> > pipeline. I
>> > > > > > > > > >> > > > don't think Airflow should be aware of that
>> process
>> > or
>> > > > > > > > opinionated
>> > > > > > > > > >> > about
>> > > > > > > > > >> > > it.
>> > > > > > > > > >> > > >
>> > > > > > > > > >> > > > Cheers,
>> > > > > > > > > >> > > > Niko
>> > > > > > > > > >> > > > ------------------------------
>> > > > > > > > > >> > > > *From:* Jens Scheffler <[email protected]>
>> > > > > > > > > >> > > > *Sent:* Monday, April 20, 2026 11:17 AM
>> > > > > > > > > >> > > > *To:* [email protected] <
>> > [email protected]>
>> > > > > > > > > >> > > > *Subject:* RE: [EXT] [DISCUSS] DAG Version
>> Pinning
>> > for
>> > > > > > > > Deployment
>> > > > > > > > > >> > Gating
>> > > > > > > > > >> > > > (Building on AIP-63)
>> > > > > > > > > >> > > >
>> > > > > > > > > >> > > > CAUTION: This email originated from outside of
>> the
>> > > > > > > organization.
>> > > > > > > > > Do
>> > > > > > > > > >> not
>> > > > > > > > > >> > > > click links or open attachments unless you can
>> > confirm
>> > > > the
>> > > > > > > > sender
>> > > > > > > > > >> and
>> > > > > > > > > >> > > know
>> > > > > > > > > >> > > > the content is safe.
>> > > > > > > > > >> > > >
>> > > > > > > > > >> > > >
>> > > > > > > > > >> > > >
>> > > > > > > > > >> > > > AVERTISSEMENT: Ce courrier électronique provient
>> > d’un
>> > > > > > > expéditeur
>> > > > > > > > > >> > externe.
>> > > > > > > > > >> > > > Ne cliquez sur aucun lien et n’ouvrez aucune
>> pièce
>> > > > jointe
>> > > > > si
>> > > > > > > > vous
>> > > > > > > > > ne
>> > > > > > > > > >> > > pouvez
>> > > > > > > > > >> > > > pas confirmer l’identité de l’expéditeur et si
>> vous
>> > > > n’êtes
>> > > > > > pas
>> > > > > > > > > >> certain
>> > > > > > > > > >> > > que
>> > > > > > > > > >> > > > le contenu ne présente aucun risque.
>> > > > > > > > > >> > > >
>> > > > > > > > > >> > > >
>> > > > > > > > > >> > > >
>> > > > > > > > > >> > > > Hi,
>> > > > > > > > > >> > > >
>> > > > > > > > > >> > > > I am still quite sceptical. Yes, if such pinning
>> is
>> > > > made,
>> > > > > > then
>> > > > > > > > per
>> > > > > > > > > >> Dag
>> > > > > > > > > >> > a
>> > > > > > > > > >> > > > change need to be possible via UI and API. But I
>> > still
>> > > > see
>> > > > > > it
>> > > > > > > as
>> > > > > > > > > >> > > > checken-and-egg - so you want to run a pinned
>> > version
>> > > > but
>> > > > > > then
>> > > > > > > > how
>> > > > > > > > > >> do
>> > > > > > > > > >> > > > you test the changes (w/o moving a version pin)?
>> > Then
>> > > > > again
>> > > > > > > some
>> > > > > > > > > >> test
>> > > > > > > > > >> > > > mode is needed or per run you need to make a
>> "test
>> > > run"
>> > > > > with
>> > > > > > > > > another
>> > > > > > > > > >> > > > version. Smells a bit like mis-using a production
>> > > system
>> > > > > for
>> > > > > > > > > >> testing.
>> > > > > > > > > >> > > >
>> > > > > > > > > >> > > > On the other hand, yes if all Dags share the same
>> > Git
>> > > > repo
>> > > > > > > then
>> > > > > > > > > >> merging
>> > > > > > > > > >> > > > a branch to some other will switch all Dags at
>> the
>> > > same
>> > > > > > time.
>> > > > > > > > > Still
>> > > > > > > > > >> you
>> > > > > > > > > >> > > > could utilize standard Git tools and cherry-pick
>> > > > > individual
>> > > > > > > > > changes
>> > > > > > > > > >> and
>> > > > > > > > > >> > > > no force to always make a full rollout. At least
>> 80%
>> > > > > > possible
>> > > > > > > > with
>> > > > > > > > > >> > > > standard CI/CD tools and Git.
>> > > > > > > > > >> > > >
>> > > > > > > > > >> > > > TLDR I see the danger that instead of a proper
>> CI/CD
>> > > and
>> > > > > > test
>> > > > > > > > > system
>> > > > > > > > > >> > > > such a feature might feel like you can easily
>> test
>> > on
>> > > a
>> > > > > > > > production
>> > > > > > > > > >> > > > system. Effectively it would be needed allowing
>> to
>> > > > start a
>> > > > > > Dag
>> > > > > > > > > with
>> > > > > > > > > >> any
>> > > > > > > > > >> > > > version to also be able to jump back as a
>> reversion.
>> > > > Even
>> > > > > > > > though,
>> > > > > > > > > >> yes,
>> > > > > > > > > >> > > > agree, all is technically possible.
>> > > > > > > > > >> > > >
>> > > > > > > > > >> > > > Jens
>> > > > > > > > > >> > > >
>> > > > > > > > > >> > > > On 20.04.26 16:40, Jarek Potiuk wrote:
>> > > > > > > > > >> > > > > +1 to what Ephraim wrote. I think that was a
>> > natural
>> > > > > next
>> > > > > > > step
>> > > > > > > > > we
>> > > > > > > > > >> > > > > discussed, but it needs significant refinement,
>> > > > starting
>> > > > > > > with
>> > > > > > > > > the
>> > > > > > > > > >> > > actual
>> > > > > > > > > >> > > > > use cases it should serve and the UX for user
>> > > > > > interaction. I
>> > > > > > > > > think
>> > > > > > > > > >> > > > related
>> > > > > > > > > >> > > > > database changes are pretty secondary. Use
>> cases
>> > > cover
>> > > > > > runs,
>> > > > > > > > > >> re-runs,
>> > > > > > > > > >> > > > > backfills, CI testing, rollbacks, etc.
>> Following
>> > the
>> > > > > > > > > >> "documentation
>> > > > > > > > > >> > > > first"
>> > > > > > > > > >> > > > > approach discussed in separate thread,
>> describing
>> > > the
>> > > > > > > context
>> > > > > > > > > and
>> > > > > > > > > >> > > > intention
>> > > > > > > > > >> > > > > of what we want to achieve is much more
>> important
>> > > than
>> > > > > DB
>> > > > > > > > schema
>> > > > > > > > > >> > > changes.
>> > > > > > > > > >> > > > > Once we know which use cases we want to serve,
>> the
>> > > DB
>> > > > > > schema
>> > > > > > > > > >> changes
>> > > > > > > > > >> > > and
>> > > > > > > > > >> > > > > other related items will emerge naturally.
>> > > > > > > > > >> > > > >
>> > > > > > > > > >> > > > > On Mon, Apr 20, 2026 at 3:15 PM Ephraim
>> Anierobi <
>> > > > > > > > > >> > > > [email protected]>
>> > > > > > > > > >> > > > > wrote:
>> > > > > > > > > >> > > > >
>> > > > > > > > > >> > > > >> Hi Piyush, thanks for starting this
>> discussion.
>> > > > > > > > > >> > > > >>
>> > > > > > > > > >> > > > >> I like the proposal. We can introduce an
>> active
>> > > > > execution
>> > > > > > > > > version
>> > > > > > > > > >> > for
>> > > > > > > > > >> > > > >> "versioned bundles" and make scheduler/API
>> > resolve
>> > > > > > through
>> > > > > > > > it.
>> > > > > > > > > >> The
>> > > > > > > > > >> > > hard
>> > > > > > > > > >> > > > >> part of this is making airflow able to
>> > distinguish
>> > > > the
>> > > > > > > latest
>> > > > > > > > > >> parsed
>> > > > > > > > > >> > > > >> dagmodel's metadata from active scheduling
>> > > metadata.
>> > > > I
>> > > > > > will
>> > > > > > > > > >> suggest
>> > > > > > > > > >> > > you
>> > > > > > > > > >> > > > >> draft this in a google docs and share for
>> further
>> > > > > > > > discussions.
>> > > > > > > > > >> > > > >>
>> > > > > > > > > >> > > > >> Regards
>> > > > > > > > > >> > > > >> - Ephraim
>> > > > > > > > > >> > > > >>
>> > > > > > > > > >> > > > >> On Mon, 20 Apr 2026 at 01:31, Piyush
>> Maheshwari <
>> > > > > > > > > >> > > [email protected]>
>> > > > > > > > > >> > > > >> wrote:
>> > > > > > > > > >> > > > >>
>> > > > > > > > > >> > > > >>> Thanks for sharing your thoughts Jens.
>> > > > > > > > > >> > > > >>>
>> > > > > > > > > >> > > > >>>> be able to test it? … a Q&A/Testing
>> environment
>> > > to
>> > > > be
>> > > > > > > able
>> > > > > > > > to
>> > > > > > > > > >> > > sign-off
>> > > > > > > > > >> > > > >>> changes.
>> > > > > > > > > >> > > > >>> Yes, we’ve have built an isolated airflow
>> > > > environment
>> > > > > to
>> > > > > > > run
>> > > > > > > > > >> > > regression
>> > > > > > > > > >> > > > >>> checks before promoting to production.
>> > > > > > > > > >> > > > >>>
>> > > > > > > > > >> > > > >>> As you suggested, we’re already running both
>> > > generic
>> > > > > and
>> > > > > > > > > >> DAG-custom
>> > > > > > > > > >> > > > >> static
>> > > > > > > > > >> > > > >>> checks in a CI job as a required step to
>> merge
>> > to
>> > > > the
>> > > > > > main
>> > > > > > > > > >> branch.
>> > > > > > > > > >> > > > >>>
>> > > > > > > > > >> > > > >>>> But then the "main" branch might be best
>> suited
>> > > if
>> > > > > > > > > >> > > > >>> implemented on the test system
>> > > > > > > > > >> > > > >>> In this case, problematic commits on “main”
>> can
>> > > > choke
>> > > > > > > other
>> > > > > > > > > >> > unrelated
>> > > > > > > > > >> > > > >>> changes.
>> > > > > > > > > >> > > > >>> So the other option would be to revert the
>> > > > problematic
>> > > > > > > > commits
>> > > > > > > > > >> and
>> > > > > > > > > >> > > > deploy
>> > > > > > > > > >> > > > >>> forward.
>> > > > > > > > > >> > > > >>>
>> > > > > > > > > >> > > > >>> However, a key limitation with this approach
>> > that
>> > > > > > remains
>> > > > > > > is
>> > > > > > > > > >> that a
>> > > > > > > > > >> > > > >> commit
>> > > > > > > > > >> > > > >>> affecting multiple DAGs goes live for either
>> all
>> > > > DAGs
>> > > > > or
>> > > > > > > > none.
>> > > > > > > > > >> > > > >>>
>> > > > > > > > > >> > > > >>> Second important feature we get with this is
>> > > instant
>> > > > > > > > DAG-level
>> > > > > > > > > >> > > rollback
>> > > > > > > > > >> > > > >>> without waiting for a revert commit to merge
>> and
>> > > be
>> > > > > > picked
>> > > > > > > > by
>> > > > > > > > > >> > > airflow.
>> > > > > > > > > >> > > > >>>
>> > > > > > > > > >> > > > >>> I think DAG-level version pinning can also
>> > unlock
>> > > a
>> > > > > lot
>> > > > > > of
>> > > > > > > > > >> > > flexibility
>> > > > > > > > > >> > > > >> for
>> > > > > > > > > >> > > > >>> deployments including tiered rollouts,
>> > > auto-rollback
>> > > > > > > > triggers,
>> > > > > > > > > >> > timed
>> > > > > > > > > >> > > > >>> deployment windows and so on.
>> > > > > > > > > >> > > > >>>
>> > > > > > > > > >> > > > >>> Looking forward to hear your thoughts.
>> > > > > > > > > >> > > > >>> Regards,
>> > > > > > > > > >> > > > >>> Piyush
>> > > > > > > > > >> > > > >>>
>> > > > > > > > > >> > > > >>> On Sun, 19 Apr 2026 at 3:12 PM, Jens
>> Scheffler <
>> > > > > > > > > >> > [email protected]>
>> > > > > > > > > >> > > > >>> wrote:
>> > > > > > > > > >> > > > >>>
>> > > > > > > > > >> > > > >>>> Thanks Piyush for dropping the discussion!
>> > > > > > > > > >> > > > >>>>
>> > > > > > > > > >> > > > >>>> I think in general QA processes are
>> important
>> > > and a
>> > > > > > valid
>> > > > > > > > use
>> > > > > > > > > >> > case.
>> > > > > > > > > >> > > So
>> > > > > > > > > >> > > > >> a
>> > > > > > > > > >> > > > >>>> kind of pinning Dag versions really is
>> > important.
>> > > > > > > > > >> > > > >>>>
>> > > > > > > > > >> > > > >>>> Thinking about it, if you pin the version
>> ...
>> > how
>> > > > > would
>> > > > > > > you
>> > > > > > > > > >> then
>> > > > > > > > > >> > be
>> > > > > > > > > >> > > > >> able
>> > > > > > > > > >> > > > >>>> to test it? I assume you would need (and
>> should
>> > > > have
>> > > > > or
>> > > > > > > > > invest
>> > > > > > > > > >> > > into) a
>> > > > > > > > > >> > > > >>>> Q&A/Testing environment to be able to
>> sign-off
>> > > > > changes.
>> > > > > > > > Both
>> > > > > > > > > in
>> > > > > > > > > >> > > > >>>> infrastructure but also for Dag changes.
>> > > > > > > > > >> > > > >>>>
>> > > > > > > > > >> > > > >>>> If you are changing Dags first of all static
>> > > checks
>> > > > > on
>> > > > > > > Dag
>> > > > > > > > > code
>> > > > > > > > > >> > are
>> > > > > > > > > >> > > > >> very
>> > > > > > > > > >> > > > >>>> much proposed as well as you can have tests
>> > > > > implemented
>> > > > > > > and
>> > > > > > > > > >> test
>> > > > > > > > > >> > > your
>> > > > > > > > > >> > > > >>>> Dags and logic. Similar like software a
>> CI/CD
>> > > > system
>> > > > > > will
>> > > > > > > > be
>> > > > > > > > > a
>> > > > > > > > > >> > good
>> > > > > > > > > >> > > > >>>> setup. Alongside Dag changes also have
>> logical
>> > > > > changes
>> > > > > > > that
>> > > > > > > > > >> mostly
>> > > > > > > > > >> > > can
>> > > > > > > > > >> > > > >>>> only be tested in a live system and not as
>> > static
>> > > > > > checks.
>> > > > > > > > > >> > > > >>>>
>> > > > > > > > > >> > > > >>>> Have you considered using Git and a set of
>> > > branches
>> > > > > for
>> > > > > > > > > >> > implementing
>> > > > > > > > > >> > > > >>>> such staging? E.g. you have a git repo and
>> you
>> > > plan
>> > > > > to
>> > > > > > > make
>> > > > > > > > > >> > changes.
>> > > > > > > > > >> > > > >>>> Then you would open a PR for the change and
>> > merge
>> > > > it
>> > > > > to
>> > > > > > > the
>> > > > > > > > > >> "main"
>> > > > > > > > > >> > > > >>>> branch - and there in your CI/CD you can
>> check
>> > > all
>> > > > > > sorts
>> > > > > > > of
>> > > > > > > > > >> static
>> > > > > > > > > >> > > > >>>> checks and tests. But then the "main" branch
>> > > might
>> > > > be
>> > > > > > > best
>> > > > > > > > > >> suited
>> > > > > > > > > >> > if
>> > > > > > > > > >> > > > >>>> implemented on the test system. Once you
>> > validate
>> > > > the
>> > > > > > > > changes
>> > > > > > > > > >> > > > >> end-to-end
>> > > > > > > > > >> > > > >>>> you could make another PR for example to a
>> > "prod"
>> > > > > > branch.
>> > > > > > > > And
>> > > > > > > > > >> if
>> > > > > > > > > >> > > your
>> > > > > > > > > >> > > > >>>> production system is only pulling Dags from
>> the
>> > > > > "prod"
>> > > > > > > > branch
>> > > > > > > > > >> then
>> > > > > > > > > >> > > you
>> > > > > > > > > >> > > > >>>> can have this merging strategy as a staging
>> > > setup.
>> > > > > > > > > >> > > > >>>>
>> > > > > > > > > >> > > > >>>> Would this resolve your PING problem? Or
>> which
>> > > > other
>> > > > > > > detail
>> > > > > > > > > in
>> > > > > > > > > >> the
>> > > > > > > > > >> > > use
>> > > > > > > > > >> > > > >>>> case would require a PIN on top of a staging
>> > > > > strategy?
>> > > > > > > > > >> > > > >>>>
>> > > > > > > > > >> > > > >>>> Jens
>> > > > > > > > > >> > > > >>>>
>> > > > > > > > > >> > > > >>>> P.S.: Have enabled your confluence account
>> > after
>> > > it
>> > > > > was
>> > > > > > > > > >> created in
>> > > > > > > > > >> > > > >> order
>> > > > > > > > > >> > > > >>>> to write to Confluence, sorry, typical
>> pitfall
>> > > > after
>> > > > > > > > account
>> > > > > > > > > >> > > creation
>> > > > > > > > > >> > > > >>>> permissions were not set. Now it should
>> work.
>> > Let
>> > > > me
>> > > > > > know
>> > > > > > > > if
>> > > > > > > > > >> not.
>> > > > > > > > > >> > > > >>>>
>> > > > > > > > > >> > > > >>>> On 19.04.26 01:40, Piyush Maheshwari wrote:
>> > > > > > > > > >> > > > >>>>> Hi everyone,
>> > > > > > > > > >> > > > >>>>> I'm a new contributor to Airflow. I'd like
>> to
>> > > > > propose
>> > > > > > a
>> > > > > > > > new
>> > > > > > > > > >> > feature
>> > > > > > > > > >> > > > >> for
>> > > > > > > > > >> > > > >>>> Airflow: DAG Version Pinning.
>> > > > > > > > > >> > > > >>>>> Building on the foundation introduced by
>> > AIP-63:
>> > > > DAG
>> > > > > > > > > >> Versioning (
>> > > > > > > > > >> > > > >>
>> > > > > > > > > >> > > >
>> > > > > > > > > >> > >
>> > > > > > > > > >> >
>> > > > > > > > > >>
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-63*3A*DAG*Versioning__;JSsr!!Ci6f514n9QsL8ck!l3ZKTOw996h9qu4NR0VT4ouUryUdk_HmXUAVPbwCHwPwn0N2CCptVdx95-V0BoRFjws9huE_1Vy-THL8jw$
>> > > > > > > > > >> > > > >>> ),
>> > > > > > > > > >> > > > >>>> this proposal aims to extend Airflow's
>> > > capabilities
>> > > > > to
>> > > > > > > > > support
>> > > > > > > > > >> > true
>> > > > > > > > > >> > > > >>>> continuous deployment (CD) gating and safer
>> > > release
>> > > > > > > cycles.
>> > > > > > > > > >> > > > >>>>> The Problem & Use Cases
>> > > > > > > > > >> > > > >>>>> Currently, the scheduler always creates
>> > DagRuns
>> > > > > using
>> > > > > > > the
>> > > > > > > > > >> latest
>> > > > > > > > > >> > > > >> parsed
>> > > > > > > > > >> > > > >>>> DagVersion. This means that the updated DAG
>> > code
>> > > is
>> > > > > > > > deployed
>> > > > > > > > > >> > (takes
>> > > > > > > > > >> > > > >>> effect)
>> > > > > > > > > >> > > > >>>> right after the dag-processor processes it.
>> > While
>> > > > > this
>> > > > > > is
>> > > > > > > > > great
>> > > > > > > > > >> > for
>> > > > > > > > > >> > > > >> rapid
>> > > > > > > > > >> > > > >>>> development, teams running business-critical
>> > > > > pipelines
>> > > > > > > > often
>> > > > > > > > > >> need
>> > > > > > > > > >> > > > >>> stricter
>> > > > > > > > > >> > > > >>>> deployment mechanisms. Specifically:
>> > > > > > > > > >> > > > >>>>>     *
>> > > > > > > > > >> > > > >>>>> Safe Deployment Gating: The ability to pin
>> a
>> > DAG
>> > > > to
>> > > > > > its
>> > > > > > > > last
>> > > > > > > > > >> > known
>> > > > > > > > > >> > > > >>>> stable version while new code is parsed in
>> the
>> > > > > > > background.
>> > > > > > > > > This
>> > > > > > > > > >> > > allows
>> > > > > > > > > >> > > > >>> the
>> > > > > > > > > >> > > > >>>> new version to be held back until it passes
>> > > > automated
>> > > > > > > > > >> regression
>> > > > > > > > > >> > > tests
>> > > > > > > > > >> > > > >> or
>> > > > > > > > > >> > > > >>>> receives explicit manual approval.
>> > > > > > > > > >> > > > >>>>>     *
>> > > > > > > > > >> > > > >>>>> Instant Rollbacks: If an issue is detected
>> in
>> > a
>> > > > > newly
>> > > > > > > > > promoted
>> > > > > > > > > >> > DAG
>> > > > > > > > > >> > > > >>>> version, users need the capability to
>> instantly
>> > > > roll
>> > > > > > back
>> > > > > > > > to
>> > > > > > > > > a
>> > > > > > > > > >> > > > previous
>> > > > > > > > > >> > > > >>>> version via the UI/API, without having to
>> > revert
>> > > > the
>> > > > > > > > > underlying
>> > > > > > > > > >> > code
>> > > > > > > > > >> > > > >> and
>> > > > > > > > > >> > > > >>>> wait for the repository sync and DAG
>> processing
>> > > > > cycle.
>> > > > > > > > > >> > > > >>>>> High-Level Proposed Solution
>> > > > > > > > > >> > > > >>>>> Introduce an optional
>> active_dag_version_id to
>> > > the
>> > > > > > > > DagModel.
>> > > > > > > > > >> This
>> > > > > > > > > >> > > > >> field
>> > > > > > > > > >> > > > >>>> can be used to pin a DAG version for
>> scheduling
>> > > and
>> > > > > > > > > execution,
>> > > > > > > > > >> > while
>> > > > > > > > > >> > > > >> the
>> > > > > > > > > >> > > > >>>> dag-processor can continue to parse and
>> > register
>> > > > > newer
>> > > > > > > DAG
>> > > > > > > > > >> > versions.
>> > > > > > > > > >> > > > >>>>>     *
>> > > > > > > > > >> > > > >>>>> When this pin is set, the scheduler and API
>> > will
>> > > > > > respect
>> > > > > > > > the
>> > > > > > > > > >> > pinned
>> > > > > > > > > >> > > > >>>> version for creating runs and executing
>> tasks,
>> > > > > > separating
>> > > > > > > > the
>> > > > > > > > > >> > > parsing
>> > > > > > > > > >> > > > >> of
>> > > > > > > > > >> > > > >>>> new code from the execution of new code.
>> > > > > > > > > >> > > > >>>>>     *
>> > > > > > > > > >> > > > >>>>> If the pin is NULL, the system defaults to
>> the
>> > > > > current
>> > > > > > > > > >> behavior
>> > > > > > > > > >> > > > >> (always
>> > > > > > > > > >> > > > >>>> executing the latest parsed version). This
>> way,
>> > > we
>> > > > > can
>> > > > > > > > > maintain
>> > > > > > > > > >> > > > >> complete
>> > > > > > > > > >> > > > >>>> backwards compatibility.
>> > > > > > > > > >> > > > >>>>> I have put together some detailed notes
>> > covering
>> > > > the
>> > > > > > > data
>> > > > > > > > > >> model
>> > > > > > > > > >> > > > >>> changes,
>> > > > > > > > > >> > > > >>>> database migrations, and edge cases with
>> this
>> > > > > approach.
>> > > > > > > If
>> > > > > > > > > >> there
>> > > > > > > > > >> > is
>> > > > > > > > > >> > > > >>> general
>> > > > > > > > > >> > > > >>>> alignment that this fits the vision for
>> > Airflow,
>> > > I
>> > > > > > would
>> > > > > > > > like
>> > > > > > > > > >> to
>> > > > > > > > > >> > > take
>> > > > > > > > > >> > > > >>> this
>> > > > > > > > > >> > > > >>>> proposal through the formal AIP review
>> process.
>> > > > > > > > > >> > > > >>>>> But I would love to get the community's
>> > feedback
>> > > > on
>> > > > > > the
>> > > > > > > > > >> feature
>> > > > > > > > > >> > and
>> > > > > > > > > >> > > > >> the
>> > > > > > > > > >> > > > >>>> high-level approach.
>> > > > > > > > > >> > > > >>>>> I'll also need someone to grant me access
>> to
>> > > > create
>> > > > > > > > content
>> > > > > > > > > on
>> > > > > > > > > >> > the
>> > > > > > > > > >> > > > >>>> Airflow Confluence wiki.
>> > > > > > > > > >> > > > >>>>> Thanks for your time!
>> > > > > > > > > >> > > > >>>>> Regards,
>> > > > > > > > > >> > > > >>>>> Piyush
>> > > > > > > > > >> > > > >>>>>
>> > > > > > > > > >> > > > >>>>
>> > > > > > > > > >> > >
>> > > > > > > > >
>> > > > >
>> ---------------------------------------------------------------------
>> > > > > > > > > >> > > > >>>> To unsubscribe, e-mail:
>> > > > > > > [email protected]
>> > > > > > > > > >> > > > >>>> For additional commands, e-mail:
>> > > > > > > > [email protected]
>> > > > > > > > > >> > > > >>>>
>> > > > > > > > > >> > > > >>>>
>> > > > > > > > > >> > > >
>> > > > > > > > > >> > > >
>> > > > > > > > > >>
>> > > > > > >
>> > > ---------------------------------------------------------------------
>> > > > > > > > > >> > > > To unsubscribe, e-mail:
>> > > > > [email protected]
>> > > > > > > > > >> > > > For additional commands, e-mail:
>> > > > > > [email protected]
>> > > > > > > > > >> > > >
>> > > > > > > > > >> > > >
>> > > > > > > > > >> > >
>> > > > > > > > > >> >
>> > > > > > > > > >>
>> > > > > > > > > >>
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
>

Reply via email to