Hi Jarek,
Thanks for reviewing the AIP and sharing the feedback here:
https://lists.apache.org/thread.html/nzd1mvg5d7cgqbtqv0sj3h276kohw2s9
Bringing the conversation back to this discussion thread. I've worked
through the points raised and updated the page accordingly.
1. Rollback only works for versions that were already pinned
Fair point. The AIP now adds an opt-in mechanism to address this gap. When
enabled, a code-only change creates a new DagVersion instead of refreshing
one that task instances reference, so every version that has run stays a
rollback target. It's disabled by default because it can add up to one
version per bundle advance in a large shared bundle, and the AIP states
what rollback covers with it disabled:
https://cwiki.apache.org/confluence/spaces/AIRFLOW/pages/421958312/AIP-109+DAG+Version+Pinning#AIP109DAGVersionPinning-DagVersionImmutability
Christos and I discussed the commit-based pin alternative thoroughly
offline and are now aligned on the proposed DagVersion pin. Christos'
comments on the page confirm this alignment.
The AIP has also been revised according to the latest comments.
2. Relation to "trigger with bundle_version"
The overlap is limited to one-off runs of healthy Dags. An explicit
bundle_version doesn't change the version used by scheduled runs or other
callers, and the trigger API rejects stale Dags and Dags with import
errors, whereas the proposed pin keeps working.
On the other hand, the adhoc trigger behavior can also be supported
through the proposed mechanism. A pin update followed by a trigger can also
mimic the behavior, but the AIP doesn't propose deprecating the other
mechanism.
The version lookup problem is also fixed at the source through the option
proposed above, so the existing lookup resolves for both features. Details:
https://cwiki.apache.org/confluence/spaces/AIRFLOW/pages/421958312/AIP-109+DAG+Version+Pinning#AIP109DAGVersionPinning-Relationtoexplicitbundle_versiontriggers
3. Divergence detection needs a home
I agree with your analysis. Worker-side detection needs a new Execution
API call, and it would need its own warning type. Upon thinking about it in
detail, I'd like to move divergence detection out of AIP-109 into a
follow-up discussion thread/AIP.
I'd like your view before I change the page. Reasons for the proposal:
- It isn't specific to pinning. An impure Dag can already diverge from its
recorded version today when a task is rerun, or upon external state changes
while a run is executing.
- The serialized Dag may not capture every Dag or operator argument, so a
hash comparison can't be exhaustive. Whether that is good enough, or a
finer signal is needed, is a discussion of its own.
- The choice between a purpose-specific (hash reconciliation) Execution
API endpoint and a more general channel for warnings observed on workers is
an Execution API and security model decision rather than a version pinning
one.
- Hashing on the worker needs the same serializer as on Dag processor, and
comparisons are only valid when both sides use the same serializer version.
If that works, I'll replace the Divergence Detection section with a
pointer to the follow-up and drop it from the completion criteria. The
limitation itself stays documented under "Execution-Time Divergence for
Impure Dags" and Out of Scope > "Task Logic Divergence for Impure Dags".
https://cwiki.apache.org/confluence/spaces/AIRFLOW/pages/421958312/AIP-109+DAG+Version+Pinning#AIP109DAGVersionPinning-Execution-TimeDivergenceforImpureDags
https://cwiki.apache.org/confluence/spaces/AIRFLOW/pages/421958312/AIP-109+DAG+Version+Pinning#AIP109DAGVersionPinning-OutofScope
4. Open review threads
- As mentioned above, Christos has replied to both threads: after we
discussed it offline, he withdrew the commit-SHA suggestion. He's also
aligned that any limitations inherited from Dag versioning shouldn't block
the feature as long as they're clearly documented. They're covered in the
Applicability, Rollback availability and Out of Scope sections.
- I had already replied to both of Ash's comments on the AIP earlier, but
I followed up again end of Sept whether those comments could be considered
addressed. Ash said they'd take a look at the AIP again. Since then,
they've suggested a more user-facing name for the AIP, which I've replied
to, but I'm waiting to hear back.
5. Two more from checking the code
- Permissions: pin changes now require a new access entity,
DagAccessEntity.ACTIVE_VERSION, separate from Dag edit. It's proposed here:
https://cwiki.apache.org/confluence/spaces/AIRFLOW/pages/421958312/AIP-109+DAG+Version+Pinning#AIP109DAGVersionPinning-AuthorizationforVersionPin/Unpin
A related gap: triggering with bundle_version today only needs the trigger
permission (POST on the RUN access entity). Changing that would affect
existing callers of #61550, so I've left it out of this AIP, but it's
essentially a loophole around the new access entity.
- db clean: cleanup will skip pinned Dag rows and pinned versions, and the
text is also updated accordingly.
https://cwiki.apache.org/confluence/spaces/AIRFLOW/pages/421958312/AIP-109+DAG+Version+Pinning#AIP109DAGVersionPinning-VersionRetention
6. Target release
Added at the top of the page: "Target release: 3.5.0".
7. Other changes since the last voting round
For anyone re-reviewing, these are the other notable edits:
- Motivation and Use cases: rephrased to make the problem and the goals
clearer. The motivation now focuses on the release risk of shared code (a
change that passes its own CI can still break Dags owned by other teams or
fail against the runtime on workers) and on parse timing deciding when code
goes live. The use cases are now owner-controlled cutovers, quick selective
recovery and tiered rollout.
- Operating Models: split out rollback availability, which now lives with
the DagVersion lifecycle changes
- Added a high-level diagram covering the involved components and
interaction edges.
- API changes: added "Resolving the current Dag definition". Endpoints
that load a Dag's current definition not in the context of a dag run or an
explicit version (like Dag details, task list, graph, grid and calendar
views, backfill dry-run) resolve the active version instead of the latest.
"Synchronous Resolution on Pin/Unpin" moved under API changes, and the
trigger API now admits pinned Dags that are stale or have import errors,
matching the scheduler.
- Dag Version Immutability: restructured as: existing lifecycle, the pin
guard, a new opt-in option, rollback availability and the note on
hash-identical versions.
- Completion criteria: updated to match all of the above.
Thanks again for the concrete list of comments, Jarek. It's helped make
the document better. Please take some time to review when possible; I'll
continue addressing any comments.
Thanks,
Piyush
On Wed, Sep 23, 2026 at 3:07 PM Piyush Maheshwari <
[email protected]> wrote:
Hi all,
This is a gentle nudge on the AIP-109 review.
For context, please check the latest vote thread and my response here:
https://lists.apache.org/thread/78nbhmmnxxz4omdwx02v42jc87snn3zt
If you have any open concerns, please share what changes or decisions we
should make before proceeding.
Aligning on the acceptance criteria would be really helpful so I can
address them and call another vote.
If anyone needs more time to review the proposal, please let me know. I’m
also happy to follow up over Slack or another convenient communication
channel to clarify outstanding points and help bring the discussion to
closure. I can summarize and post any conclusions back on this thread.
Thanks,
Piyush
On Mon, Sep 7, 2026 at 12:03 PM Piyush Maheshwari <
[email protected]> wrote:
Hi everyone,
As the Airflow summit ended last week, I've restarted a fresh vote on
the AIP today.
AIP:
https://cwiki.apache.org/confluence/spaces/AIRFLOW/pages/421958312/AIP-109+DAG+Version+Pinning
Vote thread:
https://lists.apache.org/thread/78nbhmmnxxz4omdwx02v42jc87snn3zt
Thanks again for the reviews. I've addressed all comments.
Regards,
Piyush
On Sat, Aug 22, 2026 at 9:34 PM Przemysław Mirowski <[email protected]>
wrote:
Hi Piyush,
I checked the AIP again, and it looks much better! Thanks for iterating
over this proposal. I left a few minor editorial comments, and I also saw
that Christos left comments we should address before voting.
Please also keep in mind that most of the maintainers are currently
occupied by the upcoming Airflow Summit. Personally, I would wait with
voting for a week or two after the conference to give maintainers time to
catch up.
Regards,
Przemek
On 2026/08/19 06:45:30 Piyush Maheshwari wrote:
Hi everyone
Gentle reminder to review the AIP
https://cwiki.apache.org/confluence/spaces/AIRFLOW/pages/421958312/AIP-109+DAG+Version+Pinning
.
Thank you.
On Thu, 13 Aug 2026 at 4:05 PM, Piyush Maheshwari <
[email protected]> wrote:
Hi everyone,
I'd like to request the community's review of the AIP
https://cwiki.apache.org/confluence/spaces/AIRFLOW/pages/421958312/AIP-109+DAG+Version+Pinning
.
If there are no differing opinions I'd like to take it to a vote
soon.
Also, thank you Ephraim for the re-review. I've addressed all
comments
from the latest review and updated the AIP accordingly.
Thanks and regards,
Piyush
On Fri, Aug 7, 2026 at 4:10 PM Ephraim Anierobi <
[email protected]> wrote:
Hi Piyush,
Thank you for this summary. It is accurate, and it matches where we
landed.
Thank you also for the many rounds of changes. The document is much
clearer
now than when we started.
I want to state my position plainly for the thread. My main
concern was
that a pin on a DagVersion could not hold, because the row could
change
under it and the bundle pointer could move. The pin guard fixes
both. With
that, plus the retention rule, the stale DAG change, the purity
scope, and
the divergence warning, I no longer object to the pin target. The
AIP now
says what it delivers and what it does not. That was my real goal.
One thing I want to be clear about. My review was about whether
the design
holds, not about whether the feature is worth building. Those are
two
different questions, and the second one belongs to the community.
The feature adds pin awareness in several core places: the version
write
path, db clean, the scheduler's eligibility query for import
errors and
stale DAGs, DagModel and asset field resolution in the DAG
processor, and
a
synchronous update when a pin changes. Each change is small on its
own.
Together they mean these paths must consider pins from now on.
What it gives back is per-DAG control over when a version takes
effect,
for
pure DAGs that come from versioned bundles. That is real and
useful for
the
teams who need it. Whether it is worth the added complexity in
core paths
is a judgment call, and voters should make it with both sides
stated. I am
not voting against it. I only do not want my resolved concerns to
be read
as a view on that question.
I have left my remaining comments on the document itself. They are
small
and none of them changes the design, but I think they should be
settled
before the vote.
On the vote, I agree with starting it again instead of continuing
the old
thread. The document has changed a lot. A fresh call, with a short
note on
what changed, would be fair to everyone who already voted.
Thanks again for your patience through all of this. The proposal
is in
much
better shape.
Regards,
Ephraim
On Wed, 5 Aug 2026 at 10:55, Piyush Maheshwari <
[email protected]>
wrote:
Hi everyone,
I'd like to thank Ephraim for the depth of this review.
Following the
previous email, we had a series of detailed follow-up
discussions, and I
believe we have converged. This email summarizes where we landed,
answers
the concerns from the previous email for the benefit of the
thread, and
describes the updates I have now made to the AIP. I would like
the
community's review of the updated AIP (
https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-109+DAG+Version+Pinning
)
== Precise scope of the proposal ==
The discussion sharpened what the AIP promises, and it now
states this
explicitly:
- The goal is deterministic, per-DAG control over when a new
version of
a
DAG takes effect, plus an instant, API-driven rollback to a prior
version,
without affecting other DAGs sharing the same bundle.
- A pin fixes the serialized DAG the scheduler enforces (task
set,
edges,
schedule, display fields) and the bundle (name and version) the
worker
checks out. It provides graph and code-level determinism, not
behavior-level determinism, since a task that reads live
external state
at
execution time still does so under a pin.
- Pinning guarantees are stated for DAGs that are "pure",
meaning a
given
bundle (name and version) always produces the same serialized
DAG. The
AIP
now defines this purity condition up front.
- For impure DAGs, different pinning guarantees are now
documented for
impure DAGs. A divergence between the pinned serialized DAG's
snapshot
and
the worker's fresh parse for execution-time fields is silent.
The AIP
now
explicitly documents this as "silent divergence," a limitation,
and adds
detection for it.
== Answers to the five concerns from the previous email ==
1. "The latest DagVersion is mutable, so a pin does not freeze
content."
Addressed by introducing a pin guard in the DAG Processor's
version-recording path. A pinned DagVersion is treated as
referenced, so
its serialized content is never overwritten in place; a content
change
creates a new DagVersion instead (the same path that already
runs when
task
instances reference the row). Note that the DAG processor only
mutates
the
latest DagVersion row, so a pin on an older version is already
immutable
today; the guard changes behavior only when the pinned version
is also
the
latest.
2. "A pin would not pin the source code, because bundle_version
is
refreshed in place on hash-identical parses."
Addressed by the same guard, which stops the in-place
bundle_version/version_data refresh from applying to a pinned
row. When
the
bundle advances without a serialized-DAG change while the pinned
version is
the latest, the processor instead creates one new DagVersion
carrying
the
new bundle_version, which becomes the unpinned latest; subsequent
no-change
bundle advances keep updating that new latest in place exactly
as today.
This preserves the intent of the in-place refresh (the DAG's
latest
version
keeps tracking the bundle) while the pinned version's bundle
reference
stays frozen. Since new DagRuns will also carry the pinned
bundle, and
workers resolve their bundle checkout from the DagRun, the
scheduler's
view
and the worker's code stay on the same version. Note that this
version
creation, without a hash change, occurs only per pin advance,
not once
per
commit.
3. "Version identity is often accidental (impure DAGs)."
Addressed by scoping (guarantees stated for pure DAGs, with the
purity
condition defined in the AIP) and by divergence detection, where
a
DagWarning is raised when a pinned DAG's fresh parse hash no
longer
matches
the pinned DagVersion's hash, naming the pinned version. The
check is a
no-op for pure DAGs.
4. "Cleanup can delete a pinned version and silently unpin via
ON DELETE
SET NULL."
Addressed: airflow db clean will retain any DagVersion
referenced by an
active pin, the same way it retains task-instance-referenced
versions.
5. "The deletion rule works against the main use case."
Revised: a dag_id that disappears from the latest parse would
still be
marked stale on DagModel as today, but the scheduler's
eligibility query
will admit a stale DAG while it has an active pin, so it keeps
running
on
its pinned version (parallel to the import-error exception
already in
the
AIP). Actual removal requires an explicit unpin first, after
which the
normal stale-to-deleted path applies unchanged. This turns the
defect
scenario from a silent removal into an explicit act.
== On pinning the bundle instead of the DagVersion ==
We discussed this alternative at length, in two variants, and
the AIP's
"Alternatives" section now covers both:
- A global bundle pin (keep a history of bundle versions and
freeze the
whole bundle on one) is simple and atomic, but it pins every DAG
in the
bundle at once. It cannot keep a subset of DAGs on an older
version
while
the rest advance, and per-DAG control is the primary use case,
since a
breaking change in a shared bundle typically affects only some
of the
DAGs
it emits. We see it as a potentially complementary,
coarser-grained
control
rather than a replacement.
- A per-DAG pin keyed by the bundle (name and version) is not
uniquely
resolvable to a serialized DAG for impure DAGs, since the pair
can map
to
several dag_version rows. It therefore cannot be expressed as a
foreign
key
to a specific version, and the latest matching row can resolve
differently
across invocations, which is behavior drift while supposedly
pinned. It
is
also unstable over time, because a dag_version's bundle_version
is
refreshed in place on hash-identical parses, so a previously
valid
bundle
version can stop resolving exactly when a rollback needs it.
Freezing
that
would require an immutable bundle-to-dag_version mapping, which
amounts
to
pinning the dag_version too and converges towards this proposal.
== Operating models and rollback availability ==
Ephraim raised an important point about rollback availability. An
on-demand
pin can only roll back to versions Airflow already recorded, so
a change
that never altered the serialized DAG (e.g. one confined to an
imported
helper module) has no distinct version to return to unless it was
pinned at
the time. The AIP now states this honestly under a new "Usage
Models and
Rollback Availability" section, which describes two models the
primitive
serves:
1. On-demand pinning (incident response, planned cutovers):
rollback
targets are the recorded DAG versions. Changes that altered the
serialized
DAG (and had task instances) are reachable; changes that did not
are a
stated limitation.
2. Standing pins driven by an external release orchestrator:
every DAG
always carries a pin and the orchestrator advances it per
deployment.
Every
pinned state remains an immutable rollback target, closing the
gap
above,
at the cost of operating the orchestrator. The consequences of
the same
are
also covered in the AIP.
Airflow itself only ships the pin/unpin primitive; neither model
is
mandatory to use this feature.
== Updated AIP and proposed timeline ==
I have updated AIP-109 with all of the above:
https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-109+DAG+Version+Pinning
I would appreciate the community's review of the updated
proposal. If
multiple maintainers review it and if no major open questions or
comments
remain by Wednesday, August 12, I plan to restart the vote phase.
Thanks again to Ephraim for the multiple rounds of thorough
review that
got
the proposal to this shape.
Regards,
Piyush
On Tue, Jul 28, 2026 at 3:47 PM Ephraim Anierobi <
[email protected]>
wrote:
Thank you for this proposal, and for the careful work behind
it. I
took a
second, deeper look at the document, and I want to share my
concerns,
because I believe they change the direction rather than the
details.
First, I want to recognize the real problem the AIP
identifies. The
parse
loop controls when new code becomes active, and that timing is
not
deterministic. Teams need a way to control activation time,
and they
need a
fast rollback path. This is a genuine gap, and I am glad the
AIP puts
a
spotlight on it.
After the second review, however, I believe the proposal rests
on a
misunderstanding of what a DagVersion is. A DagVersion is a
historical
record of what Airflow observed after a parse. It is an output
of the
system, not an input to it. Dag versioning exists so that
running and
past
task instances keep a faithful record of the code they ran
against.
*Bundle
versioning* is the mechanism that decides which source code a
run
uses.
The
AIP reads DagVersion as a release artifact that users can
select and
activate, and the current implementation does not support that
reading:
1. *The latest DagVersion is mutable*. When no task instance
references
it,
write_dag() replaces its serialized content, hash, and Dag
code in
place,
on the same version ID. A pin does not create a task instance
reference.
So
a user could pin the current version before a risky merge, and
the
next
parse could replace the content behind that pin. The pin would
not
give
the
determinism it promises.
2. *A pin on the current DagVersion would not pin the source
code*.
When
a
new commit does not change the serialized content,
`write_dag()`
updates
bundle_version on the latest DagVersion in place. A change to
a helper
module or an SQL file does not change the serialized hash, so
workers
would
run the new commit under the pinned version.
3. *Version identity is often accidental.* Parse-time
variability and
serializer defects can create versions that no one intended.
Airflow
ships
an inflation checker and a dedicated Dag warning type because
this
problem
is real and common. Dag factories at large scale, the AIP's
own key
example, are the most exposed. A pin would often select an
accidental
snapshot rather than a deliberate release.
4. *The proposal does not address cleanup*. Today, `airflow db
clean`
keeps
the latest version per Dag and skips only versions that task
instances
reference. An older pinned version with no task instances is
eligible
for
deletion, and the proposed ON DELETE SET NULL behavior would
then
silently
return the Dag to the latest version, exactly when the pin
matters
most.
The document does not cover this interaction.
5. *The deletion rule works against the main use case*. The
AIP states
that
removal from a new bundle version deletes the Dag, even when
it is
pinned.
A defect in a dag-factory file that changes or drops Dag IDs
removes
Dags
from the parse results. In that case, Airflow would delete the
Dag and
discard the pin, so the protection would be absent in one of
the most
common failure modes the AIP is motivated by.
These are not implementation gaps that more safeguards can
close. They
all
follow from the same source: the design asks a history table
to act
as a
deployment mechanism. Making that work would mean redefining
what
DagVersion is, and I do not think we should.
The goal itself is worth pursuing at the right layer.
Deterministic
cutover
and instant rollback are operations on the *source artifact*.
Bundle
versions already represent the source, and pinning or rolling
back at
the
bundle layer would be atomic for all Dags and all support
files at
once.
I
would be glad to help explore that direction.
Thanks again for raising this discussion. The problem is real,
and I
hope
we can solve it together on a foundation that supports it.
- Ephraim
On Mon, 27 Jul 2026 at 06:53, Piyush Maheshwari <
[email protected]>
wrote:
Thank you, Sumit.
I've started a voting thread as suggested:
https://lists.apache.org/thread/bff758c5p68v1mtzhflytn7yzv0d8l1t
Regards,
Piyush
On Fri, Jul 24, 2026 at 12:08 PM Sumit Maheshwari <
[email protected]>
wrote:
I've reviewed it again today and left some comments.
Overall, it
looks
great and would be a great feature add in Airflow. I think
you can
start
an
official voting thread on this AIP.
On Sun, Jul 19, 2026 at 7:32 PM Piyush Maheshwari <
[email protected]> wrote:
Hi everyone,
Thanks for the reviews on AIP-109 (
https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-109+DAG+Version+Pinning
).
Over the past few weeks, I implemented the feature in
our fork
(nearly
done).
Based on the insights and learnings from this process, I
have
further
refined the AIP while addressing all comments.
Please review the updated version. I'll wait to collect
some
feedback
and
then take it to a vote.
I can start contributing PRs as soon as the AIP is
approved.
Thanks and regards,
Piyush
On Wed, Jun 3, 2026 at 2:01 PM Ephraim Anierobi <
[email protected]
wrote:
Hi Piyush,
Thanks for the AIP. I’ve added a few comments to the
doc.
Please
take a
look, as I think we should iron out those areas and
fully
understand
the
implications before moving forward.
Regards
- Ephraim
On Tue, 26 May 2026 at 18:24, Przemysław Mirowski <
[email protected]
wrote:
I checked the API - +1. Thanks for writing this up!
Thanks Nathan for mentioning the #63884 PR. It is
nice
addition
(already
merged), and I think that it will be really useful
for users
who
got
used
to Airflow 2 retry mechanism. I think also that your
PR and
API-109
are
complementary as one is focusing on the versioning
behaviour
for
retrying
task, the latter is focusing on versioning behaviour
for new
Dag
Runs.
________________________________
From: Christos Bisias <[email protected]>
Sent: 25 May 2026 09:46
To: [email protected] <[email protected]>
Subject: Re: [DISCUSS] DAG Version Pinning for
Deployment
Gating
(Building
on AIP-63)
Hello,
I'm a little late on the discussion but I just came
across
the
AIP
and
I
like this idea. I've actually been thinking of
working on
something
similar, to allow people to handle a bad rollout by
reverting
to
old
code
for that run without a full slow release pipeline.
And this
covers
it.
In my opinion, this seems more like a natural step
towards
what
dag
versioning is supposed to do, than a new feature.
Thank you,
Christos
On Mon, May 25, 2026 at 8:15 AM Piyush Maheshwari <
[email protected]> wrote:
Hi everyone,
I wanted to send a gentle nudge to review AIP-109:
DAG
Version
Pinning
(
https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-109+DAG+Version+Pinning
).
If there are no major concerns, I would like to
take this
to
a
vote
soon.
Thanks,
Piyush
On Fri, May 15, 2026 at 7:44 AM Piyush Maheshwari <
[email protected]> wrote:
Thanks for the note, Sumit. Based on the
feedback, I've
drafted
an
AIP
that is now up for review.
https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-109+DAG+Version+Pinning
Would like to get the community's feedback on
the same.
Nathan, I remember seeing your work (the issue
and an
older
PR)
while
reviewing all ongoing work related to DAG
versions. I
agree
with
the
PR's
intent, although I haven't reviewed it yet.
I understand your PR makes the version-pinned
execution
behavior
of
reruns
and backfills configurable.
However, this discussion revolves around the
behavior
for
new
runs
only.
We need the capability to pin a DAG to a specific
version
for
future
runs
instead.
Hope that clarifies.
Regards,
Piyush
On Thu, May 14, 2026 at 5:11 PM Nathan Hadfield <
[email protected]>
wrote:
Hello,
I saw AIP-109 that was created in relation to
this
discussion
and
thought
I’d better mention this PR that I’ve been
working on
for a
while
and
is
close to being approved.
https://github.com/apache/airflow/pull/63884
It is very much related to the motivations
described
and
implements
the
desire for control over the behaviour when
clearing/backfilling
runs.
Happy to discuss best steps for this here or on
the PR.
Cheers,
Nathan
From: Przemysław Mirowski <[email protected]>
Date: Tuesday, 28 April 2026 at 21:54
To: [email protected] <
[email protected]>
Subject: Re: [DISCUSS] DAG Version Pinning for
Deployment
Gating
(Building on AIP-63)
This Message Is From an External Sender
This message came from outside your
organization.
P.S. In my opinion, what can be done
in/around git,
should
be
done
there. Recreation of CI/CD in any form inside of
Airflow
itself
is
something which should not be done.
I'm glad we agree on this :) I suppose we just
disagree
on
what
is
possible outside of Airflow :p
I think that we just disagree on what the issue
is,
not on
what
is
possible/should be outside of Airflow.
I think we are trying to duplicate what we
already
have
in
Git.
Not really if we are only referring to version
pinning. As
far
as
I
am
aware of how things are working, there is no
possibility
to
determine
that
Dag after e.g. deployment on 1:00:00 PM will be
exactly
parsed
and
used
since 1:01:00 PM forward. Basically, what
version
pinning
would
provide
is
the full control of the time since the given
version
will
be
used
(currently we can only have more-or-less timing
which
in
some
cases,
is
not
sufficient). The "quick revert" is the
consequence of
having
above
possibility.
Looking at the general concerns, with having
that
feature
or
not,
users
can pretty easily test things on production,
but it
just
requires
more
time
between iterations without it. IMHO it will not
change
the
need
for
Airflow-related platform teams which makes
sure, by
standards/policies,
that things are properly tested before
production
deployment.
I
think
that
assumption that some users will misuse this
feature is
true
(like
with
most
of the features really), but on the other hand
it would
provide
more
control for other users. The other solution
possibly
would
be
to
make
Dag
Processor work more on "events" instead of
"simple"
parsing
loop
(I
recall
that there was some PR couple years ago with
PoC of
that,
but
I
couldn't
quickly find it).
________________________________
From: Jarek Potiuk <[email protected]>
Sent: 28 April 2026 17:02
To: [email protected] <
[email protected]>
Subject: Re: [DISCUSS] DAG Version Pinning for
Deployment
Gating
(Building on AIP-63)
Same concerns. I think we are trying to
duplicate what
we
already
have
in
Git—branches and reverts, for example—by moving
what
should
be
managed
as
part of the development process to Airflow UI.
Almost everything you describe can be done with:
* having a dev/staging system configured
properly to
use
dev/staging
branches
* Having a process of managing development and
a proper
branching
strategy
* single git command (for example, `git revert
XXXX`
followed
by
push
to
the right branch)
J.
On Tue, Apr 28, 2026 at 10:34 AM Pierre
Jeambrun <
[email protected]
wrote:
At first glance I tend to agree with Jens and
Niko.
I understand the request, but I agree that
this
resolves
CI/CD
and
testing
issues that should probably be remain outside
Airflow.
On Mon, Apr 27, 2026 at 7:43 PM Oliveira,
Niko <
[email protected]
wrote:
Hey folks!
P.S. In my opinion, what can be done
in/around
git,
should
be
done
there. Recreation of CI/CD in any form
inside of
Airflow
itself
is
something which should not be done.
I'm glad we agree on this :) I suppose we
just
disagree
on
what
is
possible outside of Airflow :p
But at this point I will bow out of the
conversation
and
let
others
weigh
in. I'm not fully convinced any of these
requested
behaviours
require
changes to Airflow (I think that's just
masking
some
dev
ops
work).
But
also I'm not completely opposed to the
change
either,
I'm
more
on
the
fence, so if others love the feature by all
means
implement
it!
:)
Cheers,
Niko
________________________________
From: Przemysław Mirowski <
[email protected]>
Sent: Thursday, April 23, 2026 3:06 PM
To: [email protected] <
[email protected]
Subject: RE: [EXT] [DISCUSS] DAG Version
Pinning
for
Deployment
Gating
(Building on AIP-63)
CAUTION: This email originated from outside
of the
organization.
Do
not
click links or open attachments unless you
can
confirm
the
sender
and
know
the content is safe.
AVERTISSEMENT: Ce courrier électronique
provient
d’un
expéditeur
externe.
Ne cliquez sur aucun lien et n’ouvrez
aucune pièce
jointe
si
vous
ne
pouvez
pas confirmer l’identité de l’expéditeur et
si vous
n’êtes
pas
certain
que
le contenu ne présente aucun risque.
Hi,
I think that CI/CD and version pining are a
little
two
different
things
here. In a use cases with some critical
systems
involved,
the
situation
when the Dag changes the version to the
latest
without
possibility
to
determine when it will exactly happen
(CI/CD will
have
some
more-or-less
time to deploy the change, the same goes
for Dag
Processor
parsing
time)
is
rather hard to do and in some systems it
can make
change
deployment
harder
and less safe. Of course, the ideal
solution would
be
to
have
proper
non-prod environment, which is fully
representative in
comparison
to
production (in some cases exposing non-prod
to prod
data/traffic/etc.
is,
just, not an option - e.g. security), but
it is not
always
possible
to do
due to various reasons like costs,
licenses, space
and/or
vendors.
I'm
agreeing especially with point 5 of Piyush
latest
message.
Having
above
in
mind, I think that version pinning would be
a nice
addition
to
the
Dag
Versioning feature with an assumption that
it is
for
critical
Airflow
Dags
when full control of the Dags version
change time
is
required
(maybe
there
is also another way to achieve that).
P.S. In my opinion, what can be done
in/around git,
should
be
done
there.
Recreation of CI/CD in any form inside of
Airflow
itself
is
something
which
should not be done.
________________________________
From: Oliveira, Niko <[email protected]>
Sent: 23 April 2026 01:50
To: [email protected] <
[email protected]
Subject: Re: [DISCUSS] DAG Version Pinning
for
Deployment
Gating
(Building
on AIP-63)
Hey Piyush,
Thanks for your reply, I do love how
clearly it is
written
and I
see
exactly the problem you're trying to solve!
I'm still just not convinced this needs to
be done
in
Airflow,
at
least
not with a first class feature. As
interesting as I
think
your
microservice
analogy is, Airflow is not a microservice
component,
it
is a
(very,
very)
fancy cron scheduler. And I'm not sure the
complexity
is
worth
the
use
case. Since any new code added to Airflow
must be
maintained
by
this
community and we must be cautious that any
new
pieces
serves
enough
use
cases/users to make it worth it.
To me this should either be managed outside
of an
individual
Airflow
environment e.g. you have an entirely
separate
staging/gamma/dev
Airflow
environment, which is exposed to some level
of
production
traffic
(to
borrow your microservice analogy) until it
can
graduate
to
the
production
environment. And if you really need on the
fly
toggling
of a
version,
as
you say, Airflow does this quite
responsively, if
you
deploy a
new
version
of your dags it will parse and start using
that new
version
immediately
(the problem you're trying to solve can be a
benefit
here).
You
can
even
have multiple versions of your dags
deployed at
once
and
use
configuration
to control which dag directory Airflow
reads from
(or
move/symlink
Dags
in
and out of the Dags directory as needed
from a
known
good
or
pinned
source). Or use variables or some other
parameter
store
to
control
other
pieces of runtime behaviour inside the Dags
themselves.
Between
CI/CD,
dev
ops and making use of existing Airflow
primitives I
think
you
can
achieve
what you're looking for.
But as always, this is open and community
based
software,
so
I'm
happy to
disagree and commit if the rest of the
community
thinks
this
is
a
valuable
feature :)
Cheers,
Niko
________________________________
From: Piyush Maheshwari <
[email protected]>
Sent: Tuesday, April 21, 2026 10:46 PM
To: [email protected] <
[email protected]
Subject: RE: [EXT] [DISCUSS] DAG Version
Pinning
for
Deployment
Gating
(Building on AIP-63)
CAUTION: This email originated from outside
of the
organization.
Do
not
click links or open attachments unless you
can
confirm
the
sender
and
know
the content is safe.
AVERTISSEMENT: Ce courrier électronique
provient
d’un
expéditeur
externe.
Ne cliquez sur aucun lien et n’ouvrez
aucune pièce
jointe
si
vous
ne
pouvez
pas confirmer l’identité de l’expéditeur et
si vous
n’êtes
pas
certain
que
le contenu ne présente aucun risque.
Hi Ephraim, Jarek, Jens, and Niko,
Thank you for the candid feedback. I want to
clarify a
few
things,
as
I
completely agree with Jens and Niko that
"testing
in
production"
is
an
anti-pattern. That is absolutely not the
intention
here.
1. I view this as bringing standard
microservice-like
deployment
maturity
to DAGs.
Before service deployments in our org, code
is
tested
locally,
in
a
dev
environment, and via strict unit/e2e
integration
tests
before
it
ever
makes
it to main. But even after merging and
passing
those
CI
pipelines,
we
still
use load tests, pre-prod soak times, shadow
traffic,
and
gated
production
rollouts with automated rollback triggers.
Having
deployment
gates
for
the
production environment doesn't mean the
pre-merge
checks
weren't
strict
or
that the change wasn't tested beforehand --
it just
allows
us
to
place
additional safety gates for the code to take
effect,
exactly
like
in
the
service world.
2. The core issue we are trying to solve is
that
Airflow
currently
inseparably links Code Distribution (a file
arriving
on
the
dag-processor
and being parsed) with Release Activation
(the
scheduler
executing
that
code).
To extend the microservices analogy, I can
think of
the
DAG
processor
parsing all files as "building the
artifact(s),"
while
the
scheduler
and
executor acting on the DAG versions created
thereafter
as
"deploying"
or
running the changed code.
We simply want to decouple the build from
the
deployment.
This
does
not
mean that the code arriving on the
dag-processor
will
be
tested
for
the
first time straight in production. It
should've
already
passed a
set
of
checks in the CI pipeline.
3. It is also worth calling out that Airflow
already
supports
this
decoupled behavior at the run level for task
re-runs
and
mid-execution
DAG
version bumps (by pinning the version for
the rest
of
the
execution
or
the
rerun). We are simply trying to expose this
existing
capability
at
the
DAG
level so users can govern which version new
scheduled
runs
are
created
with.
4. I also agree that Airflow itself should
not be
aware
of
our
CI/CD
pipeline, nor would it manage the deployment
orchestration
or
testing.
For our requirements, I just need Airflow
to expose
APIs
to
deploy
(pin)
a
DAG version, and to remove the pin (to
restore/enable
the
default
"auto-deploy latest" behavior).
Beyond that, we intend to use an external
release
orchestrator
that
can
explicitly tell Airflow when a parsed
version is
actually
allowed
to
run.
Until that API call is made, the previously
pinned
version
remains
active.
This ensures we don't introduce assumptions
or
awareness
of
the
presence
of
any external gating mechanisms to Airflow.
Also note that the intention is to keep the
default
auto-deploy
behavior
unless a user (or a system on their behalf)
explicitly
asks
Airflow
to
pin
a DAG to a specific version.
5. Most importantly, this feature provides
an
incident
response
"rollback"
behavior. If a bad DAG version slips
through CI/CD
into
production,
either
an on-call engineer or a rollback-trigger
(airflow-external)
can
instantly
roll back to the previous pinned version
via the
API/UI
to
mitigate.
Without this, users have to revert the code
in Git
and
wait
for
the
entire
CI/CD pipeline and file-sync process to
run, which
is
often
too
slow
during
an outage.
6. Jarek - You are right, database schema
changes
can
be
discussed
later.
My intention was only to share a very brief
summary of
how I
deemed
it to
be technically feasible for early feedback.
I did
briefly
share
the
high-level use cases ("Safe Deployment
Gating" and
"Instant
Rollbacks")
in
the original mail, but I completely agree
that
aligning
on
the
UX
first
would be a good next step.
If there are no major remaining concerns
after this
response,
I
can
draft
and share an AIP to detail the UX, followed
by a
high-level
proposal,
caveats and next steps.
Thanks for your time.
Regards,
Piyush
On Tue, Apr 21, 2026 at 5:59 PM Oliveira,
Niko <
[email protected]
wrote:
I am with Jens on this one. I think we're
complicating
Airflow
to
get
around a bad practice. If stability of
your Dags
is
critical
and
they
are
highly versioned then I think as Jens
suggested
running
them
through a
pipeline that first deploys them to a dev
or
gamma
environment
which
verifies that quality of the Dags is what
you
expect.
If
something
slips
through, then it's just normal software
practices of
either
reverting
and
rolling back or rolling forward with a
fix pushed
through
the
pipeline. I
don't think Airflow should be aware of
that
process
or
opinionated
about
it.
Cheers,
Niko
------------------------------
*From:* Jens Scheffler <
[email protected]>
*Sent:* Monday, April 20, 2026 11:17 AM
*To:* [email protected] <
[email protected]>
*Subject:* RE: [EXT] [DISCUSS] DAG Version
Pinning
for
Deployment
Gating
(Building on AIP-63)
CAUTION: This email originated from
outside of
the
organization.
Do
not
click links or open attachments unless
you can
confirm
the
sender
and
know
the content is safe.
AVERTISSEMENT: Ce courrier électronique
provient
d’un
expéditeur
externe.
Ne cliquez sur aucun lien et n’ouvrez
aucune
pièce
jointe
si
vous
ne
pouvez
pas confirmer l’identité de l’expéditeur
et si
vous
n’êtes
pas
certain
que
le contenu ne présente aucun risque.
Hi,
I am still quite sceptical. Yes, if such
pinning
is
made,
then
per
Dag
a
change need to be possible via UI and
API. But I
still
see
it
as
checken-and-egg - so you want to run a
pinned
version
but
then
how
do
you test the changes (w/o moving a
version pin)?
Then
again
some
test
mode is needed or per run you need to
make a
"test
run"
with
another
version. Smells a bit like mis-using a
production
system
for
testing.
On the other hand, yes if all Dags share
the same
Git
repo
then
merging
a branch to some other will switch all
Dags at
the
same
time.
Still
you
could utilize standard Git tools and
cherry-pick
individual
changes
and
no force to always make a full rollout.
At least
80%
possible
with
standard CI/CD tools and Git.
TLDR I see the danger that instead of a
proper
CI/CD
and
test
system
such a feature might feel like you can
easily
test
on
a
production
system. Effectively it would be needed
allowing
to
start a
Dag
with
any
version to also be able to jump back as a
reversion.
Even
though,
yes,
agree, all is technically possible.
Jens
On 20.04.26 16:40, Jarek Potiuk wrote:
+1 to what Ephraim wrote. I think that
was a
natural
next
step
we
discussed, but it needs significant
refinement,
starting
with
the
actual
use cases it should serve and the UX
for user
interaction. I
think
related
database changes are pretty secondary.
Use
cases
cover
runs,
re-runs,
backfills, CI testing, rollbacks, etc.
Following
the
"documentation
first"
approach discussed in separate thread,
describing
the
context
and
intention
of what we want to achieve is much more
important
than
DB
schema
changes.
Once we know which use cases we want to
serve,
the
DB
schema
changes
and
other related items will emerge
naturally.
On Mon, Apr 20, 2026 at 3:15 PM Ephraim
Anierobi <
[email protected]>
wrote:
Hi Piyush, thanks for starting this
discussion.
I like the proposal. We can introduce
an
active
execution
version
for
"versioned bundles" and make
scheduler/API
resolve
through
it.
The
hard
part of this is making airflow able to
distinguish
the
latest
parsed
dagmodel's metadata from active
scheduling
metadata.
I
will
suggest
you
draft this in a google docs and share
for
further
discussions.
Regards
- Ephraim
On Mon, 20 Apr 2026 at 01:31, Piyush
Maheshwari <
[email protected]>
wrote:
Thanks for sharing your thoughts Jens.
be able to test it? … a Q&A/Testing
environment
to
be
able
to
sign-off
changes.
Yes, we’ve have built an isolated
airflow
environment
to
run
regression
checks before promoting to production.
As you suggested, we’re already
running both
generic
and
DAG-custom
static
checks in a CI job as a required step
to
merge
to
the
main
branch.
But then the "main" branch might be
best
suited
if
implemented on the test system
In this case, problematic commits on
“main”
can
choke
other
unrelated
changes.
So the other option would be to
revert the
problematic
commits
and
deploy
forward.
However, a key limitation with this
approach
that
remains
is
that a
commit
affecting multiple DAGs goes live for
either
all
DAGs
or
none.
Second important feature we get with
this is
instant
DAG-level
rollback
without waiting for a revert commit
to merge
and
be
picked
by
airflow.
I think DAG-level version pinning can
also
unlock
a
lot
of
flexibility
for
deployments including tiered rollouts,
auto-rollback
triggers,
timed
deployment windows and so on.
Looking forward to hear your thoughts.
Regards,
Piyush
On Sun, 19 Apr 2026 at 3:12 PM, Jens
Scheffler <
[email protected]>
wrote:
Thanks Piyush for dropping the
discussion!
I think in general QA processes are
important
and a
valid
use
case.
So
a
kind of pinning Dag versions really
is
important.
Thinking about it, if you pin the
version
...
how
would
you
then
be
able
to test it? I assume you would need
(and
should
have
or
invest
into) a
Q&A/Testing environment to be able to
sign-off
changes.
Both
in
infrastructure but also for Dag
changes.
If you are changing Dags first of
all static
checks
on
Dag
code
are
very
much proposed as well as you can
have tests
implemented
and
test
your
Dags and logic. Similar like
software a
CI/CD
system
will
be
a
good
setup. Alongside Dag changes also
have
logical
changes
that
mostly
can
only be tested in a live system and
not as
static
checks.
Have you considered using Git and a
set of
branches
for
implementing
such staging? E.g. you have a git
repo and
you
plan
to
make
changes.
Then you would open a PR for the
change and
merge
it
to
the
"main"
branch - and there in your CI/CD you
can
check
all
sorts
of
static
checks and tests. But then the
"main" branch
might
be
best
suited
if
implemented on the test system. Once
you
validate
the
changes
end-to-end
you could make another PR for
example to a
"prod"
branch.
And
if
your
production system is only pulling
Dags from
the
"prod"
branch
then
you
can have this merging strategy as a
staging
setup.
Would this resolve your PING
problem? Or
which
other
detail
in
the
use
case would require a PIN on top of a
staging
strategy?
Jens
P.S.: Have enabled your confluence
account
after
it
was
created in
order
to write to Confluence, sorry,
typical
pitfall
after
account
creation
permissions were not set. Now it
should
work.
Let
me
know
if
not.
On 19.04.26 01:40, Piyush Maheshwari
wrote:
Hi everyone,
I'm a new contributor to Airflow.
I'd like
to
propose
a
new
feature
for
Airflow: DAG Version Pinning.
Building on the foundation
introduced by
AIP-63:
DAG
Versioning (
https://urldefense.com/v3/__https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-63*3A*DAG*Versioning__;JSsr!!Ci6f514n9QsL8ck!l3ZKTOw996h9qu4NR0VT4ouUryUdk_HmXUAVPbwCHwPwn0N2CCptVdx95-V0BoRFjws9huE_1Vy-THL8jw$
),
this proposal aims to extend
Airflow's
capabilities
to
support
true
continuous deployment (CD) gating
and safer
release
cycles.
The Problem & Use Cases
Currently, the scheduler always
creates
DagRuns
using
the
latest
parsed
DagVersion. This means that the
updated DAG
code
is
deployed
(takes
effect)
right after the dag-processor
processes it.
While
this
is
great
for
rapid
development, teams running
business-critical
pipelines
often
need
stricter
deployment mechanisms. Specifically:
*
Safe Deployment Gating: The ability
to pin
a
DAG
to
its
last
known
stable version while new code is
parsed in
the
background.
This
allows
the
new version to be held back until it
passes
automated
regression
tests
or
receives explicit manual approval.
*
Instant Rollbacks: If an issue is
detected
in
a
newly
promoted
DAG
version, users need the capability to
instantly
roll
back
to
a
previous
version via the UI/API, without
having to
revert
the
underlying
code
and
wait for the repository sync and DAG
processing
cycle.
High-Level Proposed Solution
Introduce an optional
active_dag_version_id to
the
DagModel.
This
field
can be used to pin a DAG version for
scheduling
and
execution,
while
the
dag-processor can continue to parse
and
register
newer
DAG
versions.
*
When this pin is set, the scheduler
and API
will
respect
the
pinned
version for creating runs and
executing
tasks,
separating
the
parsing
of
new code from the execution of new
code.
*
If the pin is NULL, the system
defaults to
the
current
behavior
(always
executing the latest parsed
version). This
way,
we
can
maintain
complete
backwards compatibility.
I have put together some detailed
notes
covering
the
data
model
changes,
database migrations, and edge cases
with
this
approach.
If
there
is
general
alignment that this fits the vision
for
Airflow,
I
would
like
to
take
this
proposal through the formal AIP
review
process.
But I would love to get the
community's
feedback
on
the
feature
and
the
high-level approach.
I'll also need someone to grant me
access
to
create
content
on
the
Airflow Confluence wiki.
Thanks for your time!
Regards,
Piyush
---------------------------------------------------------------------
To unsubscribe, e-mail:
[email protected]
For additional commands, e-mail:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail:
[email protected]
For additional commands, e-mail:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
