Hi everyone Gentle reminder to review the AIP https://cwiki.apache.org/confluence/spaces/AIRFLOW/pages/421958312/AIP-109+DAG+Version+Pinning .
Thank you. On Thu, 13 Aug 2026 at 4:05 PM, Piyush Maheshwari < [email protected]> wrote: > Hi everyone, > I'd like to request the community's review of the AIP > https://cwiki.apache.org/confluence/spaces/AIRFLOW/pages/421958312/AIP-109+DAG+Version+Pinning > . > If there are no differing opinions I'd like to take it to a vote soon. > > Also, thank you Ephraim for the re-review. I've addressed all comments > from the latest review and updated the AIP accordingly. > > Thanks and regards, > Piyush > > On Fri, Aug 7, 2026 at 4:10 PM Ephraim Anierobi < > [email protected]> wrote: > >> Hi Piyush, >> >> Thank you for this summary. It is accurate, and it matches where we >> landed. >> Thank you also for the many rounds of changes. The document is much >> clearer >> now than when we started. >> >> I want to state my position plainly for the thread. My main concern was >> that a pin on a DagVersion could not hold, because the row could change >> under it and the bundle pointer could move. The pin guard fixes both. With >> that, plus the retention rule, the stale DAG change, the purity scope, and >> the divergence warning, I no longer object to the pin target. The AIP now >> says what it delivers and what it does not. That was my real goal. >> >> One thing I want to be clear about. My review was about whether the design >> holds, not about whether the feature is worth building. Those are two >> different questions, and the second one belongs to the community. >> >> The feature adds pin awareness in several core places: the version write >> path, db clean, the scheduler's eligibility query for import errors and >> stale DAGs, DagModel and asset field resolution in the DAG processor, and >> a >> synchronous update when a pin changes. Each change is small on its own. >> Together they mean these paths must consider pins from now on. >> >> What it gives back is per-DAG control over when a version takes effect, >> for >> pure DAGs that come from versioned bundles. That is real and useful for >> the >> teams who need it. Whether it is worth the added complexity in core paths >> is a judgment call, and voters should make it with both sides stated. I am >> not voting against it. I only do not want my resolved concerns to be read >> as a view on that question. >> >> I have left my remaining comments on the document itself. They are small >> and none of them changes the design, but I think they should be settled >> before the vote. >> >> On the vote, I agree with starting it again instead of continuing the old >> thread. The document has changed a lot. A fresh call, with a short note on >> what changed, would be fair to everyone who already voted. >> >> Thanks again for your patience through all of this. The proposal is in >> much >> better shape. >> >> Regards, >> Ephraim >> >> On Wed, 5 Aug 2026 at 10:55, Piyush Maheshwari < >> [email protected]> >> wrote: >> >> > Hi everyone, >> > >> > I'd like to thank Ephraim for the depth of this review. Following the >> > previous email, we had a series of detailed follow-up discussions, and I >> > believe we have converged. This email summarizes where we landed, >> answers >> > the concerns from the previous email for the benefit of the thread, and >> > describes the updates I have now made to the AIP. I would like the >> > community's review of the updated AIP ( >> > >> > >> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-109+DAG+Version+Pinning >> > ) >> > >> > == Precise scope of the proposal == >> > >> > The discussion sharpened what the AIP promises, and it now states this >> > explicitly: >> > >> > - The goal is deterministic, per-DAG control over when a new version of >> a >> > DAG takes effect, plus an instant, API-driven rollback to a prior >> version, >> > without affecting other DAGs sharing the same bundle. >> > >> > - A pin fixes the serialized DAG the scheduler enforces (task set, >> edges, >> > schedule, display fields) and the bundle (name and version) the worker >> > checks out. It provides graph and code-level determinism, not >> > behavior-level determinism, since a task that reads live external state >> at >> > execution time still does so under a pin. >> > >> > - Pinning guarantees are stated for DAGs that are "pure", meaning a >> given >> > bundle (name and version) always produces the same serialized DAG. The >> AIP >> > now defines this purity condition up front. >> > >> > - For impure DAGs, different pinning guarantees are now documented for >> > impure DAGs. A divergence between the pinned serialized DAG's snapshot >> and >> > the worker's fresh parse for execution-time fields is silent. The AIP >> now >> > explicitly documents this as "silent divergence," a limitation, and adds >> > detection for it. >> > >> > == Answers to the five concerns from the previous email == >> > >> > 1. "The latest DagVersion is mutable, so a pin does not freeze content." >> > Addressed by introducing a pin guard in the DAG Processor's >> > version-recording path. A pinned DagVersion is treated as referenced, so >> > its serialized content is never overwritten in place; a content change >> > creates a new DagVersion instead (the same path that already runs when >> task >> > instances reference the row). Note that the DAG processor only mutates >> the >> > latest DagVersion row, so a pin on an older version is already immutable >> > today; the guard changes behavior only when the pinned version is also >> the >> > latest. >> > >> > 2. "A pin would not pin the source code, because bundle_version is >> > refreshed in place on hash-identical parses." >> > Addressed by the same guard, which stops the in-place >> > bundle_version/version_data refresh from applying to a pinned row. When >> the >> > bundle advances without a serialized-DAG change while the pinned >> version is >> > the latest, the processor instead creates one new DagVersion carrying >> the >> > new bundle_version, which becomes the unpinned latest; subsequent >> no-change >> > bundle advances keep updating that new latest in place exactly as today. >> > This preserves the intent of the in-place refresh (the DAG's latest >> version >> > keeps tracking the bundle) while the pinned version's bundle reference >> > stays frozen. Since new DagRuns will also carry the pinned bundle, and >> > workers resolve their bundle checkout from the DagRun, the scheduler's >> view >> > and the worker's code stay on the same version. Note that this version >> > creation, without a hash change, occurs only per pin advance, not once >> per >> > commit. >> > >> > 3. "Version identity is often accidental (impure DAGs)." >> > Addressed by scoping (guarantees stated for pure DAGs, with the purity >> > condition defined in the AIP) and by divergence detection, where a >> > DagWarning is raised when a pinned DAG's fresh parse hash no longer >> matches >> > the pinned DagVersion's hash, naming the pinned version. The check is a >> > no-op for pure DAGs. >> > >> > 4. "Cleanup can delete a pinned version and silently unpin via ON DELETE >> > SET NULL." >> > Addressed: airflow db clean will retain any DagVersion referenced by an >> > active pin, the same way it retains task-instance-referenced versions. >> > >> > 5. "The deletion rule works against the main use case." >> > Revised: a dag_id that disappears from the latest parse would still be >> > marked stale on DagModel as today, but the scheduler's eligibility query >> > will admit a stale DAG while it has an active pin, so it keeps running >> on >> > its pinned version (parallel to the import-error exception already in >> the >> > AIP). Actual removal requires an explicit unpin first, after which the >> > normal stale-to-deleted path applies unchanged. This turns the defect >> > scenario from a silent removal into an explicit act. >> > >> > == On pinning the bundle instead of the DagVersion == >> > >> > We discussed this alternative at length, in two variants, and the AIP's >> > "Alternatives" section now covers both: >> > >> > - A global bundle pin (keep a history of bundle versions and freeze the >> > whole bundle on one) is simple and atomic, but it pins every DAG in the >> > bundle at once. It cannot keep a subset of DAGs on an older version >> while >> > the rest advance, and per-DAG control is the primary use case, since a >> > breaking change in a shared bundle typically affects only some of the >> DAGs >> > it emits. We see it as a potentially complementary, coarser-grained >> control >> > rather than a replacement. >> > >> > - A per-DAG pin keyed by the bundle (name and version) is not uniquely >> > resolvable to a serialized DAG for impure DAGs, since the pair can map >> to >> > several dag_version rows. It therefore cannot be expressed as a foreign >> key >> > to a specific version, and the latest matching row can resolve >> differently >> > across invocations, which is behavior drift while supposedly pinned. It >> is >> > also unstable over time, because a dag_version's bundle_version is >> > refreshed in place on hash-identical parses, so a previously valid >> bundle >> > version can stop resolving exactly when a rollback needs it. Freezing >> that >> > would require an immutable bundle-to-dag_version mapping, which amounts >> to >> > pinning the dag_version too and converges towards this proposal. >> > >> > == Operating models and rollback availability == >> > >> > Ephraim raised an important point about rollback availability. An >> on-demand >> > pin can only roll back to versions Airflow already recorded, so a change >> > that never altered the serialized DAG (e.g. one confined to an imported >> > helper module) has no distinct version to return to unless it was >> pinned at >> > the time. The AIP now states this honestly under a new "Usage Models and >> > Rollback Availability" section, which describes two models the primitive >> > serves: >> > >> > 1. On-demand pinning (incident response, planned cutovers): rollback >> > targets are the recorded DAG versions. Changes that altered the >> serialized >> > DAG (and had task instances) are reachable; changes that did not are a >> > stated limitation. >> > 2. Standing pins driven by an external release orchestrator: every DAG >> > always carries a pin and the orchestrator advances it per deployment. >> Every >> > pinned state remains an immutable rollback target, closing the gap >> above, >> > at the cost of operating the orchestrator. The consequences of the same >> are >> > also covered in the AIP. >> > >> > Airflow itself only ships the pin/unpin primitive; neither model is >> > mandatory to use this feature. >> > >> > == Updated AIP and proposed timeline == >> > >> > I have updated AIP-109 with all of the above: >> > >> > >> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-109+DAG+Version+Pinning >> > >> > I would appreciate the community's review of the updated proposal. If >> > multiple maintainers review it and if no major open questions or >> comments >> > remain by Wednesday, August 12, I plan to restart the vote phase. >> > >> > Thanks again to Ephraim for the multiple rounds of thorough review that >> got >> > the proposal to this shape. >> > >> > Regards, >> > Piyush >> > >> > On Tue, Jul 28, 2026 at 3:47 PM Ephraim Anierobi < >> > [email protected]> >> > wrote: >> > >> > > Thank you for this proposal, and for the careful work behind it. I >> took a >> > > second, deeper look at the document, and I want to share my concerns, >> > > because I believe they change the direction rather than the details. >> > > >> > > First, I want to recognize the real problem the AIP identifies. The >> parse >> > > loop controls when new code becomes active, and that timing is not >> > > deterministic. Teams need a way to control activation time, and they >> > need a >> > > fast rollback path. This is a genuine gap, and I am glad the AIP puts >> a >> > > spotlight on it. >> > > >> > > After the second review, however, I believe the proposal rests on a >> > > misunderstanding of what a DagVersion is. A DagVersion is a historical >> > > record of what Airflow observed after a parse. It is an output of the >> > > system, not an input to it. Dag versioning exists so that running and >> > past >> > > task instances keep a faithful record of the code they ran against. >> > *Bundle >> > > versioning* is the mechanism that decides which source code a run >> uses. >> > The >> > > AIP reads DagVersion as a release artifact that users can select and >> > > activate, and the current implementation does not support that >> reading: >> > > >> > > 1. *The latest DagVersion is mutable*. When no task instance >> references >> > it, >> > > write_dag() replaces its serialized content, hash, and Dag code in >> place, >> > > on the same version ID. A pin does not create a task instance >> reference. >> > So >> > > a user could pin the current version before a risky merge, and the >> next >> > > parse could replace the content behind that pin. The pin would not >> give >> > the >> > > determinism it promises. >> > > >> > > 2. *A pin on the current DagVersion would not pin the source code*. >> When >> > a >> > > new commit does not change the serialized content, `write_dag()` >> updates >> > > bundle_version on the latest DagVersion in place. A change to a helper >> > > module or an SQL file does not change the serialized hash, so workers >> > would >> > > run the new commit under the pinned version. >> > > >> > > 3. *Version identity is often accidental.* Parse-time variability and >> > > serializer defects can create versions that no one intended. Airflow >> > ships >> > > an inflation checker and a dedicated Dag warning type because this >> > problem >> > > is real and common. Dag factories at large scale, the AIP's own key >> > > example, are the most exposed. A pin would often select an accidental >> > > snapshot rather than a deliberate release. >> > > >> > > 4. *The proposal does not address cleanup*. Today, `airflow db clean` >> > keeps >> > > the latest version per Dag and skips only versions that task instances >> > > reference. An older pinned version with no task instances is eligible >> for >> > > deletion, and the proposed ON DELETE SET NULL behavior would then >> > silently >> > > return the Dag to the latest version, exactly when the pin matters >> most. >> > > The document does not cover this interaction. >> > > >> > > 5. *The deletion rule works against the main use case*. The AIP states >> > that >> > > removal from a new bundle version deletes the Dag, even when it is >> > pinned. >> > > A defect in a dag-factory file that changes or drops Dag IDs removes >> Dags >> > > from the parse results. In that case, Airflow would delete the Dag and >> > > discard the pin, so the protection would be absent in one of the most >> > > common failure modes the AIP is motivated by. >> > > >> > > These are not implementation gaps that more safeguards can close. They >> > all >> > > follow from the same source: the design asks a history table to act >> as a >> > > deployment mechanism. Making that work would mean redefining what >> > > DagVersion is, and I do not think we should. >> > > >> > > The goal itself is worth pursuing at the right layer. Deterministic >> > cutover >> > > and instant rollback are operations on the *source artifact*. Bundle >> > > versions already represent the source, and pinning or rolling back at >> the >> > > bundle layer would be atomic for all Dags and all support files at >> once. >> > I >> > > would be glad to help explore that direction. >> > > >> > > Thanks again for raising this discussion. The problem is real, and I >> hope >> > > we can solve it together on a foundation that supports it. >> > > >> > > - Ephraim >> > > >> > > On Mon, 27 Jul 2026 at 06:53, Piyush Maheshwari < >> > > [email protected]> >> > > wrote: >> > > >> > > > Thank you, Sumit. >> > > > >> > > > I've started a voting thread as suggested: >> > > > https://lists.apache.org/thread/bff758c5p68v1mtzhflytn7yzv0d8l1t >> > > > >> > > > Regards, >> > > > Piyush >> > > > >> > > > On Fri, Jul 24, 2026 at 12:08 PM Sumit Maheshwari < >> > > [email protected]> >> > > > wrote: >> > > > >> > > > > I've reviewed it again today and left some comments. Overall, it >> > looks >> > > > > great and would be a great feature add in Airflow. I think you can >> > > start >> > > > an >> > > > > official voting thread on this AIP. >> > > > > >> > > > > >> > > > > On Sun, Jul 19, 2026 at 7:32 PM Piyush Maheshwari < >> > > > > [email protected]> wrote: >> > > > > >> > > > > > Hi everyone, >> > > > > > Thanks for the reviews on AIP-109 ( >> > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-109+DAG+Version+Pinning >> > > > > > ). >> > > > > > Over the past few weeks, I implemented the feature in our fork >> > > (nearly >> > > > > > done). >> > > > > > Based on the insights and learnings from this process, I have >> > further >> > > > > > refined the AIP while addressing all comments. >> > > > > > >> > > > > > Please review the updated version. I'll wait to collect some >> > feedback >> > > > and >> > > > > > then take it to a vote. >> > > > > > I can start contributing PRs as soon as the AIP is approved. >> > > > > > >> > > > > > Thanks and regards, >> > > > > > Piyush >> > > > > > >> > > > > > On Wed, Jun 3, 2026 at 2:01 PM Ephraim Anierobi < >> > > > > [email protected] >> > > > > > > >> > > > > > wrote: >> > > > > > >> > > > > > > Hi Piyush, >> > > > > > > >> > > > > > > Thanks for the AIP. I’ve added a few comments to the doc. >> Please >> > > > take a >> > > > > > > look, as I think we should iron out those areas and fully >> > > understand >> > > > > the >> > > > > > > implications before moving forward. >> > > > > > > >> > > > > > > Regards >> > > > > > > - Ephraim >> > > > > > > >> > > > > > > On Tue, 26 May 2026 at 18:24, Przemysław Mirowski < >> > > [email protected] >> > > > > >> > > > > > > wrote: >> > > > > > > >> > > > > > > > I checked the API - +1. Thanks for writing this up! >> > > > > > > > >> > > > > > > > Thanks Nathan for mentioning the #63884 PR. It is nice >> addition >> > > > > > (already >> > > > > > > > merged), and I think that it will be really useful for users >> > who >> > > > got >> > > > > > used >> > > > > > > > to Airflow 2 retry mechanism. I think also that your PR and >> > > API-109 >> > > > > are >> > > > > > > > complementary as one is focusing on the versioning behaviour >> > for >> > > > > > retrying >> > > > > > > > task, the latter is focusing on versioning behaviour for new >> > Dag >> > > > > Runs. >> > > > > > > > ________________________________ >> > > > > > > > From: Christos Bisias <[email protected]> >> > > > > > > > Sent: 25 May 2026 09:46 >> > > > > > > > To: [email protected] <[email protected]> >> > > > > > > > Subject: Re: [DISCUSS] DAG Version Pinning for Deployment >> > Gating >> > > > > > > (Building >> > > > > > > > on AIP-63) >> > > > > > > > >> > > > > > > > Hello, >> > > > > > > > >> > > > > > > > I'm a little late on the discussion but I just came across >> the >> > > AIP >> > > > > and >> > > > > > I >> > > > > > > > like this idea. I've actually been thinking of working on >> > > something >> > > > > > > > similar, to allow people to handle a bad rollout by >> reverting >> > to >> > > > old >> > > > > > code >> > > > > > > > for that run without a full slow release pipeline. And this >> > > covers >> > > > > it. >> > > > > > > > >> > > > > > > > In my opinion, this seems more like a natural step towards >> what >> > > dag >> > > > > > > > versioning is supposed to do, than a new feature. >> > > > > > > > >> > > > > > > > Thank you, >> > > > > > > > Christos >> > > > > > > > >> > > > > > > > >> > > > > > > > On Mon, May 25, 2026 at 8:15 AM Piyush Maheshwari < >> > > > > > > > [email protected]> wrote: >> > > > > > > > >> > > > > > > > > Hi everyone, >> > > > > > > > > I wanted to send a gentle nudge to review AIP-109: DAG >> > Version >> > > > > > Pinning >> > > > > > > ( >> > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-109+DAG+Version+Pinning >> > > > > > > > > ). >> > > > > > > > > If there are no major concerns, I would like to take this >> to >> > a >> > > > vote >> > > > > > > soon. >> > > > > > > > > >> > > > > > > > > Thanks, >> > > > > > > > > Piyush >> > > > > > > > > >> > > > > > > > > On Fri, May 15, 2026 at 7:44 AM Piyush Maheshwari < >> > > > > > > > > [email protected]> wrote: >> > > > > > > > > >> > > > > > > > > > Thanks for the note, Sumit. Based on the feedback, I've >> > > drafted >> > > > > an >> > > > > > > AIP >> > > > > > > > > > that is now up for review. >> > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-109+DAG+Version+Pinning >> > > > > > > > > > >> > > > > > > > > > Would like to get the community's feedback on the same. >> > > > > > > > > > >> > > > > > > > > > Nathan, I remember seeing your work (the issue and an >> older >> > > PR) >> > > > > > while >> > > > > > > > > > reviewing all ongoing work related to DAG versions. I >> agree >> > > > with >> > > > > > the >> > > > > > > > PR's >> > > > > > > > > > intent, although I haven't reviewed it yet. >> > > > > > > > > > I understand your PR makes the version-pinned execution >> > > > behavior >> > > > > of >> > > > > > > > > reruns >> > > > > > > > > > and backfills configurable. >> > > > > > > > > > However, this discussion revolves around the behavior >> for >> > new >> > > > > runs >> > > > > > > > only. >> > > > > > > > > > We need the capability to pin a DAG to a specific >> version >> > for >> > > > > > future >> > > > > > > > runs >> > > > > > > > > > instead. >> > > > > > > > > > Hope that clarifies. >> > > > > > > > > > >> > > > > > > > > > Regards, >> > > > > > > > > > Piyush >> > > > > > > > > > >> > > > > > > > > > On Thu, May 14, 2026 at 5:11 PM Nathan Hadfield < >> > > > > > > > > [email protected]> >> > > > > > > > > > wrote: >> > > > > > > > > > >> > > > > > > > > >> Hello, >> > > > > > > > > >> >> > > > > > > > > >> I saw AIP-109 that was created in relation to this >> > > discussion >> > > > > and >> > > > > > > > > thought >> > > > > > > > > >> I’d better mention this PR that I’ve been working on >> for a >> > > > while >> > > > > > and >> > > > > > > > is >> > > > > > > > > >> close to being approved. >> > > > > > > > > >> >> > > > > > > > > >> https://github.com/apache/airflow/pull/63884 >> > > > > > > > > >> >> > > > > > > > > >> It is very much related to the motivations described >> and >> > > > > > implements >> > > > > > > > the >> > > > > > > > > >> desire for control over the behaviour when >> > > > clearing/backfilling >> > > > > > > runs. >> > > > > > > > > >> >> > > > > > > > > >> Happy to discuss best steps for this here or on the PR. >> > > > > > > > > >> >> > > > > > > > > >> Cheers, >> > > > > > > > > >> >> > > > > > > > > >> Nathan >> > > > > > > > > >> >> > > > > > > > > >> From: Przemysław Mirowski <[email protected]> >> > > > > > > > > >> Date: Tuesday, 28 April 2026 at 21:54 >> > > > > > > > > >> To: [email protected] <[email protected]> >> > > > > > > > > >> Subject: Re: [DISCUSS] DAG Version Pinning for >> Deployment >> > > > Gating >> > > > > > > > > >> (Building on AIP-63) >> > > > > > > > > >> >> > > > > > > > > >> This Message Is From an External Sender >> > > > > > > > > >> This message came from outside your organization. >> > > > > > > > > >> >> > > > > > > > > >> >> > > > > > > > > >> > P.S. In my opinion, what can be done in/around git, >> > should >> > > > be >> > > > > > done >> > > > > > > > > >> there. Recreation of CI/CD in any form inside of >> Airflow >> > > > itself >> > > > > is >> > > > > > > > > >> something which should not be done. >> > > > > > > > > >> > I'm glad we agree on this :) I suppose we just >> disagree >> > on >> > > > > what >> > > > > > is >> > > > > > > > > >> possible outside of Airflow :p >> > > > > > > > > >> >> > > > > > > > > >> I think that we just disagree on what the issue is, >> not on >> > > > what >> > > > > is >> > > > > > > > > >> possible/should be outside of Airflow. >> > > > > > > > > >> >> > > > > > > > > >> > I think we are trying to duplicate what we already >> have >> > in >> > > > > Git. >> > > > > > > > > >> >> > > > > > > > > >> Not really if we are only referring to version >> pinning. As >> > > far >> > > > > as >> > > > > > I >> > > > > > > am >> > > > > > > > > >> aware of how things are working, there is no >> possibility >> > to >> > > > > > > determine >> > > > > > > > > that >> > > > > > > > > >> Dag after e.g. deployment on 1:00:00 PM will be exactly >> > > parsed >> > > > > and >> > > > > > > > used >> > > > > > > > > >> since 1:01:00 PM forward. Basically, what version >> pinning >> > > > would >> > > > > > > > provide >> > > > > > > > > is >> > > > > > > > > >> the full control of the time since the given version >> will >> > be >> > > > > used >> > > > > > > > > >> (currently we can only have more-or-less timing which >> in >> > > some >> > > > > > cases, >> > > > > > > > is >> > > > > > > > > not >> > > > > > > > > >> sufficient). The "quick revert" is the consequence of >> > having >> > > > > above >> > > > > > > > > >> possibility. >> > > > > > > > > >> >> > > > > > > > > >> Looking at the general concerns, with having that >> feature >> > or >> > > > > not, >> > > > > > > > users >> > > > > > > > > >> can pretty easily test things on production, but it >> just >> > > > > requires >> > > > > > > more >> > > > > > > > > time >> > > > > > > > > >> between iterations without it. IMHO it will not change >> the >> > > > need >> > > > > > for >> > > > > > > > > >> Airflow-related platform teams which makes sure, by >> > > > > > > > standards/policies, >> > > > > > > > > >> that things are properly tested before production >> > > deployment. >> > > > I >> > > > > > > think >> > > > > > > > > that >> > > > > > > > > >> assumption that some users will misuse this feature is >> > true >> > > > > (like >> > > > > > > with >> > > > > > > > > most >> > > > > > > > > >> of the features really), but on the other hand it would >> > > > provide >> > > > > > more >> > > > > > > > > >> control for other users. The other solution possibly >> would >> > > be >> > > > to >> > > > > > > make >> > > > > > > > > Dag >> > > > > > > > > >> Processor work more on "events" instead of "simple" >> > parsing >> > > > loop >> > > > > > (I >> > > > > > > > > recall >> > > > > > > > > >> that there was some PR couple years ago with PoC of >> that, >> > > but >> > > > I >> > > > > > > > couldn't >> > > > > > > > > >> quickly find it). >> > > > > > > > > >> >> > > > > > > > > >> ________________________________ >> > > > > > > > > >> From: Jarek Potiuk <[email protected]> >> > > > > > > > > >> Sent: 28 April 2026 17:02 >> > > > > > > > > >> To: [email protected] <[email protected]> >> > > > > > > > > >> Subject: Re: [DISCUSS] DAG Version Pinning for >> Deployment >> > > > Gating >> > > > > > > > > >> (Building on AIP-63) >> > > > > > > > > >> >> > > > > > > > > >> Same concerns. I think we are trying to duplicate what >> we >> > > > > already >> > > > > > > have >> > > > > > > > > in >> > > > > > > > > >> Git—branches and reverts, for example—by moving what >> > should >> > > be >> > > > > > > managed >> > > > > > > > > as >> > > > > > > > > >> part of the development process to Airflow UI. >> > > > > > > > > >> >> > > > > > > > > >> Almost everything you describe can be done with: >> > > > > > > > > >> >> > > > > > > > > >> * having a dev/staging system configured properly to >> use >> > > > > > dev/staging >> > > > > > > > > >> branches >> > > > > > > > > >> * Having a process of managing development and a proper >> > > > > branching >> > > > > > > > > strategy >> > > > > > > > > >> * single git command (for example, `git revert XXXX` >> > > followed >> > > > by >> > > > > > > push >> > > > > > > > to >> > > > > > > > > >> the right branch) >> > > > > > > > > >> >> > > > > > > > > >> J. >> > > > > > > > > >> >> > > > > > > > > >> >> > > > > > > > > >> On Tue, Apr 28, 2026 at 10:34 AM Pierre Jeambrun < >> > > > > > > > [email protected] >> > > > > > > > > > >> > > > > > > > > >> wrote: >> > > > > > > > > >> >> > > > > > > > > >> > At first glance I tend to agree with Jens and Niko. >> > > > > > > > > >> > >> > > > > > > > > >> > I understand the request, but I agree that this >> resolves >> > > > CI/CD >> > > > > > and >> > > > > > > > > >> testing >> > > > > > > > > >> > issues that should probably be remain outside >> Airflow. >> > > > > > > > > >> > >> > > > > > > > > >> > On Mon, Apr 27, 2026 at 7:43 PM Oliveira, Niko < >> > > > > > > [email protected] >> > > > > > > > > >> > > > > > > > > >> > wrote: >> > > > > > > > > >> > >> > > > > > > > > >> > > Hey folks! >> > > > > > > > > >> > > >> > > > > > > > > >> > > > P.S. In my opinion, what can be done in/around >> git, >> > > > should >> > > > > > be >> > > > > > > > done >> > > > > > > > > >> > > there. Recreation of CI/CD in any form inside of >> > Airflow >> > > > > > itself >> > > > > > > is >> > > > > > > > > >> > > something which should not be done. >> > > > > > > > > >> > > >> > > > > > > > > >> > > I'm glad we agree on this :) I suppose we just >> > disagree >> > > on >> > > > > > what >> > > > > > > is >> > > > > > > > > >> > > possible outside of Airflow :p >> > > > > > > > > >> > > >> > > > > > > > > >> > > But at this point I will bow out of the >> conversation >> > and >> > > > let >> > > > > > > > others >> > > > > > > > > >> weigh >> > > > > > > > > >> > > in. I'm not fully convinced any of these requested >> > > > > behaviours >> > > > > > > > > require >> > > > > > > > > >> > > changes to Airflow (I think that's just masking >> some >> > dev >> > > > ops >> > > > > > > > work). >> > > > > > > > > >> But >> > > > > > > > > >> > > also I'm not completely opposed to the change >> either, >> > > I'm >> > > > > more >> > > > > > > on >> > > > > > > > > the >> > > > > > > > > >> > > fence, so if others love the feature by all means >> > > > implement >> > > > > > it! >> > > > > > > :) >> > > > > > > > > >> > > >> > > > > > > > > >> > > Cheers, >> > > > > > > > > >> > > Niko >> > > > > > > > > >> > > ________________________________ >> > > > > > > > > >> > > From: Przemysław Mirowski <[email protected]> >> > > > > > > > > >> > > Sent: Thursday, April 23, 2026 3:06 PM >> > > > > > > > > >> > > To: [email protected] <[email protected] >> > >> > > > > > > > > >> > > Subject: RE: [EXT] [DISCUSS] DAG Version Pinning >> for >> > > > > > Deployment >> > > > > > > > > Gating >> > > > > > > > > >> > > (Building on AIP-63) >> > > > > > > > > >> > > >> > > > > > > > > >> > > CAUTION: This email originated from outside of the >> > > > > > organization. >> > > > > > > > Do >> > > > > > > > > >> not >> > > > > > > > > >> > > click links or open attachments unless you can >> confirm >> > > the >> > > > > > > sender >> > > > > > > > > and >> > > > > > > > > >> > know >> > > > > > > > > >> > > the content is safe. >> > > > > > > > > >> > > >> > > > > > > > > >> > > >> > > > > > > > > >> > > >> > > > > > > > > >> > > AVERTISSEMENT: Ce courrier électronique provient >> d’un >> > > > > > expéditeur >> > > > > > > > > >> externe. >> > > > > > > > > >> > > Ne cliquez sur aucun lien et n’ouvrez aucune pièce >> > > jointe >> > > > si >> > > > > > > vous >> > > > > > > > ne >> > > > > > > > > >> > pouvez >> > > > > > > > > >> > > pas confirmer l’identité de l’expéditeur et si vous >> > > n’êtes >> > > > > pas >> > > > > > > > > certain >> > > > > > > > > >> > que >> > > > > > > > > >> > > le contenu ne présente aucun risque. >> > > > > > > > > >> > > >> > > > > > > > > >> > > >> > > > > > > > > >> > > >> > > > > > > > > >> > > Hi, >> > > > > > > > > >> > > >> > > > > > > > > >> > > I think that CI/CD and version pining are a little >> two >> > > > > > different >> > > > > > > > > >> things >> > > > > > > > > >> > > here. In a use cases with some critical systems >> > > involved, >> > > > > the >> > > > > > > > > >> situation >> > > > > > > > > >> > > when the Dag changes the version to the latest >> without >> > > > > > > possibility >> > > > > > > > > to >> > > > > > > > > >> > > determine when it will exactly happen (CI/CD will >> have >> > > > some >> > > > > > > > > >> more-or-less >> > > > > > > > > >> > > time to deploy the change, the same goes for Dag >> > > Processor >> > > > > > > parsing >> > > > > > > > > >> time) >> > > > > > > > > >> > is >> > > > > > > > > >> > > rather hard to do and in some systems it can make >> > change >> > > > > > > > deployment >> > > > > > > > > >> > harder >> > > > > > > > > >> > > and less safe. Of course, the ideal solution would >> be >> > to >> > > > > have >> > > > > > > > proper >> > > > > > > > > >> > > non-prod environment, which is fully >> representative in >> > > > > > > comparison >> > > > > > > > to >> > > > > > > > > >> > > production (in some cases exposing non-prod to prod >> > > > > > > > > data/traffic/etc. >> > > > > > > > > >> is, >> > > > > > > > > >> > > just, not an option - e.g. security), but it is not >> > > always >> > > > > > > > possible >> > > > > > > > > >> to do >> > > > > > > > > >> > > due to various reasons like costs, licenses, space >> > > and/or >> > > > > > > vendors. >> > > > > > > > > I'm >> > > > > > > > > >> > > agreeing especially with point 5 of Piyush latest >> > > message. >> > > > > > > Having >> > > > > > > > > >> above >> > > > > > > > > >> > in >> > > > > > > > > >> > > mind, I think that version pinning would be a nice >> > > > addition >> > > > > to >> > > > > > > the >> > > > > > > > > Dag >> > > > > > > > > >> > > Versioning feature with an assumption that it is >> for >> > > > > critical >> > > > > > > > > Airflow >> > > > > > > > > >> > Dags >> > > > > > > > > >> > > when full control of the Dags version change time >> is >> > > > > required >> > > > > > > > (maybe >> > > > > > > > > >> > there >> > > > > > > > > >> > > is also another way to achieve that). >> > > > > > > > > >> > > >> > > > > > > > > >> > > P.S. In my opinion, what can be done in/around git, >> > > should >> > > > > be >> > > > > > > done >> > > > > > > > > >> there. >> > > > > > > > > >> > > Recreation of CI/CD in any form inside of Airflow >> > itself >> > > > is >> > > > > > > > > something >> > > > > > > > > >> > which >> > > > > > > > > >> > > should not be done. >> > > > > > > > > >> > > ________________________________ >> > > > > > > > > >> > > From: Oliveira, Niko <[email protected]> >> > > > > > > > > >> > > Sent: 23 April 2026 01:50 >> > > > > > > > > >> > > To: [email protected] <[email protected] >> > >> > > > > > > > > >> > > Subject: Re: [DISCUSS] DAG Version Pinning for >> > > Deployment >> > > > > > Gating >> > > > > > > > > >> > (Building >> > > > > > > > > >> > > on AIP-63) >> > > > > > > > > >> > > >> > > > > > > > > >> > > Hey Piyush, >> > > > > > > > > >> > > >> > > > > > > > > >> > > Thanks for your reply, I do love how clearly it is >> > > written >> > > > > > and I >> > > > > > > > see >> > > > > > > > > >> > > exactly the problem you're trying to solve! >> > > > > > > > > >> > > >> > > > > > > > > >> > > I'm still just not convinced this needs to be done >> in >> > > > > Airflow, >> > > > > > > at >> > > > > > > > > >> least >> > > > > > > > > >> > > not with a first class feature. As interesting as I >> > > think >> > > > > your >> > > > > > > > > >> > microservice >> > > > > > > > > >> > > analogy is, Airflow is not a microservice >> component, >> > it >> > > > is a >> > > > > > > > (very, >> > > > > > > > > >> very) >> > > > > > > > > >> > > fancy cron scheduler. And I'm not sure the >> complexity >> > is >> > > > > worth >> > > > > > > the >> > > > > > > > > use >> > > > > > > > > >> > > case. Since any new code added to Airflow must be >> > > > maintained >> > > > > > by >> > > > > > > > this >> > > > > > > > > >> > > community and we must be cautious that any new >> pieces >> > > > serves >> > > > > > > > enough >> > > > > > > > > >> use >> > > > > > > > > >> > > cases/users to make it worth it. >> > > > > > > > > >> > > To me this should either be managed outside of an >> > > > individual >> > > > > > > > Airflow >> > > > > > > > > >> > > environment e.g. you have an entirely separate >> > > > > > staging/gamma/dev >> > > > > > > > > >> Airflow >> > > > > > > > > >> > > environment, which is exposed to some level of >> > > production >> > > > > > > traffic >> > > > > > > > > (to >> > > > > > > > > >> > > borrow your microservice analogy) until it can >> > graduate >> > > to >> > > > > the >> > > > > > > > > >> production >> > > > > > > > > >> > > environment. And if you really need on the fly >> > toggling >> > > > of a >> > > > > > > > > version, >> > > > > > > > > >> as >> > > > > > > > > >> > > you say, Airflow does this quite responsively, if >> you >> > > > > deploy a >> > > > > > > new >> > > > > > > > > >> > version >> > > > > > > > > >> > > of your dags it will parse and start using that new >> > > > version >> > > > > > > > > >> immediately >> > > > > > > > > >> > > (the problem you're trying to solve can be a >> benefit >> > > > here). >> > > > > > You >> > > > > > > > can >> > > > > > > > > >> even >> > > > > > > > > >> > > have multiple versions of your dags deployed at >> once >> > and >> > > > use >> > > > > > > > > >> > configuration >> > > > > > > > > >> > > to control which dag directory Airflow reads from >> (or >> > > > > > > move/symlink >> > > > > > > > > >> Dags >> > > > > > > > > >> > in >> > > > > > > > > >> > > and out of the Dags directory as needed from a >> known >> > > good >> > > > or >> > > > > > > > pinned >> > > > > > > > > >> > > source). Or use variables or some other parameter >> > store >> > > to >> > > > > > > control >> > > > > > > > > >> other >> > > > > > > > > >> > > pieces of runtime behaviour inside the Dags >> > themselves. >> > > > > > Between >> > > > > > > > > CI/CD, >> > > > > > > > > >> > dev >> > > > > > > > > >> > > ops and making use of existing Airflow primitives I >> > > think >> > > > > you >> > > > > > > can >> > > > > > > > > >> achieve >> > > > > > > > > >> > > what you're looking for. >> > > > > > > > > >> > > >> > > > > > > > > >> > > But as always, this is open and community based >> > > software, >> > > > so >> > > > > > I'm >> > > > > > > > > >> happy to >> > > > > > > > > >> > > disagree and commit if the rest of the community >> > thinks >> > > > this >> > > > > > is >> > > > > > > a >> > > > > > > > > >> > valuable >> > > > > > > > > >> > > feature :) >> > > > > > > > > >> > > >> > > > > > > > > >> > > Cheers, >> > > > > > > > > >> > > Niko >> > > > > > > > > >> > > ________________________________ >> > > > > > > > > >> > > From: Piyush Maheshwari <[email protected]> >> > > > > > > > > >> > > Sent: Tuesday, April 21, 2026 10:46 PM >> > > > > > > > > >> > > To: [email protected] <[email protected] >> > >> > > > > > > > > >> > > Subject: RE: [EXT] [DISCUSS] DAG Version Pinning >> for >> > > > > > Deployment >> > > > > > > > > Gating >> > > > > > > > > >> > > (Building on AIP-63) >> > > > > > > > > >> > > >> > > > > > > > > >> > > CAUTION: This email originated from outside of the >> > > > > > organization. >> > > > > > > > Do >> > > > > > > > > >> not >> > > > > > > > > >> > > click links or open attachments unless you can >> confirm >> > > the >> > > > > > > sender >> > > > > > > > > and >> > > > > > > > > >> > know >> > > > > > > > > >> > > the content is safe. >> > > > > > > > > >> > > >> > > > > > > > > >> > > >> > > > > > > > > >> > > >> > > > > > > > > >> > > AVERTISSEMENT: Ce courrier électronique provient >> d’un >> > > > > > expéditeur >> > > > > > > > > >> externe. >> > > > > > > > > >> > > Ne cliquez sur aucun lien et n’ouvrez aucune pièce >> > > jointe >> > > > si >> > > > > > > vous >> > > > > > > > ne >> > > > > > > > > >> > pouvez >> > > > > > > > > >> > > pas confirmer l’identité de l’expéditeur et si vous >> > > n’êtes >> > > > > pas >> > > > > > > > > certain >> > > > > > > > > >> > que >> > > > > > > > > >> > > le contenu ne présente aucun risque. >> > > > > > > > > >> > > >> > > > > > > > > >> > > >> > > > > > > > > >> > > >> > > > > > > > > >> > > Hi Ephraim, Jarek, Jens, and Niko, >> > > > > > > > > >> > > >> > > > > > > > > >> > > Thank you for the candid feedback. I want to >> clarify a >> > > few >> > > > > > > things, >> > > > > > > > > as >> > > > > > > > > >> I >> > > > > > > > > >> > > completely agree with Jens and Niko that "testing >> in >> > > > > > production" >> > > > > > > > is >> > > > > > > > > an >> > > > > > > > > >> > > anti-pattern. That is absolutely not the intention >> > here. >> > > > > > > > > >> > > >> > > > > > > > > >> > > 1. I view this as bringing standard >> microservice-like >> > > > > > deployment >> > > > > > > > > >> maturity >> > > > > > > > > >> > > to DAGs. >> > > > > > > > > >> > > Before service deployments in our org, code is >> tested >> > > > > locally, >> > > > > > > in >> > > > > > > > a >> > > > > > > > > >> dev >> > > > > > > > > >> > > environment, and via strict unit/e2e integration >> tests >> > > > > before >> > > > > > it >> > > > > > > > > ever >> > > > > > > > > >> > makes >> > > > > > > > > >> > > it to main. But even after merging and passing >> those >> > CI >> > > > > > > pipelines, >> > > > > > > > > we >> > > > > > > > > >> > still >> > > > > > > > > >> > > use load tests, pre-prod soak times, shadow >> traffic, >> > and >> > > > > gated >> > > > > > > > > >> production >> > > > > > > > > >> > > rollouts with automated rollback triggers. Having >> > > > deployment >> > > > > > > gates >> > > > > > > > > for >> > > > > > > > > >> > the >> > > > > > > > > >> > > production environment doesn't mean the pre-merge >> > checks >> > > > > > weren't >> > > > > > > > > >> strict >> > > > > > > > > >> > or >> > > > > > > > > >> > > that the change wasn't tested beforehand -- it just >> > > allows >> > > > > us >> > > > > > to >> > > > > > > > > place >> > > > > > > > > >> > > additional safety gates for the code to take >> effect, >> > > > exactly >> > > > > > > like >> > > > > > > > in >> > > > > > > > > >> the >> > > > > > > > > >> > > service world. >> > > > > > > > > >> > > >> > > > > > > > > >> > > 2. The core issue we are trying to solve is that >> > Airflow >> > > > > > > currently >> > > > > > > > > >> > > inseparably links Code Distribution (a file >> arriving >> > on >> > > > the >> > > > > > > > > >> dag-processor >> > > > > > > > > >> > > and being parsed) with Release Activation (the >> > scheduler >> > > > > > > executing >> > > > > > > > > >> that >> > > > > > > > > >> > > code). >> > > > > > > > > >> > > To extend the microservices analogy, I can think of >> > the >> > > > DAG >> > > > > > > > > processor >> > > > > > > > > >> > > parsing all files as "building the artifact(s)," >> while >> > > the >> > > > > > > > scheduler >> > > > > > > > > >> and >> > > > > > > > > >> > > executor acting on the DAG versions created >> thereafter >> > > as >> > > > > > > > > "deploying" >> > > > > > > > > >> or >> > > > > > > > > >> > > running the changed code. >> > > > > > > > > >> > > We simply want to decouple the build from the >> > > deployment. >> > > > > This >> > > > > > > > does >> > > > > > > > > >> not >> > > > > > > > > >> > > mean that the code arriving on the dag-processor >> will >> > be >> > > > > > tested >> > > > > > > > for >> > > > > > > > > >> the >> > > > > > > > > >> > > first time straight in production. It should've >> > already >> > > > > > passed a >> > > > > > > > set >> > > > > > > > > >> of >> > > > > > > > > >> > > checks in the CI pipeline. >> > > > > > > > > >> > > >> > > > > > > > > >> > > 3. It is also worth calling out that Airflow >> already >> > > > > supports >> > > > > > > this >> > > > > > > > > >> > > decoupled behavior at the run level for task >> re-runs >> > and >> > > > > > > > > mid-execution >> > > > > > > > > >> > DAG >> > > > > > > > > >> > > version bumps (by pinning the version for the rest >> of >> > > the >> > > > > > > > execution >> > > > > > > > > or >> > > > > > > > > >> > the >> > > > > > > > > >> > > rerun). We are simply trying to expose this >> existing >> > > > > > capability >> > > > > > > at >> > > > > > > > > the >> > > > > > > > > >> > DAG >> > > > > > > > > >> > > level so users can govern which version new >> scheduled >> > > runs >> > > > > are >> > > > > > > > > created >> > > > > > > > > >> > > with. >> > > > > > > > > >> > > >> > > > > > > > > >> > > 4. I also agree that Airflow itself should not be >> > aware >> > > of >> > > > > our >> > > > > > > > CI/CD >> > > > > > > > > >> > > pipeline, nor would it manage the deployment >> > > orchestration >> > > > > or >> > > > > > > > > testing. >> > > > > > > > > >> > > For our requirements, I just need Airflow to expose >> > APIs >> > > > to >> > > > > > > deploy >> > > > > > > > > >> (pin) >> > > > > > > > > >> > a >> > > > > > > > > >> > > DAG version, and to remove the pin (to >> restore/enable >> > > the >> > > > > > > default >> > > > > > > > > >> > > "auto-deploy latest" behavior). >> > > > > > > > > >> > > Beyond that, we intend to use an external release >> > > > > orchestrator >> > > > > > > > that >> > > > > > > > > >> can >> > > > > > > > > >> > > explicitly tell Airflow when a parsed version is >> > > actually >> > > > > > > allowed >> > > > > > > > to >> > > > > > > > > >> run. >> > > > > > > > > >> > > Until that API call is made, the previously pinned >> > > version >> > > > > > > remains >> > > > > > > > > >> > active. >> > > > > > > > > >> > > This ensures we don't introduce assumptions or >> > awareness >> > > > of >> > > > > > the >> > > > > > > > > >> presence >> > > > > > > > > >> > of >> > > > > > > > > >> > > any external gating mechanisms to Airflow. >> > > > > > > > > >> > > Also note that the intention is to keep the default >> > > > > > auto-deploy >> > > > > > > > > >> behavior >> > > > > > > > > >> > > unless a user (or a system on their behalf) >> explicitly >> > > > asks >> > > > > > > > Airflow >> > > > > > > > > to >> > > > > > > > > >> > pin >> > > > > > > > > >> > > a DAG to a specific version. >> > > > > > > > > >> > > >> > > > > > > > > >> > > 5. Most importantly, this feature provides an >> incident >> > > > > > response >> > > > > > > > > >> > "rollback" >> > > > > > > > > >> > > behavior. If a bad DAG version slips through CI/CD >> > into >> > > > > > > > production, >> > > > > > > > > >> > either >> > > > > > > > > >> > > an on-call engineer or a rollback-trigger >> > > > (airflow-external) >> > > > > > can >> > > > > > > > > >> > instantly >> > > > > > > > > >> > > roll back to the previous pinned version via the >> > API/UI >> > > to >> > > > > > > > mitigate. >> > > > > > > > > >> > > Without this, users have to revert the code in Git >> and >> > > > wait >> > > > > > for >> > > > > > > > the >> > > > > > > > > >> > entire >> > > > > > > > > >> > > CI/CD pipeline and file-sync process to run, which >> is >> > > > often >> > > > > > too >> > > > > > > > slow >> > > > > > > > > >> > during >> > > > > > > > > >> > > an outage. >> > > > > > > > > >> > > >> > > > > > > > > >> > > 6. Jarek - You are right, database schema changes >> can >> > be >> > > > > > > discussed >> > > > > > > > > >> later. >> > > > > > > > > >> > > My intention was only to share a very brief >> summary of >> > > > how I >> > > > > > > > deemed >> > > > > > > > > >> it to >> > > > > > > > > >> > > be technically feasible for early feedback. I did >> > > briefly >> > > > > > share >> > > > > > > > the >> > > > > > > > > >> > > high-level use cases ("Safe Deployment Gating" and >> > > > "Instant >> > > > > > > > > >> Rollbacks") >> > > > > > > > > >> > in >> > > > > > > > > >> > > the original mail, but I completely agree that >> > aligning >> > > on >> > > > > the >> > > > > > > UX >> > > > > > > > > >> first >> > > > > > > > > >> > > would be a good next step. >> > > > > > > > > >> > > >> > > > > > > > > >> > > If there are no major remaining concerns after this >> > > > > response, >> > > > > > I >> > > > > > > > can >> > > > > > > > > >> draft >> > > > > > > > > >> > > and share an AIP to detail the UX, followed by a >> > > > high-level >> > > > > > > > > proposal, >> > > > > > > > > >> > > caveats and next steps. >> > > > > > > > > >> > > >> > > > > > > > > >> > > Thanks for your time. >> > > > > > > > > >> > > Regards, >> > > > > > > > > >> > > Piyush >> > > > > > > > > >> > > >> > > > > > > > > >> > > On Tue, Apr 21, 2026 at 5:59 PM Oliveira, Niko < >> > > > > > > > [email protected] >> > > > > > > > > > >> > > > > > > > > >> > > wrote: >> > > > > > > > > >> > > >> > > > > > > > > >> > > > I am with Jens on this one. I think we're >> > complicating >> > > > > > Airflow >> > > > > > > > to >> > > > > > > > > >> get >> > > > > > > > > >> > > > around a bad practice. If stability of your Dags >> is >> > > > > critical >> > > > > > > and >> > > > > > > > > >> they >> > > > > > > > > >> > are >> > > > > > > > > >> > > > highly versioned then I think as Jens suggested >> > > running >> > > > > them >> > > > > > > > > >> through a >> > > > > > > > > >> > > > pipeline that first deploys them to a dev or >> gamma >> > > > > > environment >> > > > > > > > > which >> > > > > > > > > >> > > > verifies that quality of the Dags is what you >> > expect. >> > > If >> > > > > > > > something >> > > > > > > > > >> > slips >> > > > > > > > > >> > > > through, then it's just normal software >> practices of >> > > > > either >> > > > > > > > > >> reverting >> > > > > > > > > >> > and >> > > > > > > > > >> > > > rolling back or rolling forward with a fix pushed >> > > > through >> > > > > > the >> > > > > > > > > >> > pipeline. I >> > > > > > > > > >> > > > don't think Airflow should be aware of that >> process >> > or >> > > > > > > > opinionated >> > > > > > > > > >> > about >> > > > > > > > > >> > > it. >> > > > > > > > > >> > > > >> > > > > > > > > >> > > > Cheers, >> > > > > > > > > >> > > > Niko >> > > > > > > > > >> > > > ------------------------------ >> > > > > > > > > >> > > > *From:* Jens Scheffler <[email protected]> >> > > > > > > > > >> > > > *Sent:* Monday, April 20, 2026 11:17 AM >> > > > > > > > > >> > > > *To:* [email protected] < >> > [email protected]> >> > > > > > > > > >> > > > *Subject:* RE: [EXT] [DISCUSS] DAG Version >> Pinning >> > for >> > > > > > > > Deployment >> > > > > > > > > >> > Gating >> > > > > > > > > >> > > > (Building on AIP-63) >> > > > > > > > > >> > > > >> > > > > > > > > >> > > > CAUTION: This email originated from outside of >> the >> > > > > > > organization. >> > > > > > > > > Do >> > > > > > > > > >> not >> > > > > > > > > >> > > > click links or open attachments unless you can >> > confirm >> > > > the >> > > > > > > > sender >> > > > > > > > > >> and >> > > > > > > > > >> > > know >> > > > > > > > > >> > > > the content is safe. >> > > > > > > > > >> > > > >> > > > > > > > > >> > > > >> > > > > > > > > >> > > > >> > > > > > > > > >> > > > AVERTISSEMENT: Ce courrier électronique provient >> > d’un >> > > > > > > expéditeur >> > > > > > > > > >> > externe. >> > > > > > > > > >> > > > Ne cliquez sur aucun lien et n’ouvrez aucune >> pièce >> > > > jointe >> > > > > si >> > > > > > > > vous >> > > > > > > > > ne >> > > > > > > > > >> > > pouvez >> > > > > > > > > >> > > > pas confirmer l’identité de l’expéditeur et si >> vous >> > > > n’êtes >> > > > > > pas >> > > > > > > > > >> certain >> > > > > > > > > >> > > que >> > > > > > > > > >> > > > le contenu ne présente aucun risque. >> > > > > > > > > >> > > > >> > > > > > > > > >> > > > >> > > > > > > > > >> > > > >> > > > > > > > > >> > > > Hi, >> > > > > > > > > >> > > > >> > > > > > > > > >> > > > I am still quite sceptical. Yes, if such pinning >> is >> > > > made, >> > > > > > then >> > > > > > > > per >> > > > > > > > > >> Dag >> > > > > > > > > >> > a >> > > > > > > > > >> > > > change need to be possible via UI and API. But I >> > still >> > > > see >> > > > > > it >> > > > > > > as >> > > > > > > > > >> > > > checken-and-egg - so you want to run a pinned >> > version >> > > > but >> > > > > > then >> > > > > > > > how >> > > > > > > > > >> do >> > > > > > > > > >> > > > you test the changes (w/o moving a version pin)? >> > Then >> > > > > again >> > > > > > > some >> > > > > > > > > >> test >> > > > > > > > > >> > > > mode is needed or per run you need to make a >> "test >> > > run" >> > > > > with >> > > > > > > > > another >> > > > > > > > > >> > > > version. Smells a bit like mis-using a production >> > > system >> > > > > for >> > > > > > > > > >> testing. >> > > > > > > > > >> > > > >> > > > > > > > > >> > > > On the other hand, yes if all Dags share the same >> > Git >> > > > repo >> > > > > > > then >> > > > > > > > > >> merging >> > > > > > > > > >> > > > a branch to some other will switch all Dags at >> the >> > > same >> > > > > > time. >> > > > > > > > > Still >> > > > > > > > > >> you >> > > > > > > > > >> > > > could utilize standard Git tools and cherry-pick >> > > > > individual >> > > > > > > > > changes >> > > > > > > > > >> and >> > > > > > > > > >> > > > no force to always make a full rollout. At least >> 80% >> > > > > > possible >> > > > > > > > with >> > > > > > > > > >> > > > standard CI/CD tools and Git. >> > > > > > > > > >> > > > >> > > > > > > > > >> > > > TLDR I see the danger that instead of a proper >> CI/CD >> > > and >> > > > > > test >> > > > > > > > > system >> > > > > > > > > >> > > > such a feature might feel like you can easily >> test >> > on >> > > a >> > > > > > > > production >> > > > > > > > > >> > > > system. Effectively it would be needed allowing >> to >> > > > start a >> > > > > > Dag >> > > > > > > > > with >> > > > > > > > > >> any >> > > > > > > > > >> > > > version to also be able to jump back as a >> reversion. >> > > > Even >> > > > > > > > though, >> > > > > > > > > >> yes, >> > > > > > > > > >> > > > agree, all is technically possible. >> > > > > > > > > >> > > > >> > > > > > > > > >> > > > Jens >> > > > > > > > > >> > > > >> > > > > > > > > >> > > > On 20.04.26 16:40, Jarek Potiuk wrote: >> > > > > > > > > >> > > > > +1 to what Ephraim wrote. I think that was a >> > natural >> > > > > next >> > > > > > > step >> > > > > > > > > we >> > > > > > > > > >> > > > > discussed, but it needs significant refinement, >> > > > starting >> > > > > > > with >> > > > > > > > > the >> > > > > > > > > >> > > actual >> > > > > > > > > >> > > > > use cases it should serve and the UX for user >> > > > > > interaction. I >> > > > > > > > > think >> > > > > > > > > >> > > > related >> > > > > > > > > >> > > > > database changes are pretty secondary. Use >> cases >> > > cover >> > > > > > runs, >> > > > > > > > > >> re-runs, >> > > > > > > > > >> > > > > backfills, CI testing, rollbacks, etc. >> Following >> > the >> > > > > > > > > >> "documentation >> > > > > > > > > >> > > > first" >> > > > > > > > > >> > > > > approach discussed in separate thread, >> describing >> > > the >> > > > > > > context >> > > > > > > > > and >> > > > > > > > > >> > > > intention >> > > > > > > > > >> > > > > of what we want to achieve is much more >> important >> > > than >> > > > > DB >> > > > > > > > schema >> > > > > > > > > >> > > changes. >> > > > > > > > > >> > > > > Once we know which use cases we want to serve, >> the >> > > DB >> > > > > > schema >> > > > > > > > > >> changes >> > > > > > > > > >> > > and >> > > > > > > > > >> > > > > other related items will emerge naturally. >> > > > > > > > > >> > > > > >> > > > > > > > > >> > > > > On Mon, Apr 20, 2026 at 3:15 PM Ephraim >> Anierobi < >> > > > > > > > > >> > > > [email protected]> >> > > > > > > > > >> > > > > wrote: >> > > > > > > > > >> > > > > >> > > > > > > > > >> > > > >> Hi Piyush, thanks for starting this >> discussion. >> > > > > > > > > >> > > > >> >> > > > > > > > > >> > > > >> I like the proposal. We can introduce an >> active >> > > > > execution >> > > > > > > > > version >> > > > > > > > > >> > for >> > > > > > > > > >> > > > >> "versioned bundles" and make scheduler/API >> > resolve >> > > > > > through >> > > > > > > > it. >> > > > > > > > > >> The >> > > > > > > > > >> > > hard >> > > > > > > > > >> > > > >> part of this is making airflow able to >> > distinguish >> > > > the >> > > > > > > latest >> > > > > > > > > >> parsed >> > > > > > > > > >> > > > >> dagmodel's metadata from active scheduling >> > > metadata. >> > > > I >> > > > > > will >> > > > > > > > > >> suggest >> > > > > > > > > >> > > you >> > > > > > > > > >> > > > >> draft this in a google docs and share for >> further >> > > > > > > > discussions. >> > > > > > > > > >> > > > >> >> > > > > > > > > >> > > > >> Regards >> > > > > > > > > >> > > > >> - Ephraim >> > > > > > > > > >> > > > >> >> > > > > > > > > >> > > > >> On Mon, 20 Apr 2026 at 01:31, Piyush >> Maheshwari < >> > > > > > > > > >> > > [email protected]> >> > > > > > > > > >> > > > >> wrote: >> > > > > > > > > >> > > > >> >> > > > > > > > > >> > > > >>> Thanks for sharing your thoughts Jens. >> > > > > > > > > >> > > > >>> >> > > > > > > > > >> > > > >>>> be able to test it? … a Q&A/Testing >> environment >> > > to >> > > > be >> > > > > > > able >> > > > > > > > to >> > > > > > > > > >> > > sign-off >> > > > > > > > > >> > > > >>> changes. >> > > > > > > > > >> > > > >>> Yes, we’ve have built an isolated airflow >> > > > environment >> > > > > to >> > > > > > > run >> > > > > > > > > >> > > regression >> > > > > > > > > >> > > > >>> checks before promoting to production. >> > > > > > > > > >> > > > >>> >> > > > > > > > > >> > > > >>> As you suggested, we’re already running both >> > > generic >> > > > > and >> > > > > > > > > >> DAG-custom >> > > > > > > > > >> > > > >> static >> > > > > > > > > >> > > > >>> checks in a CI job as a required step to >> merge >> > to >> > > > the >> > > > > > main >> > > > > > > > > >> branch. >> > > > > > > > > >> > > > >>> >> > > > > > > > > >> > > > >>>> But then the "main" branch might be best >> suited >> > > if >> > > > > > > > > >> > > > >>> implemented on the test system >> > > > > > > > > >> > > > >>> In this case, problematic commits on “main” >> can >> > > > choke >> > > > > > > other >> > > > > > > > > >> > unrelated >> > > > > > > > > >> > > > >>> changes. >> > > > > > > > > >> > > > >>> So the other option would be to revert the >> > > > problematic >> > > > > > > > commits >> > > > > > > > > >> and >> > > > > > > > > >> > > > deploy >> > > > > > > > > >> > > > >>> forward. >> > > > > > > > > >> > > > >>> >> > > > > > > > > >> > > > >>> However, a key limitation with this approach >> > that >> > > > > > remains >> > > > > > > is >> > > > > > > > > >> that a >> > > > > > > > > >> > > > >> commit >> > > > > > > > > >> > > > >>> affecting multiple DAGs goes live for either >> all >> > > > DAGs >> > > > > or >> > > > > > > > none. >> > > > > > > > > >> > > > >>> >> > > > > > > > > >> > > > >>> Second important feature we get with this is >> > > instant >> > > > > > > > DAG-level >> > > > > > > > > >> > > rollback >> > > > > > > > > >> > > > >>> without waiting for a revert commit to merge >> and >> > > be >> > > > > > picked >> > > > > > > > by >> > > > > > > > > >> > > airflow. >> > > > > > > > > >> > > > >>> >> > > > > > > > > >> > > > >>> I think DAG-level version pinning can also >> > unlock >> > > a >> > > > > lot >> > > > > > of >> > > > > > > > > >> > > flexibility >> > > > > > > > > >> > > > >> for >> > > > > > > > > >> > > > >>> deployments including tiered rollouts, >> > > auto-rollback >> > > > > > > > triggers, >> > > > > > > > > >> > timed >> > > > > > > > > >> > > > >>> deployment windows and so on. >> > > > > > > > > >> > > > >>> >> > > > > > > > > >> > > > >>> Looking forward to hear your thoughts. >> > > > > > > > > >> > > > >>> Regards, >> > > > > > > > > >> > > > >>> Piyush >> > > > > > > > > >> > > > >>> >> > > > > > > > > >> > > > >>> On Sun, 19 Apr 2026 at 3:12 PM, Jens >> Scheffler < >> > > > > > > > > >> > [email protected]> >> > > > > > > > > >> > > > >>> wrote: >> > > > > > > > > >> > > > >>> >> > > > > > > > > >> > > > >>>> Thanks Piyush for dropping the discussion! >> > > > > > > > > >> > > > >>>> >> > > > > > > > > >> > > > >>>> I think in general QA processes are >> important >> > > and a >> > > > > > valid >> > > > > > > > use >> > > > > > > > > >> > case. >> > > > > > > > > >> > > So >> > > > > > > > > >> > > > >> a >> > > > > > > > > >> > > > >>>> kind of pinning Dag versions really is >> > important. >> > > > > > > > > >> > > > >>>> >> > > > > > > > > >> > > > >>>> Thinking about it, if you pin the version >> ... >> > how >> > > > > would >> > > > > > > you >> > > > > > > > > >> then >> > > > > > > > > >> > be >> > > > > > > > > >> > > > >> able >> > > > > > > > > >> > > > >>>> to test it? I assume you would need (and >> should >> > > > have >> > > > > or >> > > > > > > > > invest >> > > > > > > > > >> > > into) a >> > > > > > > > > >> > > > >>>> Q&A/Testing environment to be able to >> sign-off >> > > > > changes. >> > > > > > > > Both >> > > > > > > > > in >> > > > > > > > > >> > > > >>>> infrastructure but also for Dag changes. >> > > > > > > > > >> > > > >>>> >> > > > > > > > > >> > > > >>>> If you are changing Dags first of all static >> > > checks >> > > > > on >> > > > > > > Dag >> > > > > > > > > code >> > > > > > > > > >> > are >> > > > > > > > > >> > > > >> very >> > > > > > > > > >> > > > >>>> much proposed as well as you can have tests >> > > > > implemented >> > > > > > > and >> > > > > > > > > >> test >> > > > > > > > > >> > > your >> > > > > > > > > >> > > > >>>> Dags and logic. Similar like software a >> CI/CD >> > > > system >> > > > > > will >> > > > > > > > be >> > > > > > > > > a >> > > > > > > > > >> > good >> > > > > > > > > >> > > > >>>> setup. Alongside Dag changes also have >> logical >> > > > > changes >> > > > > > > that >> > > > > > > > > >> mostly >> > > > > > > > > >> > > can >> > > > > > > > > >> > > > >>>> only be tested in a live system and not as >> > static >> > > > > > checks. >> > > > > > > > > >> > > > >>>> >> > > > > > > > > >> > > > >>>> Have you considered using Git and a set of >> > > branches >> > > > > for >> > > > > > > > > >> > implementing >> > > > > > > > > >> > > > >>>> such staging? E.g. you have a git repo and >> you >> > > plan >> > > > > to >> > > > > > > make >> > > > > > > > > >> > changes. >> > > > > > > > > >> > > > >>>> Then you would open a PR for the change and >> > merge >> > > > it >> > > > > to >> > > > > > > the >> > > > > > > > > >> "main" >> > > > > > > > > >> > > > >>>> branch - and there in your CI/CD you can >> check >> > > all >> > > > > > sorts >> > > > > > > of >> > > > > > > > > >> static >> > > > > > > > > >> > > > >>>> checks and tests. But then the "main" branch >> > > might >> > > > be >> > > > > > > best >> > > > > > > > > >> suited >> > > > > > > > > >> > if >> > > > > > > > > >> > > > >>>> implemented on the test system. Once you >> > validate >> > > > the >> > > > > > > > changes >> > > > > > > > > >> > > > >> end-to-end >> > > > > > > > > >> > > > >>>> you could make another PR for example to a >> > "prod" >> > > > > > branch. >> > > > > > > > And >> > > > > > > > > >> if >> > > > > > > > > >> > > your >> > > > > > > > > >> > > > >>>> production system is only pulling Dags from >> the >> > > > > "prod" >> > > > > > > > branch >> > > > > > > > > >> then >> > > > > > > > > >> > > you >> > > > > > > > > >> > > > >>>> can have this merging strategy as a staging >> > > setup. >> > > > > > > > > >> > > > >>>> >> > > > > > > > > >> > > > >>>> Would this resolve your PING problem? Or >> which >> > > > other >> > > > > > > detail >> > > > > > > > > in >> > > > > > > > > >> the >> > > > > > > > > >> > > use >> > > > > > > > > >> > > > >>>> case would require a PIN on top of a staging >> > > > > strategy? >> > > > > > > > > >> > > > >>>> >> > > > > > > > > >> > > > >>>> Jens >> > > > > > > > > >> > > > >>>> >> > > > > > > > > >> > > > >>>> P.S.: Have enabled your confluence account >> > after >> > > it >> > > > > was >> > > > > > > > > >> created in >> > > > > > > > > >> > > > >> order >> > > > > > > > > >> > > > >>>> to write to Confluence, sorry, typical >> pitfall >> > > > after >> > > > > > > > account >> > > > > > > > > >> > > creation >> > > > > > > > > >> > > > >>>> permissions were not set. Now it should >> work. >> > Let >> > > > me >> > > > > > know >> > > > > > > > if >> > > > > > > > > >> not. >> > > > > > > > > >> > > > >>>> >> > > > > > > > > >> > > > >>>> On 19.04.26 01:40, Piyush Maheshwari wrote: >> > > > > > > > > >> > > > >>>>> Hi everyone, >> > > > > > > > > >> > > > >>>>> I'm a new contributor to Airflow. I'd like >> to >> > > > > propose >> > > > > > a >> > > > > > > > new >> > > > > > > > > >> > feature >> > > > > > > > > >> > > > >> for >> > > > > > > > > >> > > > >>>> Airflow: DAG Version Pinning. >> > > > > > > > > >> > > > >>>>> Building on the foundation introduced by >> > AIP-63: >> > > > DAG >> > > > > > > > > >> Versioning ( >> > > > > > > > > >> > > > >> >> > > > > > > > > >> > > > >> > > > > > > > > >> > > >> > > > > > > > > >> > >> > > > > > > > > >> >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-63*3A*DAG*Versioning__;JSsr!!Ci6f514n9QsL8ck!l3ZKTOw996h9qu4NR0VT4ouUryUdk_HmXUAVPbwCHwPwn0N2CCptVdx95-V0BoRFjws9huE_1Vy-THL8jw$ >> > > > > > > > > >> > > > >>> ), >> > > > > > > > > >> > > > >>>> this proposal aims to extend Airflow's >> > > capabilities >> > > > > to >> > > > > > > > > support >> > > > > > > > > >> > true >> > > > > > > > > >> > > > >>>> continuous deployment (CD) gating and safer >> > > release >> > > > > > > cycles. >> > > > > > > > > >> > > > >>>>> The Problem & Use Cases >> > > > > > > > > >> > > > >>>>> Currently, the scheduler always creates >> > DagRuns >> > > > > using >> > > > > > > the >> > > > > > > > > >> latest >> > > > > > > > > >> > > > >> parsed >> > > > > > > > > >> > > > >>>> DagVersion. This means that the updated DAG >> > code >> > > is >> > > > > > > > deployed >> > > > > > > > > >> > (takes >> > > > > > > > > >> > > > >>> effect) >> > > > > > > > > >> > > > >>>> right after the dag-processor processes it. >> > While >> > > > > this >> > > > > > is >> > > > > > > > > great >> > > > > > > > > >> > for >> > > > > > > > > >> > > > >> rapid >> > > > > > > > > >> > > > >>>> development, teams running business-critical >> > > > > pipelines >> > > > > > > > often >> > > > > > > > > >> need >> > > > > > > > > >> > > > >>> stricter >> > > > > > > > > >> > > > >>>> deployment mechanisms. Specifically: >> > > > > > > > > >> > > > >>>>> * >> > > > > > > > > >> > > > >>>>> Safe Deployment Gating: The ability to pin >> a >> > DAG >> > > > to >> > > > > > its >> > > > > > > > last >> > > > > > > > > >> > known >> > > > > > > > > >> > > > >>>> stable version while new code is parsed in >> the >> > > > > > > background. >> > > > > > > > > This >> > > > > > > > > >> > > allows >> > > > > > > > > >> > > > >>> the >> > > > > > > > > >> > > > >>>> new version to be held back until it passes >> > > > automated >> > > > > > > > > >> regression >> > > > > > > > > >> > > tests >> > > > > > > > > >> > > > >> or >> > > > > > > > > >> > > > >>>> receives explicit manual approval. >> > > > > > > > > >> > > > >>>>> * >> > > > > > > > > >> > > > >>>>> Instant Rollbacks: If an issue is detected >> in >> > a >> > > > > newly >> > > > > > > > > promoted >> > > > > > > > > >> > DAG >> > > > > > > > > >> > > > >>>> version, users need the capability to >> instantly >> > > > roll >> > > > > > back >> > > > > > > > to >> > > > > > > > > a >> > > > > > > > > >> > > > previous >> > > > > > > > > >> > > > >>>> version via the UI/API, without having to >> > revert >> > > > the >> > > > > > > > > underlying >> > > > > > > > > >> > code >> > > > > > > > > >> > > > >> and >> > > > > > > > > >> > > > >>>> wait for the repository sync and DAG >> processing >> > > > > cycle. >> > > > > > > > > >> > > > >>>>> High-Level Proposed Solution >> > > > > > > > > >> > > > >>>>> Introduce an optional >> active_dag_version_id to >> > > the >> > > > > > > > DagModel. >> > > > > > > > > >> This >> > > > > > > > > >> > > > >> field >> > > > > > > > > >> > > > >>>> can be used to pin a DAG version for >> scheduling >> > > and >> > > > > > > > > execution, >> > > > > > > > > >> > while >> > > > > > > > > >> > > > >> the >> > > > > > > > > >> > > > >>>> dag-processor can continue to parse and >> > register >> > > > > newer >> > > > > > > DAG >> > > > > > > > > >> > versions. >> > > > > > > > > >> > > > >>>>> * >> > > > > > > > > >> > > > >>>>> When this pin is set, the scheduler and API >> > will >> > > > > > respect >> > > > > > > > the >> > > > > > > > > >> > pinned >> > > > > > > > > >> > > > >>>> version for creating runs and executing >> tasks, >> > > > > > separating >> > > > > > > > the >> > > > > > > > > >> > > parsing >> > > > > > > > > >> > > > >> of >> > > > > > > > > >> > > > >>>> new code from the execution of new code. >> > > > > > > > > >> > > > >>>>> * >> > > > > > > > > >> > > > >>>>> If the pin is NULL, the system defaults to >> the >> > > > > current >> > > > > > > > > >> behavior >> > > > > > > > > >> > > > >> (always >> > > > > > > > > >> > > > >>>> executing the latest parsed version). This >> way, >> > > we >> > > > > can >> > > > > > > > > maintain >> > > > > > > > > >> > > > >> complete >> > > > > > > > > >> > > > >>>> backwards compatibility. >> > > > > > > > > >> > > > >>>>> I have put together some detailed notes >> > covering >> > > > the >> > > > > > > data >> > > > > > > > > >> model >> > > > > > > > > >> > > > >>> changes, >> > > > > > > > > >> > > > >>>> database migrations, and edge cases with >> this >> > > > > approach. >> > > > > > > If >> > > > > > > > > >> there >> > > > > > > > > >> > is >> > > > > > > > > >> > > > >>> general >> > > > > > > > > >> > > > >>>> alignment that this fits the vision for >> > Airflow, >> > > I >> > > > > > would >> > > > > > > > like >> > > > > > > > > >> to >> > > > > > > > > >> > > take >> > > > > > > > > >> > > > >>> this >> > > > > > > > > >> > > > >>>> proposal through the formal AIP review >> process. >> > > > > > > > > >> > > > >>>>> But I would love to get the community's >> > feedback >> > > > on >> > > > > > the >> > > > > > > > > >> feature >> > > > > > > > > >> > and >> > > > > > > > > >> > > > >> the >> > > > > > > > > >> > > > >>>> high-level approach. >> > > > > > > > > >> > > > >>>>> I'll also need someone to grant me access >> to >> > > > create >> > > > > > > > content >> > > > > > > > > on >> > > > > > > > > >> > the >> > > > > > > > > >> > > > >>>> Airflow Confluence wiki. >> > > > > > > > > >> > > > >>>>> Thanks for your time! >> > > > > > > > > >> > > > >>>>> Regards, >> > > > > > > > > >> > > > >>>>> Piyush >> > > > > > > > > >> > > > >>>>> >> > > > > > > > > >> > > > >>>> >> > > > > > > > > >> > > >> > > > > > > > > >> > > > > >> --------------------------------------------------------------------- >> > > > > > > > > >> > > > >>>> To unsubscribe, e-mail: >> > > > > > > [email protected] >> > > > > > > > > >> > > > >>>> For additional commands, e-mail: >> > > > > > > > [email protected] >> > > > > > > > > >> > > > >>>> >> > > > > > > > > >> > > > >>>> >> > > > > > > > > >> > > > >> > > > > > > > > >> > > > >> > > > > > > > > >> >> > > > > > > >> > > --------------------------------------------------------------------- >> > > > > > > > > >> > > > To unsubscribe, e-mail: >> > > > > [email protected] >> > > > > > > > > >> > > > For additional commands, e-mail: >> > > > > > [email protected] >> > > > > > > > > >> > > > >> > > > > > > > > >> > > > >> > > > > > > > > >> > > >> > > > > > > > > >> > >> > > > > > > > > >> >> > > > > > > > > >> >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> >
