let me address :) 1. As I stated. In many companies pipelines are not owned by a single person. I for example was the owner of the Airflow DAG but I did not own the models and their deployment. That happened by different people with different teams and with a different code package. Our current state with Airflow doesn't support this. If I want to deploy my changes to an existed dag the tasks will fail because model is not ready. That is not desired and that is the problem I want to solve.
2. The term all future runs means that when incident happens I don't know when it will be fixed. If my DAG runs every 5 minutes and a specific part of the pipeline has outage (say EMR) with ETA to fix of 4 hours It makes no sense to put on hold a specific run. I want to put it on all future runs till the issue is fixed. When fixed I can remove the hold and all tasks that were not executed will continue. On Fri, Nov 13, 2026 at 6:35 PM Ash Berlin-Taylor via dev < [email protected]> wrote: > Elad, > > The same came be said to your ask. Why _should_ airflow support this given > it is already addressable with changing the dag code over time? > > I feel this is especially true of Example #1 you gave ("Gradual Deployment > on a dag in production”). This is the most diverging from “pipelines as > code”. It’s now "pipelines as code plus some state managed in the DB/UI > that you need to check”. Essentially: this is state about the dag that > lives in the DB, not git, and _that_ is the departure. > > And Example #2 would be justification for a task hold on a _specific_ run, > but I don’t think this justifies hold on all future runs, especially with > the Drain Dag Run (i.e. finish the existing ones, don’t schedule any new > ones) that Dheraj added recently. > > > > On 13 Nov 2026, at 16:03, Elad Kalif via dev <[email protected]> > wrote: > > > > Ash can you be more specific? > > I don't think general concerns are enough to justify not doing this. > > > > Pipelines are code. But production gets broken. When things go bad. You > > need the ability to recover and tooling to help you get things under > > control. This is not designed to be an indefinite hold the same way as > > mapped tasks are not designed to be used with 100K indices. Yes, users > can > > abuse it but this is not a reason not to deliver the feature. > > > > I am very happy to address any specific concrete example you have in > mind. > > > > On Fri, Nov 13, 2026 at 5:49 PM Ash Berlin-Taylor via dev < > > [email protected]> wrote: > > > >> I’m concerned, particularly of the “all future runs of this task” part > of > >> this api. > >> > >> Both of these example use cases could be addressed with “deploy a new > >> version of the dag”. Airflow’s entire ethos started up around > “pipelines as > >> code”. I feel this goes further than anything else we’ve done to change > >> that in a really fundamental way. > >> > >> So let’s just say that at present I’m not convinced this is a good fit > for > >> Airflow. > >> > >> -ash > >> > >>> On 12 Nov 2026, at 11:22, Elad Kalif via dev <[email protected]> > >> wrote: > >>> > >>> Thanks for the comments > >>> > >>> I'd like to address the question of motivation for the task hold (for > >>> future runs) as it was raised by Niko, Dheeraj and David and we will > >> update > >>> the AIP accordingly to reflect it. > >>> The need for this capability is not theoretical. It is something that I > >>> personally encountered as missing features in my time as Data Engineer > >> and > >>> I can not express enough how much I wished for this capability. > >>> > >>> Examples: > >>> 1. Gradual Deployment on a dag in production. I as a data engineer own > >> the > >>> pipeline but I didn't always own the artifact (models: Sagemaker > >>> endpoints and others). So while My new branch with new capability to an > >>> existing dag is finished and can be merged, the artifact that needs to > be > >>> invoked sometimes was not, and that was by another person (MLE/DS) on a > >>> whole different Git repo/code project. So My changes were made on > Github. > >>> The PR was not merged because I had to wait for the endpoint to be > ready. > >>> That was very frustrating and got me sometimes in cases where the > >> endpoint > >>> was eventually ready but I could not complete the deployment cycle > >>> because there was a code freeze/weekend. This feature solves that > >> problem. > >>> It allows me to deploy my changes to the dag (the new branch in the > dag) > >>> and place a hold on that branch. That way I finished my work. When the > >>> model is ready all I need to do is lift the lock no code deployment is > >>> needed on my side. People may argue that I can develop a new dag and > put > >> it > >>> on, but that is not necessarily desirable. > >>> > >>> 2. A production incident involves a specific part of the pipeline. Some > >>> resource is decommissioned or changed and there is a need to hold > >> specific > >>> parts of the pipeline until outage is completed. That is not just a > >>> specific run but all runs until the system stabilizes. Today you can > not > >> do > >>> that. You either turn off the dag as a whole or you just let it run and > >> let > >>> the tasks related to the outage fail then come back later and rerun > them. > >>> That is very annoying. I don't want to get failure callbacks. > >>> > >>> To my understanding, the concerns were not around the implementation > >> point > >>> of view but more around the motivation. So I hope that at least helps > to > >>> address that. For implantation I think the key questions, if any, are > >> more > >>> for a code review phase and not for AIP approval phase. Should be > needed > >>> during code review we can always decide to extract specific parts for > >> later > >>> phases. I don't think it will come to it though. > >>> > >>> Elad > >>> > >>> On Thu, Nov 12, 2026 at 10:59 AM Blain David <[email protected]> > >>> wrote: > >>> > >>>> Hi Shivam, Elad, > >>>> > >>>> Thanks for the proposal and for the demo. I read the AIP and the > >>>> comments from Niko and Dheeraj on the wiki. My feedback is below; I > >>>> can copy the detailed points to the wiki as inline comments if that > >>>> is easier for you. > >>>> > >>>> Summary: I am in favour of the task instance hold, built the way you > >>>> propose (it only blocks the start, it is enforced in the scheduler, > >>>> and there is no new state). I would leave the task hold across all > >>>> runs out of the first version, and I would rebuild the motivation. > >>>> > >>>> 1. The Motivation > >>>> > >>>> The three examples can each be solved today when the Dag is designed > >>>> for it: > >>>> > >>>> - Bad data: validate the data, emit an Asset only when the check > >>>> passes, and let a second Dag do the publishing. When the check > >>>> fails there is no Asset event, the publishing Dag never starts and > >>>> there is nothing to hold. For bad data that the check missed, > >>>> pausing the publishing Dag does the same job. > >>>> - Vendor outage: give the external system its own pool and set it to > >>>> 0 slots. The task stays in scheduled, its state is untouched and > >>>> everything else runs. That is the behaviour the AIP describes for > >>>> a task hold. > >>>> - Wrong run: put a human-in-the-loop task (AIP-90) before the > >>>> irreversible step. > >>>> > >>>> So these examples read more as an argument for better Dag design > >>>> than for a new feature, which is close to what Niko asks in his > >>>> first comments. > >>>> > >>>> What is left is the unplanned case, and I think it is a real one: > >>>> > >>>> - A Dag you operate but did not write. A platform team on call for > >>>> the Dags of other teams cannot split a Dag or add a pool during an > >>>> incident, because both need a deploy by the owner. > >>>> - One run out of many. Pausing a Dag and emptying a pool act on > >>>> every run. Only a task instance hold stops the 09:00 run while the > >>>> 10:00 run and a backfill continue. > >>>> > >>>> I would build the Motivation on these two. > >>>> > >>>> 2. Scope: task instance hold only > >>>> > >>>> To your question whether the two scopes are right: I agree with Niko > >>>> and Dheeraj to start with the task instance hold. My extra reason is > >>>> that the task hold behaves like a 0-slot pool with a reason and an > >>>> end time. The AIP sets pools aside in one line; it would help to say > >>>> what a task hold adds beyond that. > >>>> > >>>> 3. A hold cannot be placed before clearing a finished run > >>>> > >>>> The AIP says a task instance hold ends when its run finishes and > >>>> stays ended if the run is cleared. Problems in a run are often > >>>> noticed after it has finished. Then I have to clear first and place > >>>> the hold second, while the scheduler is already picking up the > >>>> cleared tasks. Could clearing take a "hold these tasks" option, or > >>>> could a hold placed on a finished run stay dormant until the task is > >>>> cleared? > >>>> > >>>> 4. Attempts that are already queued > >>>> > >>>> To your other question, whether a hold should only block the start: > >>>> yes, but the cost is higher than it looks. Holds are placed during > >>>> incidents, which is when the executor backlog is deepest, so most > >>>> attempts get through exactly then. Did you consider a third check in > >>>> the Execution API, when the worker asks to move the task from queued > >>>> to running? It does not depend on the executor, and that endpoint > >>>> already answers 409 when the task is not in the state it expects. If > >>>> you considered and dropped it, the reason would fit under "Rejected > >>>> alternatives". > >>>> > >>>> 5. Mapped tasks and task loops > >>>> > >>>> The design sections do not mention mapped tasks. Does a hold on a > >>>> mapped task stop the expansion, or does the task expand and the > >>>> expanded task instances wait? Does a hold also cover map indexes > >>>> created after it was placed? > >>>> > >>>> The same question decides what a hold does to a task loop (AIP-111, > >>>> in 3.4). The AIP lists loops under "Related features" as something > >>>> that keeps working, but a loop is the case where it matters most: > >>>> every iteration is its own task instance, created through the > >>>> mapped-task expansion and started in order by the scheduler. So: > >>>> > >>>> - A hold on a task inside a loop covers all its iterations. The loop > >>>> then stops before the next pass and the passes that already ran > >>>> are kept. Is that the intended behaviour? Holding one given > >>>> iteration would need the hold per map index that is listed as > >>>> future work. > >>>> - Can the synthesized gate task and completion node be held? Holding > >>>> the gate is a way to stop a loop after the current pass. > >>>> - Clearing an iteration re-opens the iterations after it. Does a > >>>> hold that was placed before the clear still apply to them? > >>>> > >>>> I think this is a good use case for the task instance hold: stopping > >>>> a refinement loop between two passes to look at the result, without > >>>> a human-in-the-loop step planned in the Dag. It would fit in the > >>>> Motivation. > >>>> > >>>> Task iteration (AIP-104) and task steps (AIP-112) are the simple > >>>> cases and one line each is enough: the items and the steps run > >>>> inside one task instance on the worker, so a hold blocks the start > >>>> of the whole task and never acts between items or steps. > >>>> > >>>> 6. Expiry > >>>> > >>>> A hold can expire at its end time, and the expiry gets an audit log > >>>> entry. Which component ends the hold and writes that entry, and how > >>>> is it written once when several schedulers run? > >>>> > >>>> Best regards, > >>>> David > >>>> > >>>> > >>>> ________________________________ > >>>> From: [email protected] <[email protected]> > >>>> Sent: Thursday, November 05, 2026 01:54 > >>>> To: [email protected] <[email protected]> > >>>> Cc: [email protected] <[email protected]> > >>>> Subject: [DISCUSS] AIP-128: Task/TaskInstance Hold > >>>> > >>>> EXTERNAL MAIL: Indien je de afzender van deze e-mail niet kent en deze > >>>> niet vertrouwt, klik niet op een link of open geen bijlages. Bij > >> twijfel, > >>>> stuur deze e-mail als bijlage naar [email protected]<mailto: > >>>> [email protected]>. > >>>> > >>>> Hi all, > >>>> > >>>> Elad and I would like to start the discussion on AIP-128, which > >>>> proposes introducing task holds in Airflow. > >>>> > >>>> Users sometimes need one task to wait while unrelated work > >>>> continues—for example, while correcting bad data or waiting for a > >>>> vendor API to recover. A hold lets them prevent that task from > >>>> starting without pausing the whole Dag. > >>>> > >>>> AIP: > >>>> > >> > https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fspaces%2FAIRFLOW%2Fpages%2F462815942%2FAIP-128%2BTask%2BTaskInstance%2BHold&data=05%7C02%7Cdavid.blain%40infrabel.be%7C30a3ee14a3d544ddd58808df3b130743%7Cb82bc314ab8e4d6fb18946f02e1f27f2%7C0%7C0%7C639294625069508792%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=9PmV937Pams2fBP9CxuX5xgcfhU6Byag45pVYWwjkY8%3D&reserved=0 > >>>> < > >>>> > >> > https://cwiki.apache.org/confluence/spaces/AIRFLOW/pages/462815942/AIP-128+Task+TaskInstance+Hold > >>>>> > >>>> > >>>> Demo (1.5 min): > >>>> > >>>> > >> > https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdownload%2Fattachments%2F462815942%2Ftask-holds-demo-short.mp4%3Fapi%3Dv2&data=05%7C02%7Cdavid.blain%40infrabel.be%7C30a3ee14a3d544ddd58808df3b130743%7Cb82bc314ab8e4d6fb18946f02e1f27f2%7C0%7C0%7C639294625069537794%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=mWnu%2FCu9NSqPQ2OQdAmM8L8dzUdELbKxIjqPcqPmZ0s%3D&reserved=0 > >>>> < > >>>> > >> > https://cwiki.apache.org/confluence/download/attachments/462815942/task-holds-demo-short.mp4?api=v2 > >>>>> > >>>> > >>>> The proposal supports two kinds of hold: > >>>> > >>>> - Task instance hold: one task in one run, for example publish_report > >>>> in today's run while bad data is fixed. > >>>> - Task hold: one task in every run, for example while a vendor API is > >>>> unavailable. > >>>> > >>>> The scheduler checks holds before picking tasks to start. An attempt > >>>> it has already picked may still run. A hold doesn't change the task's > >>>> state; other state changes, such as upstream failure, happen as usual. > >>>> Once all holds end, normal scheduling can continue. > >>>> > >>>> Every hold has a reason, an optional end time and an audit log entry. > >>>> We have a working prototype, shown in the video. > >>>> > >>>> The AIP explains the key design decisions and their trade-offs. We'd > >>>> particularly welcome feedback on whether a hold should only block a > >>>> task's start, and whether the two proposed scopes are sufficient for a > >>>> first version. > >>>> > >>>> Please comment on the AIP doc, or reply here. > >>>> > >>>> Thanks, > >>>> Shivam > >>>> > >>>> --------------------------------------------------------------------- > >>>> To unsubscribe, e-mail: [email protected] > >>>> For additional commands, e-mail: [email protected] > >>>> > >>>> > >>>> General (Internal Property) > >>>> > >> > >> > >> --------------------------------------------------------------------- > >> To unsubscribe, e-mail: [email protected] > >> For additional commands, e-mail: [email protected] > >> > >> > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > >
