> This means that your runs "never finish”, so you have 10s? 100s? of runs in the running state? How do you know what to look at to see if it is actually running, or “running" cos it’s on hold.
No different than how you'd approach tasks/dags unable to execute because the pool doesn't exist or pool slots are 0. The dagrun is in running mode which is an indication to check what is going on. The bad pitfalls you are worried about are very just we plan to add an active lock section to the main cluster activity dashboard so there is visibility around what is going on. > Once they are on hold, how do you actually mange it? What is the life cycle for these tasks? How do users inevitably tidy this up when the waited on delivery takes longer etc. Isn’t this also what start_date on a task already handles? Or deploy new version and backfill? Placing a hold is a manual action. Designed for a short duration (I agree that short may mean different things for different people) so we expect someone to monitor what is going on and if the lock can be lifted - we can also implement auto release of the lock at a specific time/day. No backfill nor other actions are required. Since tasks that are after the lock were not executed and the dagrun is in Running status this means that once the lock is lifted everything will play as it should have without the lock. > Side point, regardless of my view: we need to find ways of adding this to the UI that does not involve adding more tabs please! We have a plan on how to do it. > And side side point: what did you use to “annotate” the demo recording? It looks very nice. Shivam did that I will let him explain or maybe show in the next dev call :) On Sat, Nov 14, 2026 at 10:11 AM Ash Berlin-Taylor via dev < [email protected]> wrote: > My view is of course just one among many but I’m still not convinced this > part the feature makes sense., and honestly I think Example1 is an > anti-feature, not a reason for having it. > > This means that your runs "never finish”, so you have 10s? 100s? of runs > in the running state? How do you know what to look at to see if it is > actually running, or “running" cos it’s on hold. Likely for days. Once they > are on hold, how do you actually mange it? What is the life cycle for these > tasks? How do users inevitably tidy this up when the waited on delivery > takes longer etc. Isn’t this also what start_date on a task already > handles? Or deploy new version and backfill? > > > Side point, regardless of my view: we need to find ways of adding this to > the UI that does not involve adding more tabs please! > > And side side point: what did you use to “annotate” the demo recording? It > looks very nice. > > > > On 13 Nov 2026, at 17:18, Elad Kalif via dev <[email protected]> > wrote: > > > > I can argue about several features we added that there are workarounds. > > Loops which we recently added can also be deemed as not needed since you > > can use TriggerDagRun for the same dag to invoke a loop. Many things that > > you can workaround become easier when you have a first class feature that > > makes your life simpler. Like the loop case, the drain case, the mapped > > task case and many other features. Airflow is powerful; you can > workaround > > almost anything; the incident recovery theme requires more capability and > > this suggested feature is primarily targeted to that. The rest of the use > > cases I described are because they also fit so it's cool to have them > > solved as well. > > > >> I don’t get why you want to deploy the changes to the dag before they > are > > ready to run? Why not deploy it after? > > > > You can. This is probably what most people do but now you won't have to. > > I don't want to be called to the group leader to explain why the > > pipeline isn't ready to run and stuck on a deployment when the model is. > If > > I can be ready in advance - I want that :) > > This feature allows parallel work. I can deploy my code independently, in > > parallel from the other team and we both are ready. That is very easy. > > Consider also how this relates to the 2nd case of incident recovery. > Let's > > say the model team informed me they have a possible bug and are doing an > > investigation. With the hold feature I can just place this part of the > > pipeline to stop until they deploy their fix from their side. Right now > in > > Airflow you have no way to handle this case easily. What most would do is > > just let it run/fail and then when the issue is fixed to invoke a > > backfill/clear. This feature allows one to avoid this wastefulness and be > > carefree. When the issue is fixed you just lift the hold. > > > >> On 2. “It runs every 5 minutes and I don’t know when it will be fixed” - > > Why do you want to put all future copies of a task on hold? Why don’t you > > want to pause the dag, or drain the dag so that it finished the current > run > > then pauses so nothing else gets scheduled. or even: if this is an > > incident, why isn’t the correct response “pause the dag now!” And then > > unpause when it’s fixed? > > > > That is what people do today. They pause the whole dag or change pool > size > > to zero. > > Now consider my pipeline is composed of parallel branches for Athena, > > Spark, SageMaker and I have an incident just with Spark. With that > feature > > I can allow Athena and Sagemaker continue to run and hold only the Spark > > part. Yes, that can somewhat be achieved also if I set spark_pool=0 as > none > > of the spark resources will run but not all orgs use it that way. Pools > are > > sometimes defined per owner and not per resource and also pools don't > give > > you granularity of controlling what specific tasks in specific dags to > hold. > > > > > > > > On Fri, Nov 13, 2026 at 6:54 PM Ash Berlin-Taylor via dev < > > [email protected]> wrote: > > > >> I don’t get why you want to deploy the changes to the dag before they > are > >> ready to run? Why not deploy it after? > >> > >> On 2. “It runs every 5 minutes and I don’t know when it will be fixed” - > >> Why do you want to put all future copies of a task on hold? Why don’t > you > >> want to pause the dag, or drain the dag so that it finished the current > run > >> then pauses so nothing else gets scheduled. or even: if this is an > >> incident, why isn’t the correct response “pause the dag now!” And then > >> unpause when it’s fixed? > >> > >> > >>> On 13 Nov 2026, at 16:42, Elad Kalif <[email protected]> wrote: > >>> > >>> let me address :) > >>> > >>> 1. As I stated. In many companies pipelines are not owned by a single > >>> person. I for example was the owner of the Airflow DAG but I did not > own > >>> the models and their deployment. That happened by different people with > >>> different teams and with a different code package. Our current state > with > >>> Airflow doesn't support this. If I want to deploy my changes to an > >> existed > >>> dag the tasks will fail because model is not ready. That is not desired > >> and > >>> that is the problem I want to solve. > >>> > >>> 2. The term all future runs means that when incident happens I don't > know > >>> when it will be fixed. If my DAG runs every 5 minutes and a specific > part > >>> of the pipeline has outage (say EMR) with ETA to fix of 4 hours It > makes > >> no > >>> sense to put on hold a specific run. I want to put it on all future > runs > >>> till the issue is fixed. When fixed I can remove the hold and all tasks > >>> that were not executed will continue. > >>> > >>> > >>> > >>> On Fri, Nov 13, 2026 at 6:35 PM Ash Berlin-Taylor via dev < > >>> [email protected]> wrote: > >>> > >>>> Elad, > >>>> > >>>> The same came be said to your ask. Why _should_ airflow support this > >> given > >>>> it is already addressable with changing the dag code over time? > >>>> > >>>> I feel this is especially true of Example #1 you gave ("Gradual > >> Deployment > >>>> on a dag in production”). This is the most diverging from “pipelines > as > >>>> code”. It’s now "pipelines as code plus some state managed in the > DB/UI > >>>> that you need to check”. Essentially: this is state about the dag that > >>>> lives in the DB, not git, and _that_ is the departure. > >>>> > >>>> And Example #2 would be justification for a task hold on a _specific_ > >> run, > >>>> but I don’t think this justifies hold on all future runs, especially > >> with > >>>> the Drain Dag Run (i.e. finish the existing ones, don’t schedule any > new > >>>> ones) that Dheraj added recently. > >>>> > >>>> > >>>>> On 13 Nov 2026, at 16:03, Elad Kalif via dev <[email protected] > > > >>>> wrote: > >>>>> > >>>>> Ash can you be more specific? > >>>>> I don't think general concerns are enough to justify not doing this. > >>>>> > >>>>> Pipelines are code. But production gets broken. When things go bad. > You > >>>>> need the ability to recover and tooling to help you get things under > >>>>> control. This is not designed to be an indefinite hold the same way > as > >>>>> mapped tasks are not designed to be used with 100K indices. Yes, > users > >>>> can > >>>>> abuse it but this is not a reason not to deliver the feature. > >>>>> > >>>>> I am very happy to address any specific concrete example you have in > >>>> mind. > >>>>> > >>>>> On Fri, Nov 13, 2026 at 5:49 PM Ash Berlin-Taylor via dev < > >>>>> [email protected]> wrote: > >>>>> > >>>>>> I’m concerned, particularly of the “all future runs of this task” > part > >>>> of > >>>>>> this api. > >>>>>> > >>>>>> Both of these example use cases could be addressed with “deploy a > new > >>>>>> version of the dag”. Airflow’s entire ethos started up around > >>>> “pipelines as > >>>>>> code”. I feel this goes further than anything else we’ve done to > >> change > >>>>>> that in a really fundamental way. > >>>>>> > >>>>>> So let’s just say that at present I’m not convinced this is a good > fit > >>>> for > >>>>>> Airflow. > >>>>>> > >>>>>> -ash > >>>>>> > >>>>>>> On 12 Nov 2026, at 11:22, Elad Kalif via dev < > [email protected] > >>> > >>>>>> wrote: > >>>>>>> > >>>>>>> Thanks for the comments > >>>>>>> > >>>>>>> I'd like to address the question of motivation for the task hold > (for > >>>>>>> future runs) as it was raised by Niko, Dheeraj and David and we > will > >>>>>> update > >>>>>>> the AIP accordingly to reflect it. > >>>>>>> The need for this capability is not theoretical. It is something > >> that I > >>>>>>> personally encountered as missing features in my time as Data > >> Engineer > >>>>>> and > >>>>>>> I can not express enough how much I wished for this capability. > >>>>>>> > >>>>>>> Examples: > >>>>>>> 1. Gradual Deployment on a dag in production. I as a data engineer > >> own > >>>>>> the > >>>>>>> pipeline but I didn't always own the artifact (models: Sagemaker > >>>>>>> endpoints and others). So while My new branch with new capability > to > >> an > >>>>>>> existing dag is finished and can be merged, the artifact that needs > >> to > >>>> be > >>>>>>> invoked sometimes was not, and that was by another person (MLE/DS) > >> on a > >>>>>>> whole different Git repo/code project. So My changes were made on > >>>> Github. > >>>>>>> The PR was not merged because I had to wait for the endpoint to be > >>>> ready. > >>>>>>> That was very frustrating and got me sometimes in cases where the > >>>>>> endpoint > >>>>>>> was eventually ready but I could not complete the deployment cycle > >>>>>>> because there was a code freeze/weekend. This feature solves that > >>>>>> problem. > >>>>>>> It allows me to deploy my changes to the dag (the new branch in the > >>>> dag) > >>>>>>> and place a hold on that branch. That way I finished my work. When > >> the > >>>>>>> model is ready all I need to do is lift the lock no code deployment > >> is > >>>>>>> needed on my side. People may argue that I can develop a new dag > and > >>>> put > >>>>>> it > >>>>>>> on, but that is not necessarily desirable. > >>>>>>> > >>>>>>> 2. A production incident involves a specific part of the pipeline. > >> Some > >>>>>>> resource is decommissioned or changed and there is a need to hold > >>>>>> specific > >>>>>>> parts of the pipeline until outage is completed. That is not just a > >>>>>>> specific run but all runs until the system stabilizes. Today you > can > >>>> not > >>>>>> do > >>>>>>> that. You either turn off the dag as a whole or you just let it run > >> and > >>>>>> let > >>>>>>> the tasks related to the outage fail then come back later and rerun > >>>> them. > >>>>>>> That is very annoying. I don't want to get failure callbacks. > >>>>>>> > >>>>>>> To my understanding, the concerns were not around the > implementation > >>>>>> point > >>>>>>> of view but more around the motivation. So I hope that at least > helps > >>>> to > >>>>>>> address that. For implantation I think the key questions, if any, > are > >>>>>> more > >>>>>>> for a code review phase and not for AIP approval phase. Should be > >>>> needed > >>>>>>> during code review we can always decide to extract specific parts > for > >>>>>> later > >>>>>>> phases. I don't think it will come to it though. > >>>>>>> > >>>>>>> Elad > >>>>>>> > >>>>>>> On Thu, Nov 12, 2026 at 10:59 AM Blain David < > >> [email protected]> > >>>>>>> wrote: > >>>>>>> > >>>>>>>> Hi Shivam, Elad, > >>>>>>>> > >>>>>>>> Thanks for the proposal and for the demo. I read the AIP and the > >>>>>>>> comments from Niko and Dheeraj on the wiki. My feedback is below; > I > >>>>>>>> can copy the detailed points to the wiki as inline comments if > that > >>>>>>>> is easier for you. > >>>>>>>> > >>>>>>>> Summary: I am in favour of the task instance hold, built the way > you > >>>>>>>> propose (it only blocks the start, it is enforced in the > scheduler, > >>>>>>>> and there is no new state). I would leave the task hold across all > >>>>>>>> runs out of the first version, and I would rebuild the motivation. > >>>>>>>> > >>>>>>>> 1. The Motivation > >>>>>>>> > >>>>>>>> The three examples can each be solved today when the Dag is > designed > >>>>>>>> for it: > >>>>>>>> > >>>>>>>> - Bad data: validate the data, emit an Asset only when the check > >>>>>>>> passes, and let a second Dag do the publishing. When the check > >>>>>>>> fails there is no Asset event, the publishing Dag never starts and > >>>>>>>> there is nothing to hold. For bad data that the check missed, > >>>>>>>> pausing the publishing Dag does the same job. > >>>>>>>> - Vendor outage: give the external system its own pool and set it > to > >>>>>>>> 0 slots. The task stays in scheduled, its state is untouched and > >>>>>>>> everything else runs. That is the behaviour the AIP describes for > >>>>>>>> a task hold. > >>>>>>>> - Wrong run: put a human-in-the-loop task (AIP-90) before the > >>>>>>>> irreversible step. > >>>>>>>> > >>>>>>>> So these examples read more as an argument for better Dag design > >>>>>>>> than for a new feature, which is close to what Niko asks in his > >>>>>>>> first comments. > >>>>>>>> > >>>>>>>> What is left is the unplanned case, and I think it is a real one: > >>>>>>>> > >>>>>>>> - A Dag you operate but did not write. A platform team on call for > >>>>>>>> the Dags of other teams cannot split a Dag or add a pool during an > >>>>>>>> incident, because both need a deploy by the owner. > >>>>>>>> - One run out of many. Pausing a Dag and emptying a pool act on > >>>>>>>> every run. Only a task instance hold stops the 09:00 run while the > >>>>>>>> 10:00 run and a backfill continue. > >>>>>>>> > >>>>>>>> I would build the Motivation on these two. > >>>>>>>> > >>>>>>>> 2. Scope: task instance hold only > >>>>>>>> > >>>>>>>> To your question whether the two scopes are right: I agree with > Niko > >>>>>>>> and Dheeraj to start with the task instance hold. My extra reason > is > >>>>>>>> that the task hold behaves like a 0-slot pool with a reason and an > >>>>>>>> end time. The AIP sets pools aside in one line; it would help to > say > >>>>>>>> what a task hold adds beyond that. > >>>>>>>> > >>>>>>>> 3. A hold cannot be placed before clearing a finished run > >>>>>>>> > >>>>>>>> The AIP says a task instance hold ends when its run finishes and > >>>>>>>> stays ended if the run is cleared. Problems in a run are often > >>>>>>>> noticed after it has finished. Then I have to clear first and > place > >>>>>>>> the hold second, while the scheduler is already picking up the > >>>>>>>> cleared tasks. Could clearing take a "hold these tasks" option, or > >>>>>>>> could a hold placed on a finished run stay dormant until the task > is > >>>>>>>> cleared? > >>>>>>>> > >>>>>>>> 4. Attempts that are already queued > >>>>>>>> > >>>>>>>> To your other question, whether a hold should only block the > start: > >>>>>>>> yes, but the cost is higher than it looks. Holds are placed during > >>>>>>>> incidents, which is when the executor backlog is deepest, so most > >>>>>>>> attempts get through exactly then. Did you consider a third check > in > >>>>>>>> the Execution API, when the worker asks to move the task from > queued > >>>>>>>> to running? It does not depend on the executor, and that endpoint > >>>>>>>> already answers 409 when the task is not in the state it expects. > If > >>>>>>>> you considered and dropped it, the reason would fit under > "Rejected > >>>>>>>> alternatives". > >>>>>>>> > >>>>>>>> 5. Mapped tasks and task loops > >>>>>>>> > >>>>>>>> The design sections do not mention mapped tasks. Does a hold on a > >>>>>>>> mapped task stop the expansion, or does the task expand and the > >>>>>>>> expanded task instances wait? Does a hold also cover map indexes > >>>>>>>> created after it was placed? > >>>>>>>> > >>>>>>>> The same question decides what a hold does to a task loop > (AIP-111, > >>>>>>>> in 3.4). The AIP lists loops under "Related features" as something > >>>>>>>> that keeps working, but a loop is the case where it matters most: > >>>>>>>> every iteration is its own task instance, created through the > >>>>>>>> mapped-task expansion and started in order by the scheduler. So: > >>>>>>>> > >>>>>>>> - A hold on a task inside a loop covers all its iterations. The > loop > >>>>>>>> then stops before the next pass and the passes that already ran > >>>>>>>> are kept. Is that the intended behaviour? Holding one given > >>>>>>>> iteration would need the hold per map index that is listed as > >>>>>>>> future work. > >>>>>>>> - Can the synthesized gate task and completion node be held? > Holding > >>>>>>>> the gate is a way to stop a loop after the current pass. > >>>>>>>> - Clearing an iteration re-opens the iterations after it. Does a > >>>>>>>> hold that was placed before the clear still apply to them? > >>>>>>>> > >>>>>>>> I think this is a good use case for the task instance hold: > stopping > >>>>>>>> a refinement loop between two passes to look at the result, > without > >>>>>>>> a human-in-the-loop step planned in the Dag. It would fit in the > >>>>>>>> Motivation. > >>>>>>>> > >>>>>>>> Task iteration (AIP-104) and task steps (AIP-112) are the simple > >>>>>>>> cases and one line each is enough: the items and the steps run > >>>>>>>> inside one task instance on the worker, so a hold blocks the start > >>>>>>>> of the whole task and never acts between items or steps. > >>>>>>>> > >>>>>>>> 6. Expiry > >>>>>>>> > >>>>>>>> A hold can expire at its end time, and the expiry gets an audit > log > >>>>>>>> entry. Which component ends the hold and writes that entry, and > how > >>>>>>>> is it written once when several schedulers run? > >>>>>>>> > >>>>>>>> Best regards, > >>>>>>>> David > >>>>>>>> > >>>>>>>> > >>>>>>>> ________________________________ > >>>>>>>> From: [email protected] <[email protected]> > >>>>>>>> Sent: Thursday, November 05, 2026 01:54 > >>>>>>>> To: [email protected] <[email protected]> > >>>>>>>> Cc: [email protected] <[email protected]> > >>>>>>>> Subject: [DISCUSS] AIP-128: Task/TaskInstance Hold > >>>>>>>> > >>>>>>>> EXTERNAL MAIL: Indien je de afzender van deze e-mail niet kent en > >> deze > >>>>>>>> niet vertrouwt, klik niet op een link of open geen bijlages. Bij > >>>>>> twijfel, > >>>>>>>> stuur deze e-mail als bijlage naar [email protected]<mailto: > >>>>>>>> [email protected]>. > >>>>>>>> > >>>>>>>> Hi all, > >>>>>>>> > >>>>>>>> Elad and I would like to start the discussion on AIP-128, which > >>>>>>>> proposes introducing task holds in Airflow. > >>>>>>>> > >>>>>>>> Users sometimes need one task to wait while unrelated work > >>>>>>>> continues—for example, while correcting bad data or waiting for a > >>>>>>>> vendor API to recover. A hold lets them prevent that task from > >>>>>>>> starting without pausing the whole Dag. > >>>>>>>> > >>>>>>>> AIP: > >>>>>>>> > >>>>>> > >>>> > >> > https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fspaces%2FAIRFLOW%2Fpages%2F462815942%2FAIP-128%2BTask%2BTaskInstance%2BHold&data=05%7C02%7Cdavid.blain%40infrabel.be%7C30a3ee14a3d544ddd58808df3b130743%7Cb82bc314ab8e4d6fb18946f02e1f27f2%7C0%7C0%7C639294625069508792%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=9PmV937Pams2fBP9CxuX5xgcfhU6Byag45pVYWwjkY8%3D&reserved=0 > >>>>>>>> < > >>>>>>>> > >>>>>> > >>>> > >> > https://cwiki.apache.org/confluence/spaces/AIRFLOW/pages/462815942/AIP-128+Task+TaskInstance+Hold > >>>>>>>>> > >>>>>>>> > >>>>>>>> Demo (1.5 min): > >>>>>>>> > >>>>>>>> > >>>>>> > >>>> > >> > https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdownload%2Fattachments%2F462815942%2Ftask-holds-demo-short.mp4%3Fapi%3Dv2&data=05%7C02%7Cdavid.blain%40infrabel.be%7C30a3ee14a3d544ddd58808df3b130743%7Cb82bc314ab8e4d6fb18946f02e1f27f2%7C0%7C0%7C639294625069537794%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=mWnu%2FCu9NSqPQ2OQdAmM8L8dzUdELbKxIjqPcqPmZ0s%3D&reserved=0 > >>>>>>>> < > >>>>>>>> > >>>>>> > >>>> > >> > https://cwiki.apache.org/confluence/download/attachments/462815942/task-holds-demo-short.mp4?api=v2 > >>>>>>>>> > >>>>>>>> > >>>>>>>> The proposal supports two kinds of hold: > >>>>>>>> > >>>>>>>> - Task instance hold: one task in one run, for example > >> publish_report > >>>>>>>> in today's run while bad data is fixed. > >>>>>>>> - Task hold: one task in every run, for example while a vendor API > >> is > >>>>>>>> unavailable. > >>>>>>>> > >>>>>>>> The scheduler checks holds before picking tasks to start. An > attempt > >>>>>>>> it has already picked may still run. A hold doesn't change the > >> task's > >>>>>>>> state; other state changes, such as upstream failure, happen as > >> usual. > >>>>>>>> Once all holds end, normal scheduling can continue. > >>>>>>>> > >>>>>>>> Every hold has a reason, an optional end time and an audit log > >> entry. > >>>>>>>> We have a working prototype, shown in the video. > >>>>>>>> > >>>>>>>> The AIP explains the key design decisions and their trade-offs. > We'd > >>>>>>>> particularly welcome feedback on whether a hold should only block > a > >>>>>>>> task's start, and whether the two proposed scopes are sufficient > >> for a > >>>>>>>> first version. > >>>>>>>> > >>>>>>>> Please comment on the AIP doc, or reply here. > >>>>>>>> > >>>>>>>> Thanks, > >>>>>>>> Shivam > >>>>>>>> > >>>>>>>> > >> --------------------------------------------------------------------- > >>>>>>>> To unsubscribe, e-mail: [email protected] > >>>>>>>> For additional commands, e-mail: [email protected] > >>>>>>>> > >>>>>>>> > >>>>>>>> General (Internal Property) > >>>>>>>> > >>>>>> > >>>>>> > >>>>>> > --------------------------------------------------------------------- > >>>>>> To unsubscribe, e-mail: [email protected] > >>>>>> For additional commands, e-mail: [email protected] > >>>>>> > >>>>>> > >>>> > >>>> > >>>> --------------------------------------------------------------------- > >>>> To unsubscribe, e-mail: [email protected] > >>>> For additional commands, e-mail: [email protected] > >>>> > >>>> > >> > >> > >> --------------------------------------------------------------------- > >> To unsubscribe, e-mail: [email protected] > >> For additional commands, e-mail: [email protected] > >> > >> > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > >
