I’m concerned, particularly of the “all future runs of this task” part of this api.
Both of these example use cases could be addressed with “deploy a new version of the dag”. Airflow’s entire ethos started up around “pipelines as code”. I feel this goes further than anything else we’ve done to change that in a really fundamental way. So let’s just say that at present I’m not convinced this is a good fit for Airflow. -ash > On 12 Nov 2026, at 11:22, Elad Kalif via dev <[email protected]> wrote: > > Thanks for the comments > > I'd like to address the question of motivation for the task hold (for > future runs) as it was raised by Niko, Dheeraj and David and we will update > the AIP accordingly to reflect it. > The need for this capability is not theoretical. It is something that I > personally encountered as missing features in my time as Data Engineer and > I can not express enough how much I wished for this capability. > > Examples: > 1. Gradual Deployment on a dag in production. I as a data engineer own the > pipeline but I didn't always own the artifact (models: Sagemaker > endpoints and others). So while My new branch with new capability to an > existing dag is finished and can be merged, the artifact that needs to be > invoked sometimes was not, and that was by another person (MLE/DS) on a > whole different Git repo/code project. So My changes were made on Github. > The PR was not merged because I had to wait for the endpoint to be ready. > That was very frustrating and got me sometimes in cases where the endpoint > was eventually ready but I could not complete the deployment cycle > because there was a code freeze/weekend. This feature solves that problem. > It allows me to deploy my changes to the dag (the new branch in the dag) > and place a hold on that branch. That way I finished my work. When the > model is ready all I need to do is lift the lock no code deployment is > needed on my side. People may argue that I can develop a new dag and put it > on, but that is not necessarily desirable. > > 2. A production incident involves a specific part of the pipeline. Some > resource is decommissioned or changed and there is a need to hold specific > parts of the pipeline until outage is completed. That is not just a > specific run but all runs until the system stabilizes. Today you can not do > that. You either turn off the dag as a whole or you just let it run and let > the tasks related to the outage fail then come back later and rerun them. > That is very annoying. I don't want to get failure callbacks. > > To my understanding, the concerns were not around the implementation point > of view but more around the motivation. So I hope that at least helps to > address that. For implantation I think the key questions, if any, are more > for a code review phase and not for AIP approval phase. Should be needed > during code review we can always decide to extract specific parts for later > phases. I don't think it will come to it though. > > Elad > > On Thu, Nov 12, 2026 at 10:59 AM Blain David <[email protected]> > wrote: > >> Hi Shivam, Elad, >> >> Thanks for the proposal and for the demo. I read the AIP and the >> comments from Niko and Dheeraj on the wiki. My feedback is below; I >> can copy the detailed points to the wiki as inline comments if that >> is easier for you. >> >> Summary: I am in favour of the task instance hold, built the way you >> propose (it only blocks the start, it is enforced in the scheduler, >> and there is no new state). I would leave the task hold across all >> runs out of the first version, and I would rebuild the motivation. >> >> 1. The Motivation >> >> The three examples can each be solved today when the Dag is designed >> for it: >> >> - Bad data: validate the data, emit an Asset only when the check >> passes, and let a second Dag do the publishing. When the check >> fails there is no Asset event, the publishing Dag never starts and >> there is nothing to hold. For bad data that the check missed, >> pausing the publishing Dag does the same job. >> - Vendor outage: give the external system its own pool and set it to >> 0 slots. The task stays in scheduled, its state is untouched and >> everything else runs. That is the behaviour the AIP describes for >> a task hold. >> - Wrong run: put a human-in-the-loop task (AIP-90) before the >> irreversible step. >> >> So these examples read more as an argument for better Dag design >> than for a new feature, which is close to what Niko asks in his >> first comments. >> >> What is left is the unplanned case, and I think it is a real one: >> >> - A Dag you operate but did not write. A platform team on call for >> the Dags of other teams cannot split a Dag or add a pool during an >> incident, because both need a deploy by the owner. >> - One run out of many. Pausing a Dag and emptying a pool act on >> every run. Only a task instance hold stops the 09:00 run while the >> 10:00 run and a backfill continue. >> >> I would build the Motivation on these two. >> >> 2. Scope: task instance hold only >> >> To your question whether the two scopes are right: I agree with Niko >> and Dheeraj to start with the task instance hold. My extra reason is >> that the task hold behaves like a 0-slot pool with a reason and an >> end time. The AIP sets pools aside in one line; it would help to say >> what a task hold adds beyond that. >> >> 3. A hold cannot be placed before clearing a finished run >> >> The AIP says a task instance hold ends when its run finishes and >> stays ended if the run is cleared. Problems in a run are often >> noticed after it has finished. Then I have to clear first and place >> the hold second, while the scheduler is already picking up the >> cleared tasks. Could clearing take a "hold these tasks" option, or >> could a hold placed on a finished run stay dormant until the task is >> cleared? >> >> 4. Attempts that are already queued >> >> To your other question, whether a hold should only block the start: >> yes, but the cost is higher than it looks. Holds are placed during >> incidents, which is when the executor backlog is deepest, so most >> attempts get through exactly then. Did you consider a third check in >> the Execution API, when the worker asks to move the task from queued >> to running? It does not depend on the executor, and that endpoint >> already answers 409 when the task is not in the state it expects. If >> you considered and dropped it, the reason would fit under "Rejected >> alternatives". >> >> 5. Mapped tasks and task loops >> >> The design sections do not mention mapped tasks. Does a hold on a >> mapped task stop the expansion, or does the task expand and the >> expanded task instances wait? Does a hold also cover map indexes >> created after it was placed? >> >> The same question decides what a hold does to a task loop (AIP-111, >> in 3.4). The AIP lists loops under "Related features" as something >> that keeps working, but a loop is the case where it matters most: >> every iteration is its own task instance, created through the >> mapped-task expansion and started in order by the scheduler. So: >> >> - A hold on a task inside a loop covers all its iterations. The loop >> then stops before the next pass and the passes that already ran >> are kept. Is that the intended behaviour? Holding one given >> iteration would need the hold per map index that is listed as >> future work. >> - Can the synthesized gate task and completion node be held? Holding >> the gate is a way to stop a loop after the current pass. >> - Clearing an iteration re-opens the iterations after it. Does a >> hold that was placed before the clear still apply to them? >> >> I think this is a good use case for the task instance hold: stopping >> a refinement loop between two passes to look at the result, without >> a human-in-the-loop step planned in the Dag. It would fit in the >> Motivation. >> >> Task iteration (AIP-104) and task steps (AIP-112) are the simple >> cases and one line each is enough: the items and the steps run >> inside one task instance on the worker, so a hold blocks the start >> of the whole task and never acts between items or steps. >> >> 6. Expiry >> >> A hold can expire at its end time, and the expiry gets an audit log >> entry. Which component ends the hold and writes that entry, and how >> is it written once when several schedulers run? >> >> Best regards, >> David >> >> >> ________________________________ >> From: [email protected] <[email protected]> >> Sent: Thursday, November 05, 2026 01:54 >> To: [email protected] <[email protected]> >> Cc: [email protected] <[email protected]> >> Subject: [DISCUSS] AIP-128: Task/TaskInstance Hold >> >> EXTERNAL MAIL: Indien je de afzender van deze e-mail niet kent en deze >> niet vertrouwt, klik niet op een link of open geen bijlages. Bij twijfel, >> stuur deze e-mail als bijlage naar [email protected]<mailto: >> [email protected]>. >> >> Hi all, >> >> Elad and I would like to start the discussion on AIP-128, which >> proposes introducing task holds in Airflow. >> >> Users sometimes need one task to wait while unrelated work >> continues—for example, while correcting bad data or waiting for a >> vendor API to recover. A hold lets them prevent that task from >> starting without pausing the whole Dag. >> >> AIP: >> https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fspaces%2FAIRFLOW%2Fpages%2F462815942%2FAIP-128%2BTask%2BTaskInstance%2BHold&data=05%7C02%7Cdavid.blain%40infrabel.be%7C30a3ee14a3d544ddd58808df3b130743%7Cb82bc314ab8e4d6fb18946f02e1f27f2%7C0%7C0%7C639294625069508792%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=9PmV937Pams2fBP9CxuX5xgcfhU6Byag45pVYWwjkY8%3D&reserved=0 >> < >> https://cwiki.apache.org/confluence/spaces/AIRFLOW/pages/462815942/AIP-128+Task+TaskInstance+Hold >>> >> >> Demo (1.5 min): >> >> https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdownload%2Fattachments%2F462815942%2Ftask-holds-demo-short.mp4%3Fapi%3Dv2&data=05%7C02%7Cdavid.blain%40infrabel.be%7C30a3ee14a3d544ddd58808df3b130743%7Cb82bc314ab8e4d6fb18946f02e1f27f2%7C0%7C0%7C639294625069537794%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=mWnu%2FCu9NSqPQ2OQdAmM8L8dzUdELbKxIjqPcqPmZ0s%3D&reserved=0 >> < >> https://cwiki.apache.org/confluence/download/attachments/462815942/task-holds-demo-short.mp4?api=v2 >>> >> >> The proposal supports two kinds of hold: >> >> - Task instance hold: one task in one run, for example publish_report >> in today's run while bad data is fixed. >> - Task hold: one task in every run, for example while a vendor API is >> unavailable. >> >> The scheduler checks holds before picking tasks to start. An attempt >> it has already picked may still run. A hold doesn't change the task's >> state; other state changes, such as upstream failure, happen as usual. >> Once all holds end, normal scheduling can continue. >> >> Every hold has a reason, an optional end time and an audit log entry. >> We have a working prototype, shown in the video. >> >> The AIP explains the key design decisions and their trade-offs. We'd >> particularly welcome feedback on whether a hold should only block a >> task's start, and whether the two proposed scopes are sufficient for a >> first version. >> >> Please comment on the AIP doc, or reply here. >> >> Thanks, >> Shivam >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: [email protected] >> For additional commands, e-mail: [email protected] >> >> >> General (Internal Property) >> --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
