Thanks for the comments I'd like to address the question of motivation for the task hold (for future runs) as it was raised by Niko, Dheeraj and David and we will update the AIP accordingly to reflect it. The need for this capability is not theoretical. It is something that I personally encountered as missing features in my time as Data Engineer and I can not express enough how much I wished for this capability.
Examples: 1. Gradual Deployment on a dag in production. I as a data engineer own the pipeline but I didn't always own the artifact (models: Sagemaker endpoints and others). So while My new branch with new capability to an existing dag is finished and can be merged, the artifact that needs to be invoked sometimes was not, and that was by another person (MLE/DS) on a whole different Git repo/code project. So My changes were made on Github. The PR was not merged because I had to wait for the endpoint to be ready. That was very frustrating and got me sometimes in cases where the endpoint was eventually ready but I could not complete the deployment cycle because there was a code freeze/weekend. This feature solves that problem. It allows me to deploy my changes to the dag (the new branch in the dag) and place a hold on that branch. That way I finished my work. When the model is ready all I need to do is lift the lock no code deployment is needed on my side. People may argue that I can develop a new dag and put it on, but that is not necessarily desirable. 2. A production incident involves a specific part of the pipeline. Some resource is decommissioned or changed and there is a need to hold specific parts of the pipeline until outage is completed. That is not just a specific run but all runs until the system stabilizes. Today you can not do that. You either turn off the dag as a whole or you just let it run and let the tasks related to the outage fail then come back later and rerun them. That is very annoying. I don't want to get failure callbacks. To my understanding, the concerns were not around the implementation point of view but more around the motivation. So I hope that at least helps to address that. For implantation I think the key questions, if any, are more for a code review phase and not for AIP approval phase. Should be needed during code review we can always decide to extract specific parts for later phases. I don't think it will come to it though. Elad On Thu, Nov 12, 2026 at 10:59 AM Blain David <[email protected]> wrote: > Hi Shivam, Elad, > > Thanks for the proposal and for the demo. I read the AIP and the > comments from Niko and Dheeraj on the wiki. My feedback is below; I > can copy the detailed points to the wiki as inline comments if that > is easier for you. > > Summary: I am in favour of the task instance hold, built the way you > propose (it only blocks the start, it is enforced in the scheduler, > and there is no new state). I would leave the task hold across all > runs out of the first version, and I would rebuild the motivation. > > 1. The Motivation > > The three examples can each be solved today when the Dag is designed > for it: > > - Bad data: validate the data, emit an Asset only when the check > passes, and let a second Dag do the publishing. When the check > fails there is no Asset event, the publishing Dag never starts and > there is nothing to hold. For bad data that the check missed, > pausing the publishing Dag does the same job. > - Vendor outage: give the external system its own pool and set it to > 0 slots. The task stays in scheduled, its state is untouched and > everything else runs. That is the behaviour the AIP describes for > a task hold. > - Wrong run: put a human-in-the-loop task (AIP-90) before the > irreversible step. > > So these examples read more as an argument for better Dag design > than for a new feature, which is close to what Niko asks in his > first comments. > > What is left is the unplanned case, and I think it is a real one: > > - A Dag you operate but did not write. A platform team on call for > the Dags of other teams cannot split a Dag or add a pool during an > incident, because both need a deploy by the owner. > - One run out of many. Pausing a Dag and emptying a pool act on > every run. Only a task instance hold stops the 09:00 run while the > 10:00 run and a backfill continue. > > I would build the Motivation on these two. > > 2. Scope: task instance hold only > > To your question whether the two scopes are right: I agree with Niko > and Dheeraj to start with the task instance hold. My extra reason is > that the task hold behaves like a 0-slot pool with a reason and an > end time. The AIP sets pools aside in one line; it would help to say > what a task hold adds beyond that. > > 3. A hold cannot be placed before clearing a finished run > > The AIP says a task instance hold ends when its run finishes and > stays ended if the run is cleared. Problems in a run are often > noticed after it has finished. Then I have to clear first and place > the hold second, while the scheduler is already picking up the > cleared tasks. Could clearing take a "hold these tasks" option, or > could a hold placed on a finished run stay dormant until the task is > cleared? > > 4. Attempts that are already queued > > To your other question, whether a hold should only block the start: > yes, but the cost is higher than it looks. Holds are placed during > incidents, which is when the executor backlog is deepest, so most > attempts get through exactly then. Did you consider a third check in > the Execution API, when the worker asks to move the task from queued > to running? It does not depend on the executor, and that endpoint > already answers 409 when the task is not in the state it expects. If > you considered and dropped it, the reason would fit under "Rejected > alternatives". > > 5. Mapped tasks and task loops > > The design sections do not mention mapped tasks. Does a hold on a > mapped task stop the expansion, or does the task expand and the > expanded task instances wait? Does a hold also cover map indexes > created after it was placed? > > The same question decides what a hold does to a task loop (AIP-111, > in 3.4). The AIP lists loops under "Related features" as something > that keeps working, but a loop is the case where it matters most: > every iteration is its own task instance, created through the > mapped-task expansion and started in order by the scheduler. So: > > - A hold on a task inside a loop covers all its iterations. The loop > then stops before the next pass and the passes that already ran > are kept. Is that the intended behaviour? Holding one given > iteration would need the hold per map index that is listed as > future work. > - Can the synthesized gate task and completion node be held? Holding > the gate is a way to stop a loop after the current pass. > - Clearing an iteration re-opens the iterations after it. Does a > hold that was placed before the clear still apply to them? > > I think this is a good use case for the task instance hold: stopping > a refinement loop between two passes to look at the result, without > a human-in-the-loop step planned in the Dag. It would fit in the > Motivation. > > Task iteration (AIP-104) and task steps (AIP-112) are the simple > cases and one line each is enough: the items and the steps run > inside one task instance on the worker, so a hold blocks the start > of the whole task and never acts between items or steps. > > 6. Expiry > > A hold can expire at its end time, and the expiry gets an audit log > entry. Which component ends the hold and writes that entry, and how > is it written once when several schedulers run? > > Best regards, > David > > > ________________________________ > From: [email protected] <[email protected]> > Sent: Thursday, November 05, 2026 01:54 > To: [email protected] <[email protected]> > Cc: [email protected] <[email protected]> > Subject: [DISCUSS] AIP-128: Task/TaskInstance Hold > > EXTERNAL MAIL: Indien je de afzender van deze e-mail niet kent en deze > niet vertrouwt, klik niet op een link of open geen bijlages. Bij twijfel, > stuur deze e-mail als bijlage naar [email protected]<mailto: > [email protected]>. > > Hi all, > > Elad and I would like to start the discussion on AIP-128, which > proposes introducing task holds in Airflow. > > Users sometimes need one task to wait while unrelated work > continues—for example, while correcting bad data or waiting for a > vendor API to recover. A hold lets them prevent that task from > starting without pausing the whole Dag. > > AIP: > https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fspaces%2FAIRFLOW%2Fpages%2F462815942%2FAIP-128%2BTask%2BTaskInstance%2BHold&data=05%7C02%7Cdavid.blain%40infrabel.be%7C30a3ee14a3d544ddd58808df3b130743%7Cb82bc314ab8e4d6fb18946f02e1f27f2%7C0%7C0%7C639294625069508792%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=9PmV937Pams2fBP9CxuX5xgcfhU6Byag45pVYWwjkY8%3D&reserved=0 > < > https://cwiki.apache.org/confluence/spaces/AIRFLOW/pages/462815942/AIP-128+Task+TaskInstance+Hold > > > > Demo (1.5 min): > > https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdownload%2Fattachments%2F462815942%2Ftask-holds-demo-short.mp4%3Fapi%3Dv2&data=05%7C02%7Cdavid.blain%40infrabel.be%7C30a3ee14a3d544ddd58808df3b130743%7Cb82bc314ab8e4d6fb18946f02e1f27f2%7C0%7C0%7C639294625069537794%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=mWnu%2FCu9NSqPQ2OQdAmM8L8dzUdELbKxIjqPcqPmZ0s%3D&reserved=0 > < > https://cwiki.apache.org/confluence/download/attachments/462815942/task-holds-demo-short.mp4?api=v2 > > > > The proposal supports two kinds of hold: > > - Task instance hold: one task in one run, for example publish_report > in today's run while bad data is fixed. > - Task hold: one task in every run, for example while a vendor API is > unavailable. > > The scheduler checks holds before picking tasks to start. An attempt > it has already picked may still run. A hold doesn't change the task's > state; other state changes, such as upstream failure, happen as usual. > Once all holds end, normal scheduling can continue. > > Every hold has a reason, an optional end time and an audit log entry. > We have a working prototype, shown in the video. > > The AIP explains the key design decisions and their trade-offs. We'd > particularly welcome feedback on whether a hold should only block a > task's start, and whether the two proposed scopes are sufficient for a > first version. > > Please comment on the AIP doc, or reply here. > > Thanks, > Shivam > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > > > General (Internal Property) >
