Thanks Bas for your questions: 1. It will be for both specific existing run (TaskInstance level) and on future runs (Task level) so you can place a hold pattern to prevent future tasks from being executed. My previous comment to this thread explained 2 use cases for this capability and we will add it to the AIP. 2. I don't think we should? I assume it should be the same as for clear/mark as success/failed but happy to explore this part if we think it should be a separate permission.
On Thu, Nov 12, 2026 at 12:31 PM Bas Harenslak via dev < [email protected]> wrote: > For me +1 for the general idea. The AIP looks well thought out, I > especially like that it’s not yet another state. > > Two notes: > 1. There’s mention of “a held run”, but this AIP plans for holds on task > instances only, correct? > 2. Are we planning for an extra permission for this operation? > > Bas > > > On 12 Nov 2026, at 11:22, Elad Kalif via dev <[email protected]> > wrote: > > > > Thanks for the comments > > > > I'd like to address the question of motivation for the task hold (for > > future runs) as it was raised by Niko, Dheeraj and David and we will > update > > the AIP accordingly to reflect it. > > The need for this capability is not theoretical. It is something that I > > personally encountered as missing features in my time as Data Engineer > and > > I can not express enough how much I wished for this capability. > > > > Examples: > > 1. Gradual Deployment on a dag in production. I as a data engineer own > the > > pipeline but I didn't always own the artifact (models: Sagemaker > > endpoints and others). So while My new branch with new capability to an > > existing dag is finished and can be merged, the artifact that needs to be > > invoked sometimes was not, and that was by another person (MLE/DS) on a > > whole different Git repo/code project. So My changes were made on Github. > > The PR was not merged because I had to wait for the endpoint to be ready. > > That was very frustrating and got me sometimes in cases where the > endpoint > > was eventually ready but I could not complete the deployment cycle > > because there was a code freeze/weekend. This feature solves that > problem. > > It allows me to deploy my changes to the dag (the new branch in the dag) > > and place a hold on that branch. That way I finished my work. When the > > model is ready all I need to do is lift the lock no code deployment is > > needed on my side. People may argue that I can develop a new dag and put > it > > on, but that is not necessarily desirable. > > > > 2. A production incident involves a specific part of the pipeline. Some > > resource is decommissioned or changed and there is a need to hold > specific > > parts of the pipeline until outage is completed. That is not just a > > specific run but all runs until the system stabilizes. Today you can not > do > > that. You either turn off the dag as a whole or you just let it run and > let > > the tasks related to the outage fail then come back later and rerun them. > > That is very annoying. I don't want to get failure callbacks. > > > > To my understanding, the concerns were not around the implementation > point > > of view but more around the motivation. So I hope that at least helps to > > address that. For implantation I think the key questions, if any, are > more > > for a code review phase and not for AIP approval phase. Should be needed > > during code review we can always decide to extract specific parts for > later > > phases. I don't think it will come to it though. > > > > Elad > > > > On Thu, Nov 12, 2026 at 10:59 AM Blain David <[email protected]> > > wrote: > > > >> Hi Shivam, Elad, > >> > >> Thanks for the proposal and for the demo. I read the AIP and the > >> comments from Niko and Dheeraj on the wiki. My feedback is below; I > >> can copy the detailed points to the wiki as inline comments if that > >> is easier for you. > >> > >> Summary: I am in favour of the task instance hold, built the way you > >> propose (it only blocks the start, it is enforced in the scheduler, > >> and there is no new state). I would leave the task hold across all > >> runs out of the first version, and I would rebuild the motivation. > >> > >> 1. The Motivation > >> > >> The three examples can each be solved today when the Dag is designed > >> for it: > >> > >> - Bad data: validate the data, emit an Asset only when the check > >> passes, and let a second Dag do the publishing. When the check > >> fails there is no Asset event, the publishing Dag never starts and > >> there is nothing to hold. For bad data that the check missed, > >> pausing the publishing Dag does the same job. > >> - Vendor outage: give the external system its own pool and set it to > >> 0 slots. The task stays in scheduled, its state is untouched and > >> everything else runs. That is the behaviour the AIP describes for > >> a task hold. > >> - Wrong run: put a human-in-the-loop task (AIP-90) before the > >> irreversible step. > >> > >> So these examples read more as an argument for better Dag design > >> than for a new feature, which is close to what Niko asks in his > >> first comments. > >> > >> What is left is the unplanned case, and I think it is a real one: > >> > >> - A Dag you operate but did not write. A platform team on call for > >> the Dags of other teams cannot split a Dag or add a pool during an > >> incident, because both need a deploy by the owner. > >> - One run out of many. Pausing a Dag and emptying a pool act on > >> every run. Only a task instance hold stops the 09:00 run while the > >> 10:00 run and a backfill continue. > >> > >> I would build the Motivation on these two. > >> > >> 2. Scope: task instance hold only > >> > >> To your question whether the two scopes are right: I agree with Niko > >> and Dheeraj to start with the task instance hold. My extra reason is > >> that the task hold behaves like a 0-slot pool with a reason and an > >> end time. The AIP sets pools aside in one line; it would help to say > >> what a task hold adds beyond that. > >> > >> 3. A hold cannot be placed before clearing a finished run > >> > >> The AIP says a task instance hold ends when its run finishes and > >> stays ended if the run is cleared. Problems in a run are often > >> noticed after it has finished. Then I have to clear first and place > >> the hold second, while the scheduler is already picking up the > >> cleared tasks. Could clearing take a "hold these tasks" option, or > >> could a hold placed on a finished run stay dormant until the task is > >> cleared? > >> > >> 4. Attempts that are already queued > >> > >> To your other question, whether a hold should only block the start: > >> yes, but the cost is higher than it looks. Holds are placed during > >> incidents, which is when the executor backlog is deepest, so most > >> attempts get through exactly then. Did you consider a third check in > >> the Execution API, when the worker asks to move the task from queued > >> to running? It does not depend on the executor, and that endpoint > >> already answers 409 when the task is not in the state it expects. If > >> you considered and dropped it, the reason would fit under "Rejected > >> alternatives". > >> > >> 5. Mapped tasks and task loops > >> > >> The design sections do not mention mapped tasks. Does a hold on a > >> mapped task stop the expansion, or does the task expand and the > >> expanded task instances wait? Does a hold also cover map indexes > >> created after it was placed? > >> > >> The same question decides what a hold does to a task loop (AIP-111, > >> in 3.4). The AIP lists loops under "Related features" as something > >> that keeps working, but a loop is the case where it matters most: > >> every iteration is its own task instance, created through the > >> mapped-task expansion and started in order by the scheduler. So: > >> > >> - A hold on a task inside a loop covers all its iterations. The loop > >> then stops before the next pass and the passes that already ran > >> are kept. Is that the intended behaviour? Holding one given > >> iteration would need the hold per map index that is listed as > >> future work. > >> - Can the synthesized gate task and completion node be held? Holding > >> the gate is a way to stop a loop after the current pass. > >> - Clearing an iteration re-opens the iterations after it. Does a > >> hold that was placed before the clear still apply to them? > >> > >> I think this is a good use case for the task instance hold: stopping > >> a refinement loop between two passes to look at the result, without > >> a human-in-the-loop step planned in the Dag. It would fit in the > >> Motivation. > >> > >> Task iteration (AIP-104) and task steps (AIP-112) are the simple > >> cases and one line each is enough: the items and the steps run > >> inside one task instance on the worker, so a hold blocks the start > >> of the whole task and never acts between items or steps. > >> > >> 6. Expiry > >> > >> A hold can expire at its end time, and the expiry gets an audit log > >> entry. Which component ends the hold and writes that entry, and how > >> is it written once when several schedulers run? > >> > >> Best regards, > >> David > >> > >> > >> ________________________________ > >> From: [email protected] <[email protected]> > >> Sent: Thursday, November 05, 2026 01:54 > >> To: [email protected] <[email protected]> > >> Cc: [email protected] <[email protected]> > >> Subject: [DISCUSS] AIP-128: Task/TaskInstance Hold > >> > >> EXTERNAL MAIL: Indien je de afzender van deze e-mail niet kent en deze > >> niet vertrouwt, klik niet op een link of open geen bijlages. Bij > twijfel, > >> stuur deze e-mail als bijlage naar [email protected]<mailto: > >> [email protected]>. > >> > >> Hi all, > >> > >> Elad and I would like to start the discussion on AIP-128, which > >> proposes introducing task holds in Airflow. > >> > >> Users sometimes need one task to wait while unrelated work > >> continues—for example, while correcting bad data or waiting for a > >> vendor API to recover. A hold lets them prevent that task from > >> starting without pausing the whole Dag. > >> > >> AIP: > >> > https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fspaces%2FAIRFLOW%2Fpages%2F462815942%2FAIP-128%2BTask%2BTaskInstance%2BHold&data=05%7C02%7Cdavid.blain%40infrabel.be%7C30a3ee14a3d544ddd58808df3b130743%7Cb82bc314ab8e4d6fb18946f02e1f27f2%7C0%7C0%7C639294625069508792%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=9PmV937Pams2fBP9CxuX5xgcfhU6Byag45pVYWwjkY8%3D&reserved=0 > >> < > >> > https://cwiki.apache.org/confluence/spaces/AIRFLOW/pages/462815942/AIP-128+Task+TaskInstance+Hold > >>> > >> > >> Demo (1.5 min): > >> > >> > https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdownload%2Fattachments%2F462815942%2Ftask-holds-demo-short.mp4%3Fapi%3Dv2&data=05%7C02%7Cdavid.blain%40infrabel.be%7C30a3ee14a3d544ddd58808df3b130743%7Cb82bc314ab8e4d6fb18946f02e1f27f2%7C0%7C0%7C639294625069537794%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=mWnu%2FCu9NSqPQ2OQdAmM8L8dzUdELbKxIjqPcqPmZ0s%3D&reserved=0 > >> < > >> > https://cwiki.apache.org/confluence/download/attachments/462815942/task-holds-demo-short.mp4?api=v2 > >>> > >> > >> The proposal supports two kinds of hold: > >> > >> - Task instance hold: one task in one run, for example publish_report > >> in today's run while bad data is fixed. > >> - Task hold: one task in every run, for example while a vendor API is > >> unavailable. > >> > >> The scheduler checks holds before picking tasks to start. An attempt > >> it has already picked may still run. A hold doesn't change the task's > >> state; other state changes, such as upstream failure, happen as usual. > >> Once all holds end, normal scheduling can continue. > >> > >> Every hold has a reason, an optional end time and an audit log entry. > >> We have a working prototype, shown in the video. > >> > >> The AIP explains the key design decisions and their trade-offs. We'd > >> particularly welcome feedback on whether a hold should only block a > >> task's start, and whether the two proposed scopes are sufficient for a > >> first version. > >> > >> Please comment on the AIP doc, or reply here. > >> > >> Thanks, > >> Shivam > >> > >> --------------------------------------------------------------------- > >> To unsubscribe, e-mail: [email protected] > >> For additional commands, e-mail: [email protected] > >> > >> > >> General (Internal Property) > >> > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > >
