Elad,
The same came be said to your ask. Why _should_ airflow support this given it
is already addressable with changing the dag code over time?
I feel this is especially true of Example #1 you gave ("Gradual Deployment on a
dag in production”). This is the most diverging from “pipelines as code”. It’s
now "pipelines as code plus some state managed in the DB/UI that you need to
check”. Essentially: this is state about the dag that lives in the DB, not git,
and _that_ is the departure.
And Example #2 would be justification for a task hold on a _specific_ run, but
I don’t think this justifies hold on all future runs, especially with the Drain
Dag Run (i.e. finish the existing ones, don’t schedule any new ones) that
Dheraj added recently.
> On 13 Nov 2026, at 16:03, Elad Kalif via dev <[email protected]> wrote:
>
> Ash can you be more specific?
> I don't think general concerns are enough to justify not doing this.
>
> Pipelines are code. But production gets broken. When things go bad. You
> need the ability to recover and tooling to help you get things under
> control. This is not designed to be an indefinite hold the same way as
> mapped tasks are not designed to be used with 100K indices. Yes, users can
> abuse it but this is not a reason not to deliver the feature.
>
> I am very happy to address any specific concrete example you have in mind.
>
> On Fri, Nov 13, 2026 at 5:49 PM Ash Berlin-Taylor via dev <
> [email protected]> wrote:
>
>> I’m concerned, particularly of the “all future runs of this task” part of
>> this api.
>>
>> Both of these example use cases could be addressed with “deploy a new
>> version of the dag”. Airflow’s entire ethos started up around “pipelines as
>> code”. I feel this goes further than anything else we’ve done to change
>> that in a really fundamental way.
>>
>> So let’s just say that at present I’m not convinced this is a good fit for
>> Airflow.
>>
>> -ash
>>
>>> On 12 Nov 2026, at 11:22, Elad Kalif via dev <[email protected]>
>> wrote:
>>>
>>> Thanks for the comments
>>>
>>> I'd like to address the question of motivation for the task hold (for
>>> future runs) as it was raised by Niko, Dheeraj and David and we will
>> update
>>> the AIP accordingly to reflect it.
>>> The need for this capability is not theoretical. It is something that I
>>> personally encountered as missing features in my time as Data Engineer
>> and
>>> I can not express enough how much I wished for this capability.
>>>
>>> Examples:
>>> 1. Gradual Deployment on a dag in production. I as a data engineer own
>> the
>>> pipeline but I didn't always own the artifact (models: Sagemaker
>>> endpoints and others). So while My new branch with new capability to an
>>> existing dag is finished and can be merged, the artifact that needs to be
>>> invoked sometimes was not, and that was by another person (MLE/DS) on a
>>> whole different Git repo/code project. So My changes were made on Github.
>>> The PR was not merged because I had to wait for the endpoint to be ready.
>>> That was very frustrating and got me sometimes in cases where the
>> endpoint
>>> was eventually ready but I could not complete the deployment cycle
>>> because there was a code freeze/weekend. This feature solves that
>> problem.
>>> It allows me to deploy my changes to the dag (the new branch in the dag)
>>> and place a hold on that branch. That way I finished my work. When the
>>> model is ready all I need to do is lift the lock no code deployment is
>>> needed on my side. People may argue that I can develop a new dag and put
>> it
>>> on, but that is not necessarily desirable.
>>>
>>> 2. A production incident involves a specific part of the pipeline. Some
>>> resource is decommissioned or changed and there is a need to hold
>> specific
>>> parts of the pipeline until outage is completed. That is not just a
>>> specific run but all runs until the system stabilizes. Today you can not
>> do
>>> that. You either turn off the dag as a whole or you just let it run and
>> let
>>> the tasks related to the outage fail then come back later and rerun them.
>>> That is very annoying. I don't want to get failure callbacks.
>>>
>>> To my understanding, the concerns were not around the implementation
>> point
>>> of view but more around the motivation. So I hope that at least helps to
>>> address that. For implantation I think the key questions, if any, are
>> more
>>> for a code review phase and not for AIP approval phase. Should be needed
>>> during code review we can always decide to extract specific parts for
>> later
>>> phases. I don't think it will come to it though.
>>>
>>> Elad
>>>
>>> On Thu, Nov 12, 2026 at 10:59 AM Blain David <[email protected]>
>>> wrote:
>>>
>>>> Hi Shivam, Elad,
>>>>
>>>> Thanks for the proposal and for the demo. I read the AIP and the
>>>> comments from Niko and Dheeraj on the wiki. My feedback is below; I
>>>> can copy the detailed points to the wiki as inline comments if that
>>>> is easier for you.
>>>>
>>>> Summary: I am in favour of the task instance hold, built the way you
>>>> propose (it only blocks the start, it is enforced in the scheduler,
>>>> and there is no new state). I would leave the task hold across all
>>>> runs out of the first version, and I would rebuild the motivation.
>>>>
>>>> 1. The Motivation
>>>>
>>>> The three examples can each be solved today when the Dag is designed
>>>> for it:
>>>>
>>>> - Bad data: validate the data, emit an Asset only when the check
>>>> passes, and let a second Dag do the publishing. When the check
>>>> fails there is no Asset event, the publishing Dag never starts and
>>>> there is nothing to hold. For bad data that the check missed,
>>>> pausing the publishing Dag does the same job.
>>>> - Vendor outage: give the external system its own pool and set it to
>>>> 0 slots. The task stays in scheduled, its state is untouched and
>>>> everything else runs. That is the behaviour the AIP describes for
>>>> a task hold.
>>>> - Wrong run: put a human-in-the-loop task (AIP-90) before the
>>>> irreversible step.
>>>>
>>>> So these examples read more as an argument for better Dag design
>>>> than for a new feature, which is close to what Niko asks in his
>>>> first comments.
>>>>
>>>> What is left is the unplanned case, and I think it is a real one:
>>>>
>>>> - A Dag you operate but did not write. A platform team on call for
>>>> the Dags of other teams cannot split a Dag or add a pool during an
>>>> incident, because both need a deploy by the owner.
>>>> - One run out of many. Pausing a Dag and emptying a pool act on
>>>> every run. Only a task instance hold stops the 09:00 run while the
>>>> 10:00 run and a backfill continue.
>>>>
>>>> I would build the Motivation on these two.
>>>>
>>>> 2. Scope: task instance hold only
>>>>
>>>> To your question whether the two scopes are right: I agree with Niko
>>>> and Dheeraj to start with the task instance hold. My extra reason is
>>>> that the task hold behaves like a 0-slot pool with a reason and an
>>>> end time. The AIP sets pools aside in one line; it would help to say
>>>> what a task hold adds beyond that.
>>>>
>>>> 3. A hold cannot be placed before clearing a finished run
>>>>
>>>> The AIP says a task instance hold ends when its run finishes and
>>>> stays ended if the run is cleared. Problems in a run are often
>>>> noticed after it has finished. Then I have to clear first and place
>>>> the hold second, while the scheduler is already picking up the
>>>> cleared tasks. Could clearing take a "hold these tasks" option, or
>>>> could a hold placed on a finished run stay dormant until the task is
>>>> cleared?
>>>>
>>>> 4. Attempts that are already queued
>>>>
>>>> To your other question, whether a hold should only block the start:
>>>> yes, but the cost is higher than it looks. Holds are placed during
>>>> incidents, which is when the executor backlog is deepest, so most
>>>> attempts get through exactly then. Did you consider a third check in
>>>> the Execution API, when the worker asks to move the task from queued
>>>> to running? It does not depend on the executor, and that endpoint
>>>> already answers 409 when the task is not in the state it expects. If
>>>> you considered and dropped it, the reason would fit under "Rejected
>>>> alternatives".
>>>>
>>>> 5. Mapped tasks and task loops
>>>>
>>>> The design sections do not mention mapped tasks. Does a hold on a
>>>> mapped task stop the expansion, or does the task expand and the
>>>> expanded task instances wait? Does a hold also cover map indexes
>>>> created after it was placed?
>>>>
>>>> The same question decides what a hold does to a task loop (AIP-111,
>>>> in 3.4). The AIP lists loops under "Related features" as something
>>>> that keeps working, but a loop is the case where it matters most:
>>>> every iteration is its own task instance, created through the
>>>> mapped-task expansion and started in order by the scheduler. So:
>>>>
>>>> - A hold on a task inside a loop covers all its iterations. The loop
>>>> then stops before the next pass and the passes that already ran
>>>> are kept. Is that the intended behaviour? Holding one given
>>>> iteration would need the hold per map index that is listed as
>>>> future work.
>>>> - Can the synthesized gate task and completion node be held? Holding
>>>> the gate is a way to stop a loop after the current pass.
>>>> - Clearing an iteration re-opens the iterations after it. Does a
>>>> hold that was placed before the clear still apply to them?
>>>>
>>>> I think this is a good use case for the task instance hold: stopping
>>>> a refinement loop between two passes to look at the result, without
>>>> a human-in-the-loop step planned in the Dag. It would fit in the
>>>> Motivation.
>>>>
>>>> Task iteration (AIP-104) and task steps (AIP-112) are the simple
>>>> cases and one line each is enough: the items and the steps run
>>>> inside one task instance on the worker, so a hold blocks the start
>>>> of the whole task and never acts between items or steps.
>>>>
>>>> 6. Expiry
>>>>
>>>> A hold can expire at its end time, and the expiry gets an audit log
>>>> entry. Which component ends the hold and writes that entry, and how
>>>> is it written once when several schedulers run?
>>>>
>>>> Best regards,
>>>> David
>>>>
>>>>
>>>> ________________________________
>>>> From: [email protected] <[email protected]>
>>>> Sent: Thursday, November 05, 2026 01:54
>>>> To: [email protected] <[email protected]>
>>>> Cc: [email protected] <[email protected]>
>>>> Subject: [DISCUSS] AIP-128: Task/TaskInstance Hold
>>>>
>>>> EXTERNAL MAIL: Indien je de afzender van deze e-mail niet kent en deze
>>>> niet vertrouwt, klik niet op een link of open geen bijlages. Bij
>> twijfel,
>>>> stuur deze e-mail als bijlage naar [email protected]<mailto:
>>>> [email protected]>.
>>>>
>>>> Hi all,
>>>>
>>>> Elad and I would like to start the discussion on AIP-128, which
>>>> proposes introducing task holds in Airflow.
>>>>
>>>> Users sometimes need one task to wait while unrelated work
>>>> continues—for example, while correcting bad data or waiting for a
>>>> vendor API to recover. A hold lets them prevent that task from
>>>> starting without pausing the whole Dag.
>>>>
>>>> AIP:
>>>>
>> https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fspaces%2FAIRFLOW%2Fpages%2F462815942%2FAIP-128%2BTask%2BTaskInstance%2BHold&data=05%7C02%7Cdavid.blain%40infrabel.be%7C30a3ee14a3d544ddd58808df3b130743%7Cb82bc314ab8e4d6fb18946f02e1f27f2%7C0%7C0%7C639294625069508792%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=9PmV937Pams2fBP9CxuX5xgcfhU6Byag45pVYWwjkY8%3D&reserved=0
>>>> <
>>>>
>> https://cwiki.apache.org/confluence/spaces/AIRFLOW/pages/462815942/AIP-128+Task+TaskInstance+Hold
>>>>>
>>>>
>>>> Demo (1.5 min):
>>>>
>>>>
>> https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdownload%2Fattachments%2F462815942%2Ftask-holds-demo-short.mp4%3Fapi%3Dv2&data=05%7C02%7Cdavid.blain%40infrabel.be%7C30a3ee14a3d544ddd58808df3b130743%7Cb82bc314ab8e4d6fb18946f02e1f27f2%7C0%7C0%7C639294625069537794%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=mWnu%2FCu9NSqPQ2OQdAmM8L8dzUdELbKxIjqPcqPmZ0s%3D&reserved=0
>>>> <
>>>>
>> https://cwiki.apache.org/confluence/download/attachments/462815942/task-holds-demo-short.mp4?api=v2
>>>>>
>>>>
>>>> The proposal supports two kinds of hold:
>>>>
>>>> - Task instance hold: one task in one run, for example publish_report
>>>> in today's run while bad data is fixed.
>>>> - Task hold: one task in every run, for example while a vendor API is
>>>> unavailable.
>>>>
>>>> The scheduler checks holds before picking tasks to start. An attempt
>>>> it has already picked may still run. A hold doesn't change the task's
>>>> state; other state changes, such as upstream failure, happen as usual.
>>>> Once all holds end, normal scheduling can continue.
>>>>
>>>> Every hold has a reason, an optional end time and an audit log entry.
>>>> We have a working prototype, shown in the video.
>>>>
>>>> The AIP explains the key design decisions and their trade-offs. We'd
>>>> particularly welcome feedback on whether a hold should only block a
>>>> task's start, and whether the two proposed scopes are sufficient for a
>>>> first version.
>>>>
>>>> Please comment on the AIP doc, or reply here.
>>>>
>>>> Thanks,
>>>> Shivam
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: [email protected]
>>>> For additional commands, e-mail: [email protected]
>>>>
>>>>
>>>> General (Internal Property)
>>>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: [email protected]
>> For additional commands, e-mail: [email protected]
>>
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
