Hi Vikram
thank you for the detailed review.

I'll start by sharing that I spoke about the feature idea with several
users. Both large and small deployments. All agreed that putting a task on
hold (not just TI) is very much needed and without it the feature is not
that useful.
I will do my best to address any concern and work with the community to get
a green light here. Having the feature just for TI is not an option and I
want to move this off the table. I am very happy to workout whatever is
needed to get alignment for both Task and TI level.

Before I address the specific comments I'd like to offer a way to reduce
some of the concerns by offering the following:
1. The feature will be declared as experimental. That way we have no risk.
We can gather feedback from the users about what we actually deliver and
iterate until we have it working smoothly.
2. We will introduce a system flag that allows deployment managers to
enable/disable the feature for the whole Airflow instance.
enable_task_pasue=True/False (name and default value can be decided later)
3. We will introduce an operator level parameter that allows dag authors to
decide if a specific task is eligible for pausing or maybe with this
specific task there may be some underlying risk.

I think these steps take away any risk or potential problem we may have.


Now I'll address the specific points:

> Firstly, holding a task across all future runs changes the shape of the
DAG. It could change which assets get created, what downstream
asset-triggered DAGs see, and what the alerts watching this DAG mean. The
AIP mentions neither assets nor DAG versions. It does reference deadline
alerts somewhat, but doesn't reference any asset behavior or asset-based
dependencies.
A change to what a DAG does should logically be a new version, which
tremendously simplifies the user experience while retaining consistent
behavior across capabilities.

I see it differently. It's not very different from the scheduler being down
for hours. Systems that have incidents do not play smoothly. It requires
manual intervention. The key aspect that I feel needs to be highlighted is
that this feature is not designed to be automated. A human is involved in
every step of the way. This is primarily an on-call tool. Putting a task on
hold is not a change on the dag code; it's similar to setting a dag to
on/off; it doesn't generate a new version. What side effects are you
worried about in terms of assets? If the task didn't run, what is the
concern?

> Secondly, there is a race condition with in-flight runs. The AIP gives a
table for each task state, but the outcome depends on timing. With many
runs are simultaneously active for the same DAG, some could be past the
task, some are queued and so on.This behavior is undefined. The gradual
deployment example has the same problem.

That is because we see this feature as a complement to clear action. If you
want to pause a running task you will clear it and then place a lock. That
is why we don't need nor want to introduce new logic for tasks that passed
the scheduler. We already have it implemented with clear action.

> Thirdly, the main stated motivation is an unavailable external service,
meaning one task in one DAG cannot run successfully. In a real deployment
many tasks across many DAGs call the same service. An "systems operator"
would have to find every one and place a hold on each. The AIP does not
address this holistically. It dismisses pools, but a shared pool at zero
slots is the one existing control that works across DAGs, and the AIP
itself says a task
hold is "the same as setting a pool to 0 slots today." This is puzzling.

The pool example was to explain why setting a pool to 0 is not enough. I am
happy to edit the AIP with further clarification. Pools can be managed in
different ways. Some use it for resources, some use it for
features/experiences, some use it for priority and there are so many other
patterns. You can not tell someone who uses pools for features to set the
pool to 0 in case a specified resource has an issue.
My point is that while in some scenarios setting pool slots to 0 can work
it does not offer a viable solution for all the use cases. Noting also that
pools are many times used as a global resource and you can not cherry pick
specific tasks that will stop consuming pool slots.

> Fourthly, we already have patterns for this, and they belong in the
operator. Durable execution on the task state store (AIP-103) means a task
doesn't redo completed work after an interruption.
Retry policies (AIP-105) let the operator decide, per failure, whether to
retry, wait, fail, or escalate for a human decision.

Not sure I understand this one. You refer to code changes. The AIP is about
giving the power to take ad-hoc decisions that do not involve changing the
code. If you refer to a case that even if you have an incident durable will
recover from it without intervention that is very optimistic and still
doesn't address the underlying issue of you being an on-call with
production incident you want to stop a specific part of the dag.


> ManagedAgentToolset already handles provider outages with failover inside
the operator. This is cleaner for the DAG author and for the DAG operator.
It is rational to say that these only help DAGs written for them. That is
true, and it is why the operator should carry the behavior, and why the
gradual deployment case belongs with versioning. The AIP doesn't mention
AIP-103 or AIP-105.

If the gradual deployment example confuses people I am happy to drop it.
This is not the focus. The focus is on-call and dag authors who want to put
a temporary stop on a specific part of the dag. The gradual deployment is a
side benefit for organizations that use it for deployment.


> Fifthly, synchronous DAG execution. DAGs that are invoked through the REST
API and waited on become undefined. Users who trigger a DAG and wait for
the result, with the experimental wait
endpoint or their own polling, will be stuck. The AIP never mentions
API-triggered runs. With a task on hold the run
stays Running and the caller blocks until the hold is released or
dagrun_timeout expires, and that has no default. Result XComs never arrive,
and the caller cannot tell a held run from a slow one, because the AIP says
tools reading task states cannot tell a held
task from a waiting one. Anyone who can clear a task can place a task hold,
so one person handling an incident can stall
every synchronous caller of that DAG without their knowing. What should the
wait endpoint return for a run stuck on a
hold, and should triggering be refused or flagged?

I don't understand that example. If someone chooses to put a task on hold
he has permissions to do it on the specific dag. I assume he knows what he
is doing and also knows the implications. Why do you assume otherwise? If
the question is about exposing that a task is placed on hold with API then
yes this is planned. I emphasise again - the feature is primarily for
on-call, incident recovery and other manual ad-hoc usages. Every org has
procedures and protocols on how to do it. The AIP only gives them a tool.
They can use it however they see fit.


I really appreciate your review Vikram! Good stuff!
Let me know if this answers at least some of the concerns you raised.


On Wed, Nov 18, 2026 at 7:40 AM Vikram Koka via dev <[email protected]>
wrote:

> Thank you Shivam and Elad for the AIP and for building a working prototype.
>
> I think I understand the operational need. Like a couple of others, I have
> a problem with the hold on future runs, and I think the
>
> problem is architectural. I am -1 on that part as proposed.
>
>
> Firstly, holding a task across all future runs changes the shape of the
> DAG. It could change which assets get created, what
>
> downstream asset-triggered DAGs see, and what the alerts watching this DAG
> mean. The AIP mentions neither assets nor DAG
>
> versions. It does reference deadline alerts somewhat, but doesn't reference
> any asset behavior or asset-based dependencies.
>
> A change to what a DAG does should logically be a new version, which
> tremendously simplifies the user experience while retaining
>
> consistent behavior across capabilities.
>
>
> Secondly, there is a race condition with in-flight runs. The AIP gives a
> table for each task state, but the outcome depends on
>
> timing. With many runs are simultaneously active for the same DAG, some
> could be past the task, some are queued and so on.
>
> This behavior is undefined. The gradual deployment example has the same
> problem.
>
>
> Thirdly, the main stated motivation is an unavailable external service,
> meaning one task in one DAG cannot run successfully.
>
> In a real deployment many tasks across many DAGs call the same service. An
> "systems operator" would have to find every one
>
> and place a hold on each. The AIP does not address this holistically. It
> dismisses pools, but a shared pool at zero slots is the one
>
> existing control that works across DAGs, and the AIP itself says a task
> hold is "the same as setting a pool to 0 slots today."
>
> This is puzzling.
>
>
> Fourthly, we already have patterns for this, and they belong in the
> operator. Durable execution on the task state store
>
> (AIP-103) means a task doesn't redo completed work after an interruption.
> Retry policies (AIP-105) let the operator
>
> decide, per failure, whether to retry, wait, fail, or escalate for a human
> decision.
>
>
> ManagedAgentToolset already handles provider outages with failover inside
> the operator. This is cleaner for the DAG
>
> author and for the DAG operator. It is rational to say that these only help
> DAGs written for them. That is true, and it is why the
>
> operator should carry the behavior, and why the gradual deployment case
> belongs with versioning. The AIP doesn't mention AIP-103 or AIP-105.
>
>
>
> Fifthly, synchronous DAG execution. DAGs that are invoked through the REST
> API and waited on become undefined.
>
> Users who trigger a DAG and wait for the result, with the experimental wait
> endpoint or their own polling, will be stuck.
>
> The AIP never mentions API-triggered runs. With a task on hold the run
> stays Running and the caller blocks until the hold is released or
>
> dagrun_timeout expires, and that has no default. Result XComs never arrive,
> and the caller cannot tell a held run from a
>
> slow one, because the AIP says tools reading task states cannot tell a held
> task from a waiting one. Anyone who can clear
>
> a task can place a task hold, so one person handling an incident can stall
> every synchronous caller of that DAG without
>
> their knowing. What should the wait endpoint return for a run stuck on a
> hold, and should triggering be refused or flagged?
>
>
> Vikram
>
> On Tue, Nov 17, 2026 at 6:53 PM Elad Kalif via dev <[email protected]
> >
> wrote:
>
> > Hello everyone
> > Thanks for the feedback. I edited the AIP with some clarifications and
> > given the positive momentum I will start a vote thread later this week.
> > I'd like to highlight that performance concerns can be addressed with
> code
> > review and sharing benchmarks if requested. That is not something we can
> > really address on AIP level.
> > The feature scope stays as initially planned at both task level and TI
> > level.
> >
> > On Sun, Nov 15, 2026 at 12:02 AM Sameer Mesiah <[email protected]>
> > wrote:
> >
> > > Hi,
> > >
> > > So I have been through the AIP as well as this thread. Overall, I am in
> > > favour of this AIP because:
> > >
> > > 1. There is value in being able to intervene without redesigning or
> > > redeploying a DAG. Dedicated pools, asset dependencies and
> > > human-in-the-loop tasks can address anticipated scenarios, but they
> > require
> > > the DAG to have been designed accordingly. An on-call engineer may need
> > to
> > > prevent one task from starting during an incident without stopping
> > > unrelated work or waiting for a deployment. David raised a few
> > alternatives
> > > that effectively achieve what this task/ti hold feature does but I
> don’t
> > > think any are as ergonomic.
> > >
> > > 2. I see Ash raised some objections on the basis that this may violate
> > the
> > > ‘pipeline as code’ philosophy. But I don’t think that conflict is
> > inherent.
> > > Don’t we already allow users to intervene through the UI and API, such
> as
> > > pausing DAGs, clearing tasks and marking task states. A task-instance
> > hold
> > > is no different i.e. the code defines the workflow, while the hold
> > > temporarily controls whether a particular task instance can start.
> > >
> > > I think the biggest sticking point (already raised by others) is the
> > > potential performance implications. This of course depends on the final
> > > agreed scope (task/tis or just tis) but it would be interesting to see
> > what
> > > the actual planned metrics for the benchmark would be. I have left a
> > > comment on the confluence page.
> > >
> > > Thanks,
> > > Sameer.
> > >
> > > On Thu, 5 Nov 2026 at 00:55, <[email protected]> wrote:
> > >
> > > > Hi all,
> > > >
> > > > Elad and I would like to start the discussion on AIP-128, which
> > > > proposes introducing task holds in Airflow.
> > > >
> > > > Users sometimes need one task to wait while unrelated work
> > > > continues—for example, while correcting bad data or waiting for a
> > > > vendor API to recover. A hold lets them prevent that task from
> > > > starting without pausing the whole Dag.
> > > >
> > > > AIP:
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/spaces/AIRFLOW/pages/462815942/AIP-128+Task+TaskInstance+Hold
> > > >
> > > > Demo (1.5 min):
> > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/download/attachments/462815942/task-holds-demo-short.mp4?api=v2
> > > >
> > > > The proposal supports two kinds of hold:
> > > >
> > > > - Task instance hold: one task in one run, for example publish_report
> > > > in today's run while bad data is fixed.
> > > > - Task hold: one task in every run, for example while a vendor API is
> > > > unavailable.
> > > >
> > > > The scheduler checks holds before picking tasks to start. An attempt
> > > > it has already picked may still run. A hold doesn't change the task's
> > > > state; other state changes, such as upstream failure, happen as
> usual.
> > > > Once all holds end, normal scheduling can continue.
> > > >
> > > > Every hold has a reason, an optional end time and an audit log entry.
> > > > We have a working prototype, shown in the video.
> > > >
> > > > The AIP explains the key design decisions and their trade-offs. We'd
> > > > particularly welcome feedback on whether a hold should only block a
> > > > task's start, and whether the two proposed scopes are sufficient for
> a
> > > > first version.
> > > >
> > > > Please comment on the AIP doc, or reply here.
> > > >
> > > > Thanks,
> > > > Shivam
> > > >
> > > > ---------------------------------------------------------------------
> > > > To unsubscribe, e-mail: [email protected]
> > > > For additional commands, e-mail: [email protected]
> > > >
> > > >
> > >
> >
>

Reply via email to