Elad,

Thank you and let me respond in detail below.



On Wed, Nov 18, 2026 at 1:34 AM Elad Kalif via dev <[email protected]>
wrote:

> Hi Vikram
> thank you for the detailed review.
>
> I'll start by sharing that I spoke about the feature idea with several
> users. Both large and small deployments. All agreed that putting a task on
> hold (not just TI) is very much needed and without it the feature is not
> that useful.
> I will do my best to address any concern and work with the community to get
> a green light here. Having the feature just for TI is not an option and I
> want to move this off the table. I am very happy to workout whatever is
> needed to get alignment for both Task and TI level.
>
> Before I address the specific comments I'd like to offer a way to reduce
> some of the concerns by offering the following:
> 1. The feature will be declared as experimental. That way we have no risk.
> We can gather feedback from the users about what we actually deliver and
> iterate until we have it working smoothly.
> 2. We will introduce a system flag that allows deployment managers to
> enable/disable the feature for the whole Airflow instance.
> enable_task_pasue=True/False (name and default value can be decided later)
> 3. We will introduce an operator level parameter that allows dag authors to
> decide if a specific task is eligible for pausing or maybe with this
> specific task there may be some underlying risk.
>
> I think these steps take away any risk or potential problem we may have.
>
> "Experimental" defers the semantic questions and don't answer them. The
scheduler interactions and the Dag semantic changes all need to be
communicated and resolved before then, because of the ripple effects with
other
existing features.


>
> Now I'll address the specific points:
>
> > Firstly, holding a task across all future runs changes the shape of the
> DAG. It could change which assets get created, what downstream
> asset-triggered DAGs see, and what the alerts watching this DAG mean. The
> AIP mentions neither assets nor DAG versions. It does reference deadline
> alerts somewhat, but doesn't reference any asset behavior or asset-based
> dependencies.
> A change to what a DAG does should logically be a new version, which
> tremendously simplifies the user experience while retaining consistent
> behavior across capabilities.
>
> I see it differently. It's not very different from the scheduler being down
> for hours. Systems that have incidents do not play smoothly. It requires
> manual intervention. The key aspect that I feel needs to be highlighted is
> that this feature is not designed to be automated. A human is involved in
> every step of the way. This is primarily an on-call tool. Putting a task on
> hold is not a change on the dag code; it's similar to setting a dag to
> on/off; it doesn't generate a new version. What side effects are you
> worried about in terms of assets? If the task didn't run, what is the
> concern?
>

The held task doesn't emit it's Asset. A consumer Dag that waits on several
Assets including this one will never trigger (depending on the condition),
or may
trigger with partial inputs. The rest of the run carries on, so the run is
partially complete.

This is different from existing failure conditions such as:
- Scheduler down:  Delays everything equally and then resumes. A hold
changes
  what one run produces and causes cascading errors in Dags maintained by
other authors,
  without making upstream "partial failures" obvious.
- Pausing a Dag: Similar to the above, a pause is an all-or-nothing and
shows up as a Dag-level
   state. A hold is partial and invisible.

You are correct that pausing a DAG doesn't create a version either. The
difference
is the partial vs. all-or-nothing effect on an ongoing basis.



> > Secondly, there is a race condition with in-flight runs. The AIP gives a
> table for each task state, but the outcome depends on timing. With many
> runs are simultaneously active for the same DAG, some could be past the
> task, some are queued and so on.This behavior is undefined. The gradual
> deployment example has the same problem.
>
> That is because we see this feature as a complement to clear action. If you
> want to pause a running task you will clear it and then place a lock. That
> is why we don't need nor want to introduce new logic for tasks that passed
> the scheduler. We already have it implemented with clear action.
>

We are talking about two different scenarios, but even the one you pointed
out
has a race condition.
What I am speaking about are non-scheduled Dag Runs, whereas you are
referencing
a scheduled Dag run. Non-scheduled Dag runs such as Event driven, which
tend
to have multiple Dag runs concurrently for the same Dag.



>
> > Thirdly, the main stated motivation is an unavailable external service,
> meaning one task in one DAG cannot run successfully. In a real deployment
> many tasks across many DAGs call the same service. An "systems operator"
> would have to find every one and place a hold on each. The AIP does not
> address this holistically. It dismisses pools, but a shared pool at zero
> slots is the one existing control that works across DAGs, and the AIP
> itself says a task
> hold is "the same as setting a pool to 0 slots today." This is puzzling.
>
> The pool example was to explain why setting a pool to 0 is not enough. I am
> happy to edit the AIP with further clarification. Pools can be managed in
> different ways. Some use it for resources, some use it for
> features/experiences, some use it for priority and there are so many other
> patterns. You can not tell someone who uses pools for features to set the
> pool to 0 in case a specified resource has an issue.
> My point is that while in some scenarios setting pool slots to 0 can work
> it does not offer a viable solution for all the use cases. Noting also that
> pools are many times used as a global resource and you can not cherry pick
> specific tasks that will stop consuming pool slots.
>

Agreed that pools get used for features and priority.
But, my main point is that a service outage affects tasks across many Dags.
The operator would need to place one hold per task i.e. per Dag. But, if
the real unit
is the service, the scope should be the service, not the Dag.


>
> > Fourthly, we already have patterns for this, and they belong in the
> operator. Durable execution on the task state store (AIP-103) means a task
> doesn't redo completed work after an interruption.
> Retry policies (AIP-105) let the operator decide, per failure, whether to
> retry, wait, fail, or escalate for a human decision.
>
> Not sure I understand this one. You refer to code changes. The AIP is about
> giving the power to take ad-hoc decisions that do not involve changing the
> code. If you refer to a case that even if you have an incident durable will
> recover from it without intervention that is very optimistic and still
> doesn't address the underlying issue of you being an on-call with
> production incident you want to stop a specific part of the dag.
>
>
I understand that this AIP is about making ad-hoc decisions to address
production issue.
However, this specifically mitigates a production issue partially, rather
than resolving it, because
of Asset creation issues brought up earlier.

I am trying to articulate that there are simpler ways to address service
issues
proactively at the operator level. Durability is already proven for certain
use cases, such
as Databricks. But, what I am trying to highlight is the "Retry" policy and
Recovery which
could include escalation, and can be a pattern going forward to showcase
where
intervention can be safely performed.


> > ManagedAgentToolset already handles provider outages with failover inside
> the operator. This is cleaner for the DAG author and for the DAG operator.
> It is rational to say that these only help DAGs written for them. That is
> true, and it is why the operator should carry the behavior, and why the
> gradual deployment case belongs with versioning. The AIP doesn't mention
> AIP-103 or AIP-105.
>
> If the gradual deployment example confuses people I am happy to drop it.
> This is not the focus. The focus is on-call and dag authors who want to put
> a temporary stop on a specific part of the dag. The gradual deployment is a
> side benefit for organizations that use it for deployment.
>
>
> > Fifthly, synchronous DAG execution. DAGs that are invoked through the
> REST
> API and waited on become undefined. Users who trigger a DAG and wait for
> the result, with the experimental wait
> endpoint or their own polling, will be stuck. The AIP never mentions
> API-triggered runs. With a task on hold the run
> stays Running and the caller blocks until the hold is released or
> dagrun_timeout expires, and that has no default. Result XComs never arrive,
> and the caller cannot tell a held run from a slow one, because the AIP says
> tools reading task states cannot tell a held
> task from a waiting one. Anyone who can clear a task can place a task hold,
> so one person handling an incident can stall
> every synchronous caller of that DAG without their knowing. What should the
> wait endpoint return for a run stuck on a
> hold, and should triggering be refused or flagged?
>
> I don't understand that example. If someone chooses to put a task on hold
> he has permissions to do it on the specific dag. I assume he knows what he
> is doing and also knows the implications. Why do you assume otherwise? If
> the question is about exposing that a task is placed on hold with API then
> yes this is planned. I emphasise again - the feature is primarily for
> on-call, incident recovery and other manual ad-hoc usages. Every org has
> procedures and protocols on how to do it. The AIP only gives them a tool.
> They can use it however they see fit.
>

Scheduled data Dags typically follow the knowledge pattern that you
outlined above.

What I am referring to are "non-scheduled Dags" which are typically invoked
by APIs
from a calling application. Many AI workflows fit this pattern, where the
application is
invoking a worklow (Dag) based on a user action. You may recall this from
some of
the Summit presentations.

The general pattern is that the invoking application is written by someone
other than the Dag
author and Airflow is an underlying component of the overall Product. Now,
this Dag
does typically call an external service such as an AI model. This is a
situation where the callers
have no visibility into the hold and get unexpected failures. The "wait"
results in undefined behavior.


>
>
> I really appreciate your review Vikram! Good stuff!
> Let me know if this answers at least some of the concerns you raised.
>
>
> On Wed, Nov 18, 2026 at 7:40 AM Vikram Koka via dev <
> [email protected]>
> wrote:
>
> > Thank you Shivam and Elad for the AIP and for building a working
> prototype.
> >
> > I think I understand the operational need. Like a couple of others, I
> have
> > a problem with the hold on future runs, and I think the
> >
> > problem is architectural. I am -1 on that part as proposed.
> >
> >
> > Firstly, holding a task across all future runs changes the shape of the
> > DAG. It could change which assets get created, what
> >
> > downstream asset-triggered DAGs see, and what the alerts watching this
> DAG
> > mean. The AIP mentions neither assets nor DAG
> >
> > versions. It does reference deadline alerts somewhat, but doesn't
> reference
> > any asset behavior or asset-based dependencies.
> >
> > A change to what a DAG does should logically be a new version, which
> > tremendously simplifies the user experience while retaining
> >
> > consistent behavior across capabilities.
> >
> >
> > Secondly, there is a race condition with in-flight runs. The AIP gives a
> > table for each task state, but the outcome depends on
> >
> > timing. With many runs are simultaneously active for the same DAG, some
> > could be past the task, some are queued and so on.
> >
> > This behavior is undefined. The gradual deployment example has the same
> > problem.
> >
> >
> > Thirdly, the main stated motivation is an unavailable external service,
> > meaning one task in one DAG cannot run successfully.
> >
> > In a real deployment many tasks across many DAGs call the same service.
> An
> > "systems operator" would have to find every one
> >
> > and place a hold on each. The AIP does not address this holistically. It
> > dismisses pools, but a shared pool at zero slots is the one
> >
> > existing control that works across DAGs, and the AIP itself says a task
> > hold is "the same as setting a pool to 0 slots today."
> >
> > This is puzzling.
> >
> >
> > Fourthly, we already have patterns for this, and they belong in the
> > operator. Durable execution on the task state store
> >
> > (AIP-103) means a task doesn't redo completed work after an interruption.
> > Retry policies (AIP-105) let the operator
> >
> > decide, per failure, whether to retry, wait, fail, or escalate for a
> human
> > decision.
> >
> >
> > ManagedAgentToolset already handles provider outages with failover inside
> > the operator. This is cleaner for the DAG
> >
> > author and for the DAG operator. It is rational to say that these only
> help
> > DAGs written for them. That is true, and it is why the
> >
> > operator should carry the behavior, and why the gradual deployment case
> > belongs with versioning. The AIP doesn't mention AIP-103 or AIP-105.
> >
> >
> >
> > Fifthly, synchronous DAG execution. DAGs that are invoked through the
> REST
> > API and waited on become undefined.
> >
> > Users who trigger a DAG and wait for the result, with the experimental
> wait
> > endpoint or their own polling, will be stuck.
> >
> > The AIP never mentions API-triggered runs. With a task on hold the run
> > stays Running and the caller blocks until the hold is released or
> >
> > dagrun_timeout expires, and that has no default. Result XComs never
> arrive,
> > and the caller cannot tell a held run from a
> >
> > slow one, because the AIP says tools reading task states cannot tell a
> held
> > task from a waiting one. Anyone who can clear
> >
> > a task can place a task hold, so one person handling an incident can
> stall
> > every synchronous caller of that DAG without
> >
> > their knowing. What should the wait endpoint return for a run stuck on a
> > hold, and should triggering be refused or flagged?
> >
> >
> > Vikram
> >
> > On Tue, Nov 17, 2026 at 6:53 PM Elad Kalif via dev <
> [email protected]
> > >
> > wrote:
> >
> > > Hello everyone
> > > Thanks for the feedback. I edited the AIP with some clarifications and
> > > given the positive momentum I will start a vote thread later this week.
> > > I'd like to highlight that performance concerns can be addressed with
> > code
> > > review and sharing benchmarks if requested. That is not something we
> can
> > > really address on AIP level.
> > > The feature scope stays as initially planned at both task level and TI
> > > level.
> > >
> > > On Sun, Nov 15, 2026 at 12:02 AM Sameer Mesiah <[email protected]>
> > > wrote:
> > >
> > > > Hi,
> > > >
> > > > So I have been through the AIP as well as this thread. Overall, I am
> in
> > > > favour of this AIP because:
> > > >
> > > > 1. There is value in being able to intervene without redesigning or
> > > > redeploying a DAG. Dedicated pools, asset dependencies and
> > > > human-in-the-loop tasks can address anticipated scenarios, but they
> > > require
> > > > the DAG to have been designed accordingly. An on-call engineer may
> need
> > > to
> > > > prevent one task from starting during an incident without stopping
> > > > unrelated work or waiting for a deployment. David raised a few
> > > alternatives
> > > > that effectively achieve what this task/ti hold feature does but I
> > don’t
> > > > think any are as ergonomic.
> > > >
> > > > 2. I see Ash raised some objections on the basis that this may
> violate
> > > the
> > > > ‘pipeline as code’ philosophy. But I don’t think that conflict is
> > > inherent.
> > > > Don’t we already allow users to intervene through the UI and API,
> such
> > as
> > > > pausing DAGs, clearing tasks and marking task states. A task-instance
> > > hold
> > > > is no different i.e. the code defines the workflow, while the hold
> > > > temporarily controls whether a particular task instance can start.
> > > >
> > > > I think the biggest sticking point (already raised by others) is the
> > > > potential performance implications. This of course depends on the
> final
> > > > agreed scope (task/tis or just tis) but it would be interesting to
> see
> > > what
> > > > the actual planned metrics for the benchmark would be. I have left a
> > > > comment on the confluence page.
> > > >
> > > > Thanks,
> > > > Sameer.
> > > >
> > > > On Thu, 5 Nov 2026 at 00:55, <[email protected]> wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > > Elad and I would like to start the discussion on AIP-128, which
> > > > > proposes introducing task holds in Airflow.
> > > > >
> > > > > Users sometimes need one task to wait while unrelated work
> > > > > continues—for example, while correcting bad data or waiting for a
> > > > > vendor API to recover. A hold lets them prevent that task from
> > > > > starting without pausing the whole Dag.
> > > > >
> > > > > AIP:
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/spaces/AIRFLOW/pages/462815942/AIP-128+Task+TaskInstance+Hold
> > > > >
> > > > > Demo (1.5 min):
> > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/download/attachments/462815942/task-holds-demo-short.mp4?api=v2
> > > > >
> > > > > The proposal supports two kinds of hold:
> > > > >
> > > > > - Task instance hold: one task in one run, for example
> publish_report
> > > > > in today's run while bad data is fixed.
> > > > > - Task hold: one task in every run, for example while a vendor API
> is
> > > > > unavailable.
> > > > >
> > > > > The scheduler checks holds before picking tasks to start. An
> attempt
> > > > > it has already picked may still run. A hold doesn't change the
> task's
> > > > > state; other state changes, such as upstream failure, happen as
> > usual.
> > > > > Once all holds end, normal scheduling can continue.
> > > > >
> > > > > Every hold has a reason, an optional end time and an audit log
> entry.
> > > > > We have a working prototype, shown in the video.
> > > > >
> > > > > The AIP explains the key design decisions and their trade-offs.
> We'd
> > > > > particularly welcome feedback on whether a hold should only block a
> > > > > task's start, and whether the two proposed scopes are sufficient
> for
> > a
> > > > > first version.
> > > > >
> > > > > Please comment on the AIP doc, or reply here.
> > > > >
> > > > > Thanks,
> > > > > Shivam
> > > > >
> > > > >
> ---------------------------------------------------------------------
> > > > > To unsubscribe, e-mail: [email protected]
> > > > > For additional commands, e-mail: [email protected]
> > > > >
> > > > >
> > > >
> > >
> >
>

Reply via email to