Hey all, First of all, thanks Shivam and Elad for coming up with this AIP! I went over it and left a few comments on the wiki page. Overall, I support the general idea and see how holding TIs (and later tasks) can address potential use cases. However, I think that the scope and caveats should be refined more carefully. Specifically: - If we keep the AIP scoped to TIs only, we need to keep in mind that the abilities are way more limited. For example, holding the first task instance of a scheduled run is impractical: the Dag run has to be created first, and the scheduler queues its first task instances in the same pass. - Dag versions - currently the AIP doesn't refer at all to how it handles Dag versions (e.g., what happens if a task with associated TI is renamed?) - Run pile-ups should be considered, and the AIP should address how to monitor and alert on holds, as they are less visible in the UI than the pause toggles. - Gradual deployment might be a useful scenario if we don't want to depend on timing and want to avoid a second deploy.
Shahar On Thu, Nov 5, 2026 at 2:55 AM <[email protected]> wrote: > Hi all, > > Elad and I would like to start the discussion on AIP-128, which > proposes introducing task holds in Airflow. > > Users sometimes need one task to wait while unrelated work > continues—for example, while correcting bad data or waiting for a > vendor API to recover. A hold lets them prevent that task from > starting without pausing the whole Dag. > > AIP: > https://cwiki.apache.org/confluence/spaces/AIRFLOW/pages/462815942/AIP-128+Task+TaskInstance+Hold > > Demo (1.5 min): > > https://cwiki.apache.org/confluence/download/attachments/462815942/task-holds-demo-short.mp4?api=v2 > > The proposal supports two kinds of hold: > > - Task instance hold: one task in one run, for example publish_report > in today's run while bad data is fixed. > - Task hold: one task in every run, for example while a vendor API is > unavailable. > > The scheduler checks holds before picking tasks to start. An attempt > it has already picked may still run. A hold doesn't change the task's > state; other state changes, such as upstream failure, happen as usual. > Once all holds end, normal scheduling can continue. > > Every hold has a reason, an optional end time and an audit log entry. > We have a working prototype, shown in the video. > > The AIP explains the key design decisions and their trade-offs. We'd > particularly welcome feedback on whether a hold should only block a > task's start, and whether the two proposed scopes are sufficient for a > first version. > > Please comment on the AIP doc, or reply here. > > Thanks, > Shivam > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > >
