Hi all, Elad and I would like to start the discussion on AIP-128, which proposes introducing task holds in Airflow.
Users sometimes need one task to wait while unrelated work continues—for example, while correcting bad data or waiting for a vendor API to recover. A hold lets them prevent that task from starting without pausing the whole Dag. AIP: https://cwiki.apache.org/confluence/spaces/AIRFLOW/pages/462815942/AIP-128+Task+TaskInstance+Hold Demo (1.5 min): https://cwiki.apache.org/confluence/download/attachments/462815942/task-holds-demo-short.mp4?api=v2 The proposal supports two kinds of hold: - Task instance hold: one task in one run, for example publish_report in today's run while bad data is fixed. - Task hold: one task in every run, for example while a vendor API is unavailable. The scheduler checks holds before picking tasks to start. An attempt it has already picked may still run. A hold doesn't change the task's state; other state changes, such as upstream failure, happen as usual. Once all holds end, normal scheduling can continue. Every hold has a reason, an optional end time and an audit log entry. We have a working prototype, shown in the video. The AIP explains the key design decisions and their trade-offs. We'd particularly welcome feedback on whether a hold should only block a task's start, and whether the two proposed scopes are sufficient for a first version. Please comment on the AIP doc, or reply here. Thanks, Shivam --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
