dheerajturaga opened a new pull request, #71038: URL: https://github.com/apache/airflow/pull/71038
## Summary This draft PR introduces on-demand task sections as proposed in [AIP-115](https://cwiki.apache.org/confluence/spaces/AIRFLOW/pages/440305022/AIP-115+On-Demand+Task+Sections). On-demand task sections let Dag authors keep expensive, slow, risky, or rarely needed work visible in a Dag without running it automatically. A normal Dag run can finish without waiting for that optional work, while an authorized user can later run the section for the exact Dag run that needs it. The feature provides a first-class alternative to combining branching, approval tasks, and manual task clearing for this workflow. It preserves the graph as the source of truth and gives operators a discoverable action with a preview of the tasks that will be affected. ## Motivation Airflow supports adjacent workflows today, but each has different semantics: - Human-in-the-loop tasks intentionally keep a Dag run open while waiting for a response. On-demand sections are non-blocking: the default outcome is for the optional work not to run and for the Dag run to complete. - Branching and short-circuiting can omit work based on runtime logic, but do not communicate that a user may run the omitted section later. - Clearing task instances can rerun part of a Dag, but requires users to understand the graph, select the correct task instances, and choose the appropriate clearing behavior themselves. On-demand sections make that intent explicit in both the Dag definition and the operational UI. ## Authoring experience Dag authors place a `ManualGateOperator` at the start of an optional path and may give the gate a user-facing label: ```python from airflow.providers.standard.operators.manual import ManualGateOperator run_optional_enrichment = ManualGateOperator( task_id="run_optional_enrichment", label="Run optional enrichment", ) transform >> load transform >> run_optional_enrichment >> expensive_enrichment >> publish_enrichment ``` The label identifies the action to users. When no label is supplied, the gate uses its task display name or `task_id`. By default, the gate controls all of its downstream descendants. Authors who need later descendants to evaluate their own trigger rules can configure the gate to control only its direct downstream tasks. Teardown tasks are not treated as optional section tasks, preserving their cleanup semantics. ## Dag run behavior For every normal scheduled Dag run: 1. The non-optional path runs normally. 2. The manual gate succeeds. 3. Tasks controlled by the gate enter the new `bypassed` task instance state instead of running. 4. The Dag run can reach a terminal successful state without waiting for user input or optional work. `bypassed` distinguishes work intentionally omitted by a manual gate from tasks skipped for branching, trigger-rule, or other reasons. This distinction is visible in task state filters, state indicators, Grid view, Graph view, task lists, task details, and API responses. The same default applies to scheduled runs, manually triggered runs, and backfills. Backfills therefore do not unexpectedly execute costly optional sections. ## Running an on-demand section After a gate succeeds, users can run its section for that specific Dag run from either the Graph view or the gate task instance details. The action is scoped to the selected run and never implicitly targets the latest run. Before confirmation, Airflow previews the affected task instances. The confirmation flow also supports an action note and protects already-running task instances by default. Once confirmed: - the affected section becomes runnable; - the selected Dag run resumes through normal scheduling; - the gate remains successful; - downstream tasks use their existing dependencies and trigger rules; and - task execution retains normal retries, logs, durations, XComs, callbacks, pools, queues, executor behavior, and observability. The dedicated public task-instance action supports both a dry-run preview and execution, returning the affected task instances to API clients. Invalid targets and unsafe transitions produce explicit client errors rather than silently running unrelated work. The action is available for successful, non-mapped manual gate task instances. A section may be run again when needed; the default running-task protection prevents accidental duplication of work that is still active. ## Permissions and auditability Running an on-demand section is a mutating task-instance action. It requires the same Dag task-instance permission used for other task-instance mutations and participates in API action logging. This does not add per-task assignees or approval checks. Deployments that need a named person or group to approve execution should continue to use Human-in-the-loop operators. ## Example workflows - Run a costly enrichment and publication path only when a downstream consumer requests the enriched dataset for a particular Dag run. - Generate an optional report, export, or artifact from the inputs associated with one completed run without repeating the main pipeline. - Execute a rare reconciliation, validation, or repair section against a selected historical run while leaving routine scheduled runs fast and inexpensive. - Keep potentially high-impact follow-up work visible in the Dag while requiring an explicit operational action before it runs. ## Compatibility and scope This is additive. Existing Dags, skipped-task behavior, branching, Human-in-the-loop workflows, and executor contracts remain unchanged unless a Dag adopts `ManualGateOperator`. The feature intentionally does not: - turn arbitrary skipped tasks into manually runnable tasks; - replace blocking approval workflows; - introduce a separate Dag run type for optional work; - automatically run optional sections for backfills or manually triggered Dag runs; or - add task-level authorization beyond existing Dag and task-instance permissions. --- ##### Was generative AI tooling used to co-author this PR? - [X] Yes — Codex (GPT-5) Generated-by: Codex (GPT-5) following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions) --- Drafted-by: Codex (GPT-5) (no human review before posting) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
