Thanks for writing this up, Dheeraj and Shubham. I have some thoughts on simplifying it and have added it in the AIP directly.
Thanks & Regards, Amogh Desai On Fri, Jul 31, 2026 at 1:14 AM Constance Martineau via dev < [email protected]> wrote: > Hi Djeeraj, > > I left a comment in the AIP, but it would be great if you could include > some concrete examples of the workflow this serves. I understand why the > workarounds are subpar and Airflow's limitations, but it's not obvious to > me when I'd want to use this. A concrete workflow you run today would make > the case much stronger. > > Thanks, > Constance > > On Thu, Jul 30, 2026 at 2:28 PM Dheeraj Turaga <[email protected]> > wrote: > > > Dear Airflow Community, > > > > Id like to start a discussion on AIP-115: On-Demand Task Sections > > > > > > > https://cwiki.apache.org/confluence/spaces/AIRFLOW/pages/440305022/AIP-115+On-Demand+Task+Sections > > > > TL;DR - AIP-115: On-Demand Task Sections > > > > Some Dags contain work that is expensive, slow, risky, or only > > occasionally needed. Today an author can skip it (branching / > > short-circuit) and lose any first-class way to run it afterward, > > block the whole run on a human decision (HITL approval), or tell > > users to clear the right tasks by hand. GitLab's manual pipeline > > jobs are the closest existing analogue; Airflow has no Dag-level > > equivalent. > > > > AIP-115 proposes a ManualGateOperator that lets an author mark a > > section of a Dag as manual-only > > > > Scheduled runs skip past the gate and complete normally - nothing > > waits on a human. The gated tasks are shown distinctly in the grid > > and graph, and the gate offers a "Run manual section" action. > > Triggering it runs that section for the specific Dag run you chose, > > through the normal scheduler and executor path, with the usual logs, > > retries, XComs, callbacks, and trigger-rule behavior. There's an > > equivalent API endpoint for programmatic use. > > > > The result is an optional branch of a pipeline that's discoverable > > from the Dag definition, visible in the UI, auditable, and runnable > > per-run on demand - rather than a convention users have to know > > about and reproduce by clearing tasks manually. The change is > > additive: existing skipping, branching, and HITL behavior are > > unchanged. > > > > The proposal covers the section-boundary semantics, task states, API > > shape, and alternatives considered in detail > > > > ------ > > Would love to hear your thoughts & feedback > > > > Regards, > > Dheeraj > > >
