For the name, I'm find myself defaulting to calling the proposed
feature Planner Tasks, especially when discussing it alongside Mapped
Tasks and Task Loops. Plan mode doesn't fit as well with the other
concepts we have.

On Wed, Nov 18, 2026 at 12:49 PM Kaxil Naik <[email protected]> wrote:
>
> Hi all,
>
> Picking this thread back up.
>
> Airflow 3.4 shipped Task Loops and YAML Dags, and both made AIP-113
> simpler, so Ash and I reworked it on top of them: selected tasks are now
> created the way loop iterations are, and the plan uses the YAML Dag schema.
> The page is updated:
>
> https://cwiki.apache.org/confluence/spaces/AIRFLOW/pages/440304823/AIP-113+Dynamic+Task+Groups
>
> Thanks Przemysław for the earlier comments.
>
> Why now: the Common AI provider reached 1.0 in October, and more of the
> Dags people write have a model deciding what happens next. Today an LLM can
> pick which downstream tasks run (@task.llm_branch), but like any branch it
> can only select from tasks already wired in the Dag. It can't decide the
> order or the dependencies between them, so people either put the whole
> agent loop inside one opaque task, losing per-step retries, logs and
> approvals, or generate Dag files at parse time. A few shapes we keep seeing:
>
> - Incident triage: an agent reads an alert and picks diagnosis steps,
> where which probe runs next depends on what the first one found, ending in
> a fix or a page to a human.
> - Support tickets: classify, then run some of validate, refund,
> enrich, escalate and notify, in an order that depends on the ticket, with
> an approval task before anything that moves money.
> - Data-quality failures: quarantine, backfill, re-validate and notify,
> chosen per failure.
> - Selective builds, with no LLM involved: an impact analysis decides
> which models to rebuild and the edges between exactly those models.
>
> In each case the set of allowed actions is fixed and reviewed in the Dag
> file; only the selection and wiring change per run. Each action stays a
> real task with its own retries, logs and pool, and the catalog plus a
> validator keep the model inside what the author approved.
>
> On the name: neither of us likes "Dynamic Task Groups". It describes the
> mechanism, not what you get, which is building part of your Dag at run time.
> We're keeping it for the AIP so the discussion doesn't get stuck on it, but
> expect it to change before it ships.
>
> Two that came to my mind: "Plan Mode", as in a task group in plan mode, the
> way coding agents plan before they act, and "Runtime subgraphs/subdags",
> which pairs with Task Loops and the @planner decorator. Suggestions welcome.
>
> What changed since the first draft:
>
> - Authoring is a @planner task inside a task group, plus a catalog of what
> it
>   may pick from (@triage.catalog, or catalog= / extra_catalog_items=). The
>   TaskFlow and context-manager forms are side by side on the page and are
>   nearly identical. Catalog items can be operator classes or import-path
>   strings, so heavy operators don't have to be imported at parse time.
>
> - The plan uses the same schema as YAML Dags (tasks, uses, with, needs), so
>   Airflow has one vocabulary for declaring a graph of tasks. Templating is
>   explicit, as in YAML Dags: a value is rendered only when marked with $t.
>
> - No subclassing. An LLM planner would itself be a planner, shipped by a
>   provider, with a validator on top. An @task.agent returning the plan as
>   structured output should also work with no extra machinery.
>
> - Selected entries are real task instances, not map indices of one mapped
>   task. The planner submits the plan through the Execution API while it is
>   still running, and the API server creates the task instances, which is the
>   approach Task Loops took. The plan is stored against the region, so after
> a
>   clear and re-plan, the earlier plan and the task instances that ran under
> it
>   are still there to look at.
>
> - At parse time the group serializes as two nodes: the planner and a merge
>   node. The merge is always an EmptyOperator, and anything downstream of the
>   group attaches to it.
>
> - Catalog membership is by exact class, so a subclass of an approved
> operator
>   is rejected. That can be loosened later; tightening it later would be much
>   harder.
>
> Comments on the page or here are both fine.
>
> Thanks,
> Kaxil
>
> On Tue, 18 Aug 2026 at 21:37, Przemysław Mirowski <[email protected]> wrote:
>
> > Thanks for the update Kaxil. Looks much better. I've added a couple of new
> > comments.
> >
> > On 2026/08/10 21:26:24 Kaxil Naik wrote:
> > > Thanks for your feedback Przemysław, I updated the AIP
> > >
> > > On Fri, 7 Aug 2026 at 03:20, Przemysław Mirowski <[email protected]>
> > wrote:
> > >
> > > > Hi Kaxil,
> > > >
> > > > I've added some comments to the AIP. In general, the idea looks
> > > > interesting.
> > > >
> > > > Could we possibly rearrange the sections a bit? During the reading,
> > > > sometimes one thing was stated when the reader does not know a
> > particular
> > > > part of the change (e.g. merge policy - I added a comment next to it).
> > > > Also, there are a lot of new definitions - maybe it would be good to
> > create
> > > > a section which will gather them all and shortly explain?
> > > >
> > > > On 2026/07/30 10:16:20 Kaxil Naik wrote:
> > > > > Hi all,
> > > > >
> > > > > I would like to start a discussion on AIP-113: Dynamic Task Groups.
> > > > >
> > > > >
> > > >
> > https://cwiki.apache.org/confluence/spaces/AIRFLOW/pages/440304823/AIP-113+Dynamic+Task+Groups
> > > > >
> > > > > tldr:
> > > > > Dynamic Task Mapping made cardinality dynamic: run the same task
> > over a
> > > > > collection discovered at run time. What is still static is the shape,
> > > > > meaning which tasks
> > > > > run in a region and how they depend on each other.
> > > > >
> > > > > Branching gets you half of that already, and I want to be precise
> > about
> > > > > which half, because
> > > > > I had this wrong in an earlier draft. `choose_branch` can return a
> > list,
> > > > so
> > > > > a branch already
> > > > > selects an arbitrary subset of its declared successors. What a branch
> > > > > cannot do is decide
> > > > > the dependency edges *among* the tasks it selected. Selection is
> > solved;
> > > > > ordering and wiring are not. That is the
> > > > > gap this AIP is aimed at.
> > > > >
> > > > > The proposal is a Task Group whose members and internal edges are
> > decided
> > > > > at run time from a fixed catalog the author declares. A generator
> > task
> > > > > emits a plan, the group materializes
> > > > > exactly the selected entries as real task instances, and the region
> > > > > converges on a
> > > > > synthesized merge node. The catalog is what serializes, so two runs
> > of
> > > > > different shapes share
> > > > > one DAG version, and an entry the plan did not select never becomes a
> > > > task
> > > > > instance. The
> > > > > planner can be a control table, an impact analysis, a triage rule,
> > or an
> > > > > LLM.
> > > > >
> > > > > ---
> > > > >
> > > > > Would love to hear your thoughts & get feedback. Please comment on
> > the
> > > > AIP
> > > > > doc itself.
> > > > >
> > > > > Regards,
> > > > > Kaxil
> > > > >
> > > >
> > > > ---------------------------------------------------------------------
> > > > To unsubscribe, e-mail: [email protected]
> > > > For additional commands, e-mail: [email protected]
> > > >
> > > >
> > >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: [email protected]
> > For additional commands, e-mail: [email protected]
> >
> >

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to