I do like Planner Tasks

On Wed, 18 Nov 2026 at 17:56, Constance Martineau via dev <
[email protected]> wrote:

> For the name, I'm find myself defaulting to calling the proposed
> feature Planner Tasks, especially when discussing it alongside Mapped
> Tasks and Task Loops. Plan mode doesn't fit as well with the other
> concepts we have.
>
> On Wed, Nov 18, 2026 at 12:49 PM Kaxil Naik <[email protected]> wrote:
> >
> > Hi all,
> >
> > Picking this thread back up.
> >
> > Airflow 3.4 shipped Task Loops and YAML Dags, and both made AIP-113
> > simpler, so Ash and I reworked it on top of them: selected tasks are now
> > created the way loop iterations are, and the plan uses the YAML Dag
> schema.
> > The page is updated:
> >
> >
> https://cwiki.apache.org/confluence/spaces/AIRFLOW/pages/440304823/AIP-113+Dynamic+Task+Groups
> >
> > Thanks Przemysław for the earlier comments.
> >
> > Why now: the Common AI provider reached 1.0 in October, and more of the
> > Dags people write have a model deciding what happens next. Today an LLM
> can
> > pick which downstream tasks run (@task.llm_branch), but like any branch
> it
> > can only select from tasks already wired in the Dag. It can't decide the
> > order or the dependencies between them, so people either put the whole
> > agent loop inside one opaque task, losing per-step retries, logs and
> > approvals, or generate Dag files at parse time. A few shapes we keep
> seeing:
> >
> > - Incident triage: an agent reads an alert and picks diagnosis steps,
> > where which probe runs next depends on what the first one found, ending
> in
> > a fix or a page to a human.
> > - Support tickets: classify, then run some of validate, refund,
> > enrich, escalate and notify, in an order that depends on the ticket, with
> > an approval task before anything that moves money.
> > - Data-quality failures: quarantine, backfill, re-validate and notify,
> > chosen per failure.
> > - Selective builds, with no LLM involved: an impact analysis decides
> > which models to rebuild and the edges between exactly those models.
> >
> > In each case the set of allowed actions is fixed and reviewed in the Dag
> > file; only the selection and wiring change per run. Each action stays a
> > real task with its own retries, logs and pool, and the catalog plus a
> > validator keep the model inside what the author approved.
> >
> > On the name: neither of us likes "Dynamic Task Groups". It describes the
> > mechanism, not what you get, which is building part of your Dag at run
> time.
> > We're keeping it for the AIP so the discussion doesn't get stuck on it,
> but
> > expect it to change before it ships.
> >
> > Two that came to my mind: "Plan Mode", as in a task group in plan mode,
> the
> > way coding agents plan before they act, and "Runtime subgraphs/subdags",
> > which pairs with Task Loops and the @planner decorator. Suggestions
> welcome.
> >
> > What changed since the first draft:
> >
> > - Authoring is a @planner task inside a task group, plus a catalog of
> what
> > it
> >   may pick from (@triage.catalog, or catalog= / extra_catalog_items=).
> The
> >   TaskFlow and context-manager forms are side by side on the page and are
> >   nearly identical. Catalog items can be operator classes or import-path
> >   strings, so heavy operators don't have to be imported at parse time.
> >
> > - The plan uses the same schema as YAML Dags (tasks, uses, with, needs),
> so
> >   Airflow has one vocabulary for declaring a graph of tasks. Templating
> is
> >   explicit, as in YAML Dags: a value is rendered only when marked with
> $t.
> >
> > - No subclassing. An LLM planner would itself be a planner, shipped by a
> >   provider, with a validator on top. An @task.agent returning the plan as
> >   structured output should also work with no extra machinery.
> >
> > - Selected entries are real task instances, not map indices of one mapped
> >   task. The planner submits the plan through the Execution API while it
> is
> >   still running, and the API server creates the task instances, which is
> the
> >   approach Task Loops took. The plan is stored against the region, so
> after
> > a
> >   clear and re-plan, the earlier plan and the task instances that ran
> under
> > it
> >   are still there to look at.
> >
> > - At parse time the group serializes as two nodes: the planner and a
> merge
> >   node. The merge is always an EmptyOperator, and anything downstream of
> the
> >   group attaches to it.
> >
> > - Catalog membership is by exact class, so a subclass of an approved
> > operator
> >   is rejected. That can be loosened later; tightening it later would be
> much
> >   harder.
> >
> > Comments on the page or here are both fine.
> >
> > Thanks,
> > Kaxil
> >
> > On Tue, 18 Aug 2026 at 21:37, Przemysław Mirowski <[email protected]>
> wrote:
> >
> > > Thanks for the update Kaxil. Looks much better. I've added a couple of
> new
> > > comments.
> > >
> > > On 2026/08/10 21:26:24 Kaxil Naik wrote:
> > > > Thanks for your feedback Przemysław, I updated the AIP
> > > >
> > > > On Fri, 7 Aug 2026 at 03:20, Przemysław Mirowski <[email protected]>
> > > wrote:
> > > >
> > > > > Hi Kaxil,
> > > > >
> > > > > I've added some comments to the AIP. In general, the idea looks
> > > > > interesting.
> > > > >
> > > > > Could we possibly rearrange the sections a bit? During the reading,
> > > > > sometimes one thing was stated when the reader does not know a
> > > particular
> > > > > part of the change (e.g. merge policy - I added a comment next to
> it).
> > > > > Also, there are a lot of new definitions - maybe it would be good
> to
> > > create
> > > > > a section which will gather them all and shortly explain?
> > > > >
> > > > > On 2026/07/30 10:16:20 Kaxil Naik wrote:
> > > > > > Hi all,
> > > > > >
> > > > > > I would like to start a discussion on AIP-113: Dynamic Task
> Groups.
> > > > > >
> > > > > >
> > > > >
> > >
> https://cwiki.apache.org/confluence/spaces/AIRFLOW/pages/440304823/AIP-113+Dynamic+Task+Groups
> > > > > >
> > > > > > tldr:
> > > > > > Dynamic Task Mapping made cardinality dynamic: run the same task
> > > over a
> > > > > > collection discovered at run time. What is still static is the
> shape,
> > > > > > meaning which tasks
> > > > > > run in a region and how they depend on each other.
> > > > > >
> > > > > > Branching gets you half of that already, and I want to be precise
> > > about
> > > > > > which half, because
> > > > > > I had this wrong in an earlier draft. `choose_branch` can return
> a
> > > list,
> > > > > so
> > > > > > a branch already
> > > > > > selects an arbitrary subset of its declared successors. What a
> branch
> > > > > > cannot do is decide
> > > > > > the dependency edges *among* the tasks it selected. Selection is
> > > solved;
> > > > > > ordering and wiring are not. That is the
> > > > > > gap this AIP is aimed at.
> > > > > >
> > > > > > The proposal is a Task Group whose members and internal edges are
> > > decided
> > > > > > at run time from a fixed catalog the author declares. A generator
> > > task
> > > > > > emits a plan, the group materializes
> > > > > > exactly the selected entries as real task instances, and the
> region
> > > > > > converges on a
> > > > > > synthesized merge node. The catalog is what serializes, so two
> runs
> > > of
> > > > > > different shapes share
> > > > > > one DAG version, and an entry the plan did not select never
> becomes a
> > > > > task
> > > > > > instance. The
> > > > > > planner can be a control table, an impact analysis, a triage
> rule,
> > > or an
> > > > > > LLM.
> > > > > >
> > > > > > ---
> > > > > >
> > > > > > Would love to hear your thoughts & get feedback. Please comment
> on
> > > the
> > > > > AIP
> > > > > > doc itself.
> > > > > >
> > > > > > Regards,
> > > > > > Kaxil
> > > > > >
> > > > >
> > > > >
> ---------------------------------------------------------------------
> > > > > To unsubscribe, e-mail: [email protected]
> > > > > For additional commands, e-mail: [email protected]
> > > > >
> > > > >
> > > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: [email protected]
> > > For additional commands, e-mail: [email protected]
> > >
> > >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [email protected]
> For additional commands, e-mail: [email protected]
>
>

Reply via email to