Yeah dynamic was being over-used and doesn't say much so agreed.

On Fri, 20 Nov 2026 at 16:34, Ash Berlin-Taylor via dev <
[email protected]> wrote:

> (Yes, though we “soft-renamed” dynamic mapped tasks to just “mapped tasks”
> in 3.4.)
>
> > On 20 Nov 2026, at 16:21, Brent Bovenzi via dev <[email protected]>
> wrote:
> >
> > Planner Tasks sounds good.
> >
> > We already described mapped tasks as "dynamic" so using that word again
> would only confuse people.
> >
> > On 2026/11/20 12:31:44 Amogh Desai via dev wrote:
> >> Thanks for proposing this. I like the idea, and catalog + validator
> seems
> >> like a reasonable
> >> way to keep the LLM planner bounded.
> >>
> >> I left a few comments on the AIP.
> >>
> >> Thanks & Regards,
> >> Amogh Desai
> >>
> >>
> >> On Thu, Nov 19, 2026 at 12:03 AM Kaxil Naik <[email protected]>
> wrote:
> >>
> >>> I do like Planner Tasks
> >>>
> >>> On Wed, 18 Nov 2026 at 17:56, Constance Martineau via dev <
> >>> [email protected]> wrote:
> >>>
> >>>> For the name, I'm find myself defaulting to calling the proposed
> >>>> feature Planner Tasks, especially when discussing it alongside Mapped
> >>>> Tasks and Task Loops. Plan mode doesn't fit as well with the other
> >>>> concepts we have.
> >>>>
> >>>> On Wed, Nov 18, 2026 at 12:49 PM Kaxil Naik <[email protected]>
> wrote:
> >>>>>
> >>>>> Hi all,
> >>>>>
> >>>>> Picking this thread back up.
> >>>>>
> >>>>> Airflow 3.4 shipped Task Loops and YAML Dags, and both made AIP-113
> >>>>> simpler, so Ash and I reworked it on top of them: selected tasks are
> >>> now
> >>>>> created the way loop iterations are, and the plan uses the YAML Dag
> >>>> schema.
> >>>>> The page is updated:
> >>>>>
> >>>>>
> >>>>
> >>>
> https://cwiki.apache.org/confluence/spaces/AIRFLOW/pages/440304823/AIP-113+Dynamic+Task+Groups
> >>>>>
> >>>>> Thanks Przemysław for the earlier comments.
> >>>>>
> >>>>> Why now: the Common AI provider reached 1.0 in October, and more of
> the
> >>>>> Dags people write have a model deciding what happens next. Today an
> LLM
> >>>> can
> >>>>> pick which downstream tasks run (@task.llm_branch), but like any
> branch
> >>>> it
> >>>>> can only select from tasks already wired in the Dag. It can't decide
> >>> the
> >>>>> order or the dependencies between them, so people either put the
> whole
> >>>>> agent loop inside one opaque task, losing per-step retries, logs and
> >>>>> approvals, or generate Dag files at parse time. A few shapes we keep
> >>>> seeing:
> >>>>>
> >>>>> - Incident triage: an agent reads an alert and picks diagnosis steps,
> >>>>> where which probe runs next depends on what the first one found,
> ending
> >>>> in
> >>>>> a fix or a page to a human.
> >>>>> - Support tickets: classify, then run some of validate, refund,
> >>>>> enrich, escalate and notify, in an order that depends on the ticket,
> >>> with
> >>>>> an approval task before anything that moves money.
> >>>>> - Data-quality failures: quarantine, backfill, re-validate and
> notify,
> >>>>> chosen per failure.
> >>>>> - Selective builds, with no LLM involved: an impact analysis decides
> >>>>> which models to rebuild and the edges between exactly those models.
> >>>>>
> >>>>> In each case the set of allowed actions is fixed and reviewed in the
> >>> Dag
> >>>>> file; only the selection and wiring change per run. Each action
> stays a
> >>>>> real task with its own retries, logs and pool, and the catalog plus a
> >>>>> validator keep the model inside what the author approved.
> >>>>>
> >>>>> On the name: neither of us likes "Dynamic Task Groups". It describes
> >>> the
> >>>>> mechanism, not what you get, which is building part of your Dag at
> run
> >>>> time.
> >>>>> We're keeping it for the AIP so the discussion doesn't get stuck on
> it,
> >>>> but
> >>>>> expect it to change before it ships.
> >>>>>
> >>>>> Two that came to my mind: "Plan Mode", as in a task group in plan
> mode,
> >>>> the
> >>>>> way coding agents plan before they act, and "Runtime
> >>> subgraphs/subdags",
> >>>>> which pairs with Task Loops and the @planner decorator. Suggestions
> >>>> welcome.
> >>>>>
> >>>>> What changed since the first draft:
> >>>>>
> >>>>> - Authoring is a @planner task inside a task group, plus a catalog of
> >>>> what
> >>>>> it
> >>>>>  may pick from (@triage.catalog, or catalog= / extra_catalog_items=).
> >>>> The
> >>>>>  TaskFlow and context-manager forms are side by side on the page and
> >>> are
> >>>>>  nearly identical. Catalog items can be operator classes or
> >>> import-path
> >>>>>  strings, so heavy operators don't have to be imported at parse time.
> >>>>>
> >>>>> - The plan uses the same schema as YAML Dags (tasks, uses, with,
> >>> needs),
> >>>> so
> >>>>>  Airflow has one vocabulary for declaring a graph of tasks.
> Templating
> >>>> is
> >>>>>  explicit, as in YAML Dags: a value is rendered only when marked with
> >>>> $t.
> >>>>>
> >>>>> - No subclassing. An LLM planner would itself be a planner, shipped
> by
> >>> a
> >>>>>  provider, with a validator on top. An @task.agent returning the plan
> >>> as
> >>>>>  structured output should also work with no extra machinery.
> >>>>>
> >>>>> - Selected entries are real task instances, not map indices of one
> >>> mapped
> >>>>>  task. The planner submits the plan through the Execution API while
> it
> >>>> is
> >>>>>  still running, and the API server creates the task instances, which
> >>> is
> >>>> the
> >>>>>  approach Task Loops took. The plan is stored against the region, so
> >>>> after
> >>>>> a
> >>>>>  clear and re-plan, the earlier plan and the task instances that ran
> >>>> under
> >>>>> it
> >>>>>  are still there to look at.
> >>>>>
> >>>>> - At parse time the group serializes as two nodes: the planner and a
> >>>> merge
> >>>>>  node. The merge is always an EmptyOperator, and anything downstream
> >>> of
> >>>> the
> >>>>>  group attaches to it.
> >>>>>
> >>>>> - Catalog membership is by exact class, so a subclass of an approved
> >>>>> operator
> >>>>>  is rejected. That can be loosened later; tightening it later would
> be
> >>>> much
> >>>>>  harder.
> >>>>>
> >>>>> Comments on the page or here are both fine.
> >>>>>
> >>>>> Thanks,
> >>>>> Kaxil
> >>>>>
> >>>>> On Tue, 18 Aug 2026 at 21:37, Przemysław Mirowski <[email protected]>
> >>>> wrote:
> >>>>>
> >>>>>> Thanks for the update Kaxil. Looks much better. I've added a couple
> >>> of
> >>>> new
> >>>>>> comments.
> >>>>>>
> >>>>>> On 2026/08/10 21:26:24 Kaxil Naik wrote:
> >>>>>>> Thanks for your feedback Przemysław, I updated the AIP
> >>>>>>>
> >>>>>>> On Fri, 7 Aug 2026 at 03:20, Przemysław Mirowski <[email protected]
> >>>>
> >>>>>> wrote:
> >>>>>>>
> >>>>>>>> Hi Kaxil,
> >>>>>>>>
> >>>>>>>> I've added some comments to the AIP. In general, the idea looks
> >>>>>>>> interesting.
> >>>>>>>>
> >>>>>>>> Could we possibly rearrange the sections a bit? During the
> >>> reading,
> >>>>>>>> sometimes one thing was stated when the reader does not know a
> >>>>>> particular
> >>>>>>>> part of the change (e.g. merge policy - I added a comment next to
> >>>> it).
> >>>>>>>> Also, there are a lot of new definitions - maybe it would be good
> >>>> to
> >>>>>> create
> >>>>>>>> a section which will gather them all and shortly explain?
> >>>>>>>>
> >>>>>>>> On 2026/07/30 10:16:20 Kaxil Naik wrote:
> >>>>>>>>> Hi all,
> >>>>>>>>>
> >>>>>>>>> I would like to start a discussion on AIP-113: Dynamic Task
> >>>> Groups.
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>
> >>>
> https://cwiki.apache.org/confluence/spaces/AIRFLOW/pages/440304823/AIP-113+Dynamic+Task+Groups
> >>>>>>>>>
> >>>>>>>>> tldr:
> >>>>>>>>> Dynamic Task Mapping made cardinality dynamic: run the same
> >>> task
> >>>>>> over a
> >>>>>>>>> collection discovered at run time. What is still static is the
> >>>> shape,
> >>>>>>>>> meaning which tasks
> >>>>>>>>> run in a region and how they depend on each other.
> >>>>>>>>>
> >>>>>>>>> Branching gets you half of that already, and I want to be
> >>> precise
> >>>>>> about
> >>>>>>>>> which half, because
> >>>>>>>>> I had this wrong in an earlier draft. `choose_branch` can
> >>> return
> >>>> a
> >>>>>> list,
> >>>>>>>> so
> >>>>>>>>> a branch already
> >>>>>>>>> selects an arbitrary subset of its declared successors. What a
> >>>> branch
> >>>>>>>>> cannot do is decide
> >>>>>>>>> the dependency edges *among* the tasks it selected. Selection
> >>> is
> >>>>>> solved;
> >>>>>>>>> ordering and wiring are not. That is the
> >>>>>>>>> gap this AIP is aimed at.
> >>>>>>>>>
> >>>>>>>>> The proposal is a Task Group whose members and internal edges
> >>> are
> >>>>>> decided
> >>>>>>>>> at run time from a fixed catalog the author declares. A
> >>> generator
> >>>>>> task
> >>>>>>>>> emits a plan, the group materializes
> >>>>>>>>> exactly the selected entries as real task instances, and the
> >>>> region
> >>>>>>>>> converges on a
> >>>>>>>>> synthesized merge node. The catalog is what serializes, so two
> >>>> runs
> >>>>>> of
> >>>>>>>>> different shapes share
> >>>>>>>>> one DAG version, and an entry the plan did not select never
> >>>> becomes a
> >>>>>>>> task
> >>>>>>>>> instance. The
> >>>>>>>>> planner can be a control table, an impact analysis, a triage
> >>>> rule,
> >>>>>> or an
> >>>>>>>>> LLM.
> >>>>>>>>>
> >>>>>>>>> ---
> >>>>>>>>>
> >>>>>>>>> Would love to hear your thoughts & get feedback. Please comment
> >>>> on
> >>>>>> the
> >>>>>>>> AIP
> >>>>>>>>> doc itself.
> >>>>>>>>>
> >>>>>>>>> Regards,
> >>>>>>>>> Kaxil
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>> ---------------------------------------------------------------------
> >>>>>>>> To unsubscribe, e-mail: [email protected]
> >>>>>>>> For additional commands, e-mail: [email protected]
> >>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>>
> ---------------------------------------------------------------------
> >>>>>> To unsubscribe, e-mail: [email protected]
> >>>>>> For additional commands, e-mail: [email protected]
> >>>>>>
> >>>>>>
> >>>>
> >>>> ---------------------------------------------------------------------
> >>>> To unsubscribe, e-mail: [email protected]
> >>>> For additional commands, e-mail: [email protected]
> >>>>
> >>>>
> >>>
> >>
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: [email protected]
> > For additional commands, e-mail: [email protected]
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [email protected]
> For additional commands, e-mail: [email protected]
>
>

Reply via email to