(Yes, though we “soft-renamed” dynamic mapped tasks to just “mapped tasks” in 3.4.)
> On 20 Nov 2026, at 16:21, Brent Bovenzi via dev <[email protected]> > wrote: > > Planner Tasks sounds good. > > We already described mapped tasks as "dynamic" so using that word again would > only confuse people. > > On 2026/11/20 12:31:44 Amogh Desai via dev wrote: >> Thanks for proposing this. I like the idea, and catalog + validator seems >> like a reasonable >> way to keep the LLM planner bounded. >> >> I left a few comments on the AIP. >> >> Thanks & Regards, >> Amogh Desai >> >> >> On Thu, Nov 19, 2026 at 12:03 AM Kaxil Naik <[email protected]> wrote: >> >>> I do like Planner Tasks >>> >>> On Wed, 18 Nov 2026 at 17:56, Constance Martineau via dev < >>> [email protected]> wrote: >>> >>>> For the name, I'm find myself defaulting to calling the proposed >>>> feature Planner Tasks, especially when discussing it alongside Mapped >>>> Tasks and Task Loops. Plan mode doesn't fit as well with the other >>>> concepts we have. >>>> >>>> On Wed, Nov 18, 2026 at 12:49 PM Kaxil Naik <[email protected]> wrote: >>>>> >>>>> Hi all, >>>>> >>>>> Picking this thread back up. >>>>> >>>>> Airflow 3.4 shipped Task Loops and YAML Dags, and both made AIP-113 >>>>> simpler, so Ash and I reworked it on top of them: selected tasks are >>> now >>>>> created the way loop iterations are, and the plan uses the YAML Dag >>>> schema. >>>>> The page is updated: >>>>> >>>>> >>>> >>> https://cwiki.apache.org/confluence/spaces/AIRFLOW/pages/440304823/AIP-113+Dynamic+Task+Groups >>>>> >>>>> Thanks Przemysław for the earlier comments. >>>>> >>>>> Why now: the Common AI provider reached 1.0 in October, and more of the >>>>> Dags people write have a model deciding what happens next. Today an LLM >>>> can >>>>> pick which downstream tasks run (@task.llm_branch), but like any branch >>>> it >>>>> can only select from tasks already wired in the Dag. It can't decide >>> the >>>>> order or the dependencies between them, so people either put the whole >>>>> agent loop inside one opaque task, losing per-step retries, logs and >>>>> approvals, or generate Dag files at parse time. A few shapes we keep >>>> seeing: >>>>> >>>>> - Incident triage: an agent reads an alert and picks diagnosis steps, >>>>> where which probe runs next depends on what the first one found, ending >>>> in >>>>> a fix or a page to a human. >>>>> - Support tickets: classify, then run some of validate, refund, >>>>> enrich, escalate and notify, in an order that depends on the ticket, >>> with >>>>> an approval task before anything that moves money. >>>>> - Data-quality failures: quarantine, backfill, re-validate and notify, >>>>> chosen per failure. >>>>> - Selective builds, with no LLM involved: an impact analysis decides >>>>> which models to rebuild and the edges between exactly those models. >>>>> >>>>> In each case the set of allowed actions is fixed and reviewed in the >>> Dag >>>>> file; only the selection and wiring change per run. Each action stays a >>>>> real task with its own retries, logs and pool, and the catalog plus a >>>>> validator keep the model inside what the author approved. >>>>> >>>>> On the name: neither of us likes "Dynamic Task Groups". It describes >>> the >>>>> mechanism, not what you get, which is building part of your Dag at run >>>> time. >>>>> We're keeping it for the AIP so the discussion doesn't get stuck on it, >>>> but >>>>> expect it to change before it ships. >>>>> >>>>> Two that came to my mind: "Plan Mode", as in a task group in plan mode, >>>> the >>>>> way coding agents plan before they act, and "Runtime >>> subgraphs/subdags", >>>>> which pairs with Task Loops and the @planner decorator. Suggestions >>>> welcome. >>>>> >>>>> What changed since the first draft: >>>>> >>>>> - Authoring is a @planner task inside a task group, plus a catalog of >>>> what >>>>> it >>>>> may pick from (@triage.catalog, or catalog= / extra_catalog_items=). >>>> The >>>>> TaskFlow and context-manager forms are side by side on the page and >>> are >>>>> nearly identical. Catalog items can be operator classes or >>> import-path >>>>> strings, so heavy operators don't have to be imported at parse time. >>>>> >>>>> - The plan uses the same schema as YAML Dags (tasks, uses, with, >>> needs), >>>> so >>>>> Airflow has one vocabulary for declaring a graph of tasks. Templating >>>> is >>>>> explicit, as in YAML Dags: a value is rendered only when marked with >>>> $t. >>>>> >>>>> - No subclassing. An LLM planner would itself be a planner, shipped by >>> a >>>>> provider, with a validator on top. An @task.agent returning the plan >>> as >>>>> structured output should also work with no extra machinery. >>>>> >>>>> - Selected entries are real task instances, not map indices of one >>> mapped >>>>> task. The planner submits the plan through the Execution API while it >>>> is >>>>> still running, and the API server creates the task instances, which >>> is >>>> the >>>>> approach Task Loops took. The plan is stored against the region, so >>>> after >>>>> a >>>>> clear and re-plan, the earlier plan and the task instances that ran >>>> under >>>>> it >>>>> are still there to look at. >>>>> >>>>> - At parse time the group serializes as two nodes: the planner and a >>>> merge >>>>> node. The merge is always an EmptyOperator, and anything downstream >>> of >>>> the >>>>> group attaches to it. >>>>> >>>>> - Catalog membership is by exact class, so a subclass of an approved >>>>> operator >>>>> is rejected. That can be loosened later; tightening it later would be >>>> much >>>>> harder. >>>>> >>>>> Comments on the page or here are both fine. >>>>> >>>>> Thanks, >>>>> Kaxil >>>>> >>>>> On Tue, 18 Aug 2026 at 21:37, Przemysław Mirowski <[email protected]> >>>> wrote: >>>>> >>>>>> Thanks for the update Kaxil. Looks much better. I've added a couple >>> of >>>> new >>>>>> comments. >>>>>> >>>>>> On 2026/08/10 21:26:24 Kaxil Naik wrote: >>>>>>> Thanks for your feedback Przemysław, I updated the AIP >>>>>>> >>>>>>> On Fri, 7 Aug 2026 at 03:20, Przemysław Mirowski <[email protected] >>>> >>>>>> wrote: >>>>>>> >>>>>>>> Hi Kaxil, >>>>>>>> >>>>>>>> I've added some comments to the AIP. In general, the idea looks >>>>>>>> interesting. >>>>>>>> >>>>>>>> Could we possibly rearrange the sections a bit? During the >>> reading, >>>>>>>> sometimes one thing was stated when the reader does not know a >>>>>> particular >>>>>>>> part of the change (e.g. merge policy - I added a comment next to >>>> it). >>>>>>>> Also, there are a lot of new definitions - maybe it would be good >>>> to >>>>>> create >>>>>>>> a section which will gather them all and shortly explain? >>>>>>>> >>>>>>>> On 2026/07/30 10:16:20 Kaxil Naik wrote: >>>>>>>>> Hi all, >>>>>>>>> >>>>>>>>> I would like to start a discussion on AIP-113: Dynamic Task >>>> Groups. >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>> >>>> >>> https://cwiki.apache.org/confluence/spaces/AIRFLOW/pages/440304823/AIP-113+Dynamic+Task+Groups >>>>>>>>> >>>>>>>>> tldr: >>>>>>>>> Dynamic Task Mapping made cardinality dynamic: run the same >>> task >>>>>> over a >>>>>>>>> collection discovered at run time. What is still static is the >>>> shape, >>>>>>>>> meaning which tasks >>>>>>>>> run in a region and how they depend on each other. >>>>>>>>> >>>>>>>>> Branching gets you half of that already, and I want to be >>> precise >>>>>> about >>>>>>>>> which half, because >>>>>>>>> I had this wrong in an earlier draft. `choose_branch` can >>> return >>>> a >>>>>> list, >>>>>>>> so >>>>>>>>> a branch already >>>>>>>>> selects an arbitrary subset of its declared successors. What a >>>> branch >>>>>>>>> cannot do is decide >>>>>>>>> the dependency edges *among* the tasks it selected. Selection >>> is >>>>>> solved; >>>>>>>>> ordering and wiring are not. That is the >>>>>>>>> gap this AIP is aimed at. >>>>>>>>> >>>>>>>>> The proposal is a Task Group whose members and internal edges >>> are >>>>>> decided >>>>>>>>> at run time from a fixed catalog the author declares. A >>> generator >>>>>> task >>>>>>>>> emits a plan, the group materializes >>>>>>>>> exactly the selected entries as real task instances, and the >>>> region >>>>>>>>> converges on a >>>>>>>>> synthesized merge node. The catalog is what serializes, so two >>>> runs >>>>>> of >>>>>>>>> different shapes share >>>>>>>>> one DAG version, and an entry the plan did not select never >>>> becomes a >>>>>>>> task >>>>>>>>> instance. The >>>>>>>>> planner can be a control table, an impact analysis, a triage >>>> rule, >>>>>> or an >>>>>>>>> LLM. >>>>>>>>> >>>>>>>>> --- >>>>>>>>> >>>>>>>>> Would love to hear your thoughts & get feedback. Please comment >>>> on >>>>>> the >>>>>>>> AIP >>>>>>>>> doc itself. >>>>>>>>> >>>>>>>>> Regards, >>>>>>>>> Kaxil >>>>>>>>> >>>>>>>> >>>>>>>> >>>> --------------------------------------------------------------------- >>>>>>>> To unsubscribe, e-mail: [email protected] >>>>>>>> For additional commands, e-mail: [email protected] >>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>>> --------------------------------------------------------------------- >>>>>> To unsubscribe, e-mail: [email protected] >>>>>> For additional commands, e-mail: [email protected] >>>>>> >>>>>> >>>> >>>> --------------------------------------------------------------------- >>>> To unsubscribe, e-mail: [email protected] >>>> For additional commands, e-mail: [email protected] >>>> >>>> >>> >> > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
