I do like Planner Tasks On Wed, 18 Nov 2026 at 17:56, Constance Martineau via dev < [email protected]> wrote:
> For the name, I'm find myself defaulting to calling the proposed > feature Planner Tasks, especially when discussing it alongside Mapped > Tasks and Task Loops. Plan mode doesn't fit as well with the other > concepts we have. > > On Wed, Nov 18, 2026 at 12:49 PM Kaxil Naik <[email protected]> wrote: > > > > Hi all, > > > > Picking this thread back up. > > > > Airflow 3.4 shipped Task Loops and YAML Dags, and both made AIP-113 > > simpler, so Ash and I reworked it on top of them: selected tasks are now > > created the way loop iterations are, and the plan uses the YAML Dag > schema. > > The page is updated: > > > > > https://cwiki.apache.org/confluence/spaces/AIRFLOW/pages/440304823/AIP-113+Dynamic+Task+Groups > > > > Thanks Przemysław for the earlier comments. > > > > Why now: the Common AI provider reached 1.0 in October, and more of the > > Dags people write have a model deciding what happens next. Today an LLM > can > > pick which downstream tasks run (@task.llm_branch), but like any branch > it > > can only select from tasks already wired in the Dag. It can't decide the > > order or the dependencies between them, so people either put the whole > > agent loop inside one opaque task, losing per-step retries, logs and > > approvals, or generate Dag files at parse time. A few shapes we keep > seeing: > > > > - Incident triage: an agent reads an alert and picks diagnosis steps, > > where which probe runs next depends on what the first one found, ending > in > > a fix or a page to a human. > > - Support tickets: classify, then run some of validate, refund, > > enrich, escalate and notify, in an order that depends on the ticket, with > > an approval task before anything that moves money. > > - Data-quality failures: quarantine, backfill, re-validate and notify, > > chosen per failure. > > - Selective builds, with no LLM involved: an impact analysis decides > > which models to rebuild and the edges between exactly those models. > > > > In each case the set of allowed actions is fixed and reviewed in the Dag > > file; only the selection and wiring change per run. Each action stays a > > real task with its own retries, logs and pool, and the catalog plus a > > validator keep the model inside what the author approved. > > > > On the name: neither of us likes "Dynamic Task Groups". It describes the > > mechanism, not what you get, which is building part of your Dag at run > time. > > We're keeping it for the AIP so the discussion doesn't get stuck on it, > but > > expect it to change before it ships. > > > > Two that came to my mind: "Plan Mode", as in a task group in plan mode, > the > > way coding agents plan before they act, and "Runtime subgraphs/subdags", > > which pairs with Task Loops and the @planner decorator. Suggestions > welcome. > > > > What changed since the first draft: > > > > - Authoring is a @planner task inside a task group, plus a catalog of > what > > it > > may pick from (@triage.catalog, or catalog= / extra_catalog_items=). > The > > TaskFlow and context-manager forms are side by side on the page and are > > nearly identical. Catalog items can be operator classes or import-path > > strings, so heavy operators don't have to be imported at parse time. > > > > - The plan uses the same schema as YAML Dags (tasks, uses, with, needs), > so > > Airflow has one vocabulary for declaring a graph of tasks. Templating > is > > explicit, as in YAML Dags: a value is rendered only when marked with > $t. > > > > - No subclassing. An LLM planner would itself be a planner, shipped by a > > provider, with a validator on top. An @task.agent returning the plan as > > structured output should also work with no extra machinery. > > > > - Selected entries are real task instances, not map indices of one mapped > > task. The planner submits the plan through the Execution API while it > is > > still running, and the API server creates the task instances, which is > the > > approach Task Loops took. The plan is stored against the region, so > after > > a > > clear and re-plan, the earlier plan and the task instances that ran > under > > it > > are still there to look at. > > > > - At parse time the group serializes as two nodes: the planner and a > merge > > node. The merge is always an EmptyOperator, and anything downstream of > the > > group attaches to it. > > > > - Catalog membership is by exact class, so a subclass of an approved > > operator > > is rejected. That can be loosened later; tightening it later would be > much > > harder. > > > > Comments on the page or here are both fine. > > > > Thanks, > > Kaxil > > > > On Tue, 18 Aug 2026 at 21:37, Przemysław Mirowski <[email protected]> > wrote: > > > > > Thanks for the update Kaxil. Looks much better. I've added a couple of > new > > > comments. > > > > > > On 2026/08/10 21:26:24 Kaxil Naik wrote: > > > > Thanks for your feedback Przemysław, I updated the AIP > > > > > > > > On Fri, 7 Aug 2026 at 03:20, Przemysław Mirowski <[email protected]> > > > wrote: > > > > > > > > > Hi Kaxil, > > > > > > > > > > I've added some comments to the AIP. In general, the idea looks > > > > > interesting. > > > > > > > > > > Could we possibly rearrange the sections a bit? During the reading, > > > > > sometimes one thing was stated when the reader does not know a > > > particular > > > > > part of the change (e.g. merge policy - I added a comment next to > it). > > > > > Also, there are a lot of new definitions - maybe it would be good > to > > > create > > > > > a section which will gather them all and shortly explain? > > > > > > > > > > On 2026/07/30 10:16:20 Kaxil Naik wrote: > > > > > > Hi all, > > > > > > > > > > > > I would like to start a discussion on AIP-113: Dynamic Task > Groups. > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/spaces/AIRFLOW/pages/440304823/AIP-113+Dynamic+Task+Groups > > > > > > > > > > > > tldr: > > > > > > Dynamic Task Mapping made cardinality dynamic: run the same task > > > over a > > > > > > collection discovered at run time. What is still static is the > shape, > > > > > > meaning which tasks > > > > > > run in a region and how they depend on each other. > > > > > > > > > > > > Branching gets you half of that already, and I want to be precise > > > about > > > > > > which half, because > > > > > > I had this wrong in an earlier draft. `choose_branch` can return > a > > > list, > > > > > so > > > > > > a branch already > > > > > > selects an arbitrary subset of its declared successors. What a > branch > > > > > > cannot do is decide > > > > > > the dependency edges *among* the tasks it selected. Selection is > > > solved; > > > > > > ordering and wiring are not. That is the > > > > > > gap this AIP is aimed at. > > > > > > > > > > > > The proposal is a Task Group whose members and internal edges are > > > decided > > > > > > at run time from a fixed catalog the author declares. A generator > > > task > > > > > > emits a plan, the group materializes > > > > > > exactly the selected entries as real task instances, and the > region > > > > > > converges on a > > > > > > synthesized merge node. The catalog is what serializes, so two > runs > > > of > > > > > > different shapes share > > > > > > one DAG version, and an entry the plan did not select never > becomes a > > > > > task > > > > > > instance. The > > > > > > planner can be a control table, an impact analysis, a triage > rule, > > > or an > > > > > > LLM. > > > > > > > > > > > > --- > > > > > > > > > > > > Would love to hear your thoughts & get feedback. Please comment > on > > > the > > > > > AIP > > > > > > doc itself. > > > > > > > > > > > > Regards, > > > > > > Kaxil > > > > > > > > > > > > > > > > > --------------------------------------------------------------------- > > > > > To unsubscribe, e-mail: [email protected] > > > > > For additional commands, e-mail: [email protected] > > > > > > > > > > > > > > > > > > > > --------------------------------------------------------------------- > > > To unsubscribe, e-mail: [email protected] > > > For additional commands, e-mail: [email protected] > > > > > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > >
