Hi all, I would like to start a discussion on AIP-113: Dynamic Task Groups.
https://cwiki.apache.org/confluence/spaces/AIRFLOW/pages/440304823/AIP-113+Dynamic+Task+Groups tldr: Dynamic Task Mapping made cardinality dynamic: run the same task over a collection discovered at run time. What is still static is the shape, meaning which tasks run in a region and how they depend on each other. Branching gets you half of that already, and I want to be precise about which half, because I had this wrong in an earlier draft. `choose_branch` can return a list, so a branch already selects an arbitrary subset of its declared successors. What a branch cannot do is decide the dependency edges *among* the tasks it selected. Selection is solved; ordering and wiring are not. That is the gap this AIP is aimed at. The proposal is a Task Group whose members and internal edges are decided at run time from a fixed catalog the author declares. A generator task emits a plan, the group materializes exactly the selected entries as real task instances, and the region converges on a synthesized merge node. The catalog is what serializes, so two runs of different shapes share one DAG version, and an entry the plan did not select never becomes a task instance. The planner can be a control table, an impact analysis, a triage rule, or an LLM. --- Would love to hear your thoughts & get feedback. Please comment on the AIP doc itself. Regards, Kaxil
