Hey Ramit, Thanks for bringing this up! The idea makes sense to me, and I’d like to see it move forward.
Could you please clarify the following points in the AIP? 1. High-level design: Could you add a short overview or diagram showing how a lookup sets a scheduling field, such as pool or queue, and how that setting reaches the task instance? The AIP moves quickly from the problem to low-level details, which makes it hard to see the overall flow. 2. Changes after a run starts: What happens if a pool is un-nominated or deleted after a run starts? Are task instances that already exist updated, or does the change apply only when task instances are created or updated later? 3. Scope: Which use cases need trigger-time overrides? Do they need to be included in the first version alongside lookups? Thanks, Shahar On Sat, Oct 31, 2026 at 2:25 AM Ramit Kataria via dev < [email protected]> wrote: > > Hi all, > > I'd like to start the discussion on AIP-117, Per-run Task Scheduling > Controls: > > https://cwiki.apache.org/confluence/spaces/AIRFLOW/pages/446071748/WIP+AIP-117+Per-Run+Task+Scheduling+Controls > > A task's pool, pool_slots, priority_weight and queue are fixed in the Dag > file. You can't change them for one run when you trigger it, and a Dag > author can't make them depend on a param or on the kind of run. Today > people duplicate tasks and branch between them, keep two copies of a Dag, > or ask their Deployment Manager for a cluster policy that reads > dag_run.conf. > > The proposal has two parts. > > 1. Lookups, for Dag authors. A task field can take its value from a Dag > param, the run type, or an Airflow Variable instead of a literal, for > example pool=Lookup.param("warehouse_pool"). A param used this way must > declare enum (or minimum and maximum for numbers), so whoever triggers the > run can only pick values the author listed, and trigger permission is > enough. > > 2. Trigger-time overrides, for operators. An optional "overrides" field on > the trigger request sets these fields for specific tasks, TaskGroups or > pools, for Dags whose author didn't prepare anything. It needs Dag edit > permission and only works with pools and queues an administrator has > nominated. Nothing is nominated by default, so nothing changes on upgrade. > > Values are resolved when the run is created and applied whenever task > instances are created, cleared or retried. A task_instance_mutation_hook > still runs last and wins. The scheduler never runs user code, and its > critical section is unchanged. > > To make this concrete, here is roughly what the docs would say once it > lands: > > - Params: "A task's pool, pool slots, priority weight or queue can come > from a param, for example pool=Lookup.param("warehouse_pool") with > Param(default="wh_small", enum=["wh_small", "wh_large"]). The trigger form > shows it as a dropdown and notes which tasks it affects. Scheduled runs use > the default. The same value can be passed as conf from the REST API, the > CLI, TriggerDagRunOperator or a backfill." > > - Dag runs: "If you can edit a Dag, you can override these fields for one > run without changing the Dag, for example {"overrides": {"tasks": > {"extract": {"pool": "heavy_pool"}}}}. Keys can be task IDs or TaskGroup > IDs, and "pools": {"etl_pool": "etl_overflow_pool"} moves every task of the > run from one pool to another. Only pools and queues your administrator has > nominated can be used." > > - Pools: "A pool can't be used for trigger-time overrides until it's > marked as an override target. Treat that as letting anyone who can edit a > Dag send its tasks to this pool. Removing the mark takes effect > immediately, and runs already in progress fall back to their normal pool." > > - Cluster policies: "Lookups and overrides are applied before > task_instance_mutation_hook, so the hook keeps the last word. task_policy > runs at parse time and only sees defaults, so put rules that must hold for > every run in task_instance_mutation_hook." > > I'd especially like feedback on: > > - The security model: authors bound what the person triggering can pick, > and administrators nominate what Dag editors can override to. > - The name "Lookup", which is a working name. > > Thanks, > Ramit >
