Hi Ramit,

+1 to the idea, as we discussed earlier. The AIP has grown much more mature
since then.
Lookup works for me as a name.


Thanks & Regards,
Amogh Desai


On Wed, Nov 11, 2026 at 10:59 AM Ramit Kataria via dev <
[email protected]> wrote:

> Hi Shahar,
>
> Thanks for the review! I've updated the AIP for all three.
>
> 1. Overview
>
> There's now a short Overview section at the start of the proposal. In
> short:
>
> - The Dag author writes a lookup instead of a literal, e.g.
> pool=Lookup.param("warehouse_pool").
> - The Dag processor validates it and serializes it with the Dag.
> - When a run is created, the lookup is resolved from the run's conf or run
> type.
> - The value is written onto each task instance whenever its fields are set
> (creation, mapped expansion, clear, retry). task_instance_mutation_hook
> runs after that.
> - The scheduler reads pool, slots, priority and queue from the task
> instance, same as today.
>
> (The name "Lookup" may still change. Ash and I are discussing it on the
> wiki page.)
>
> 2. Changes after a run starts
>
> Task instances that already exist are never rewritten. A change only
> matters the next time a task instance's fields are written, so for new
> mapped instances, clears and retries.
>
> - Override: say "extract" was moved from etl_pool to heavy_pool, and
> heavy_pool is then un-nominated or deleted. The existing task instance
> keeps heavy_pool. If it is retried or cleared, the override no longer
> applies and it goes back to etl_pool.
> - Lookup: a deleted pool behaves like a deleted literal pool does today.
> The task waits and the Dag shows a warning.
>
> 3. Scope
>
> Lookups cover the cases the author can plan for. Overrides are for the
> ones nobody planned: someone on call rerunning one run with a heavy task in
> a bigger pool, or moving a run off a saturated pool, on a Dag they may not
> own and can't redeploy right now. While discussing this with Elad, he
> specifically mentioned the need for this override among users he's talked
> to.
>
> Ideally, I'd like to keep both in the AIP. They share the same path
> (resolved at run creation, applied at the same points, cluster policy
> last), and the part that needs agreement is the security model as a whole.
> Overrides are off until an administrator nominates a pool or queue, so
> nothing changes for deployments that don't opt in.
>
> > On Nov 7, 2026, at 10:24 AM, Shahar Epstein via dev <
> [email protected]> wrote:
> >
> > Hey Ramit,
> >
> > Thanks for bringing this up! The idea makes sense to me, and I’d like to
> > see it move forward.
> >
> > Could you please clarify the following points in the AIP?
> >
> >   1.
> >
> >   High-level design: Could you add a short overview or diagram showing
> how
> >   a lookup sets a scheduling field, such as pool or queue, and how that
> >   setting reaches the task instance? The AIP moves quickly from the
> problem
> >   to low-level details, which makes it hard to see the overall flow.
> >   2.
> >
> >   Changes after a run starts: What happens if a pool is un-nominated or
> >   deleted after a run starts? Are task instances that already exist
> updated,
> >   or does the change apply only when task instances are created or
> updated
> >   later?
> >   3.
> >
> >   Scope: Which use cases need trigger-time overrides? Do they need to be
> >   included in the first version alongside lookups?
> >
> >
> > Thanks,
> > Shahar
> >
> >> On Sat, Oct 31, 2026 at 2:25 AM Ramit Kataria via dev <
> >> [email protected]> wrote:
> >>
> >>
> >> Hi all,
> >>
> >> I'd like to start the discussion on AIP-117, Per-run Task Scheduling
> >> Controls:
> >>
> >>
> https://cwiki.apache.org/confluence/spaces/AIRFLOW/pages/446071748/WIP+AIP-117+Per-Run+Task+Scheduling+Controls
> >>
> >> A task's pool, pool_slots, priority_weight and queue are fixed in the
> Dag
> >> file. You can't change them for one run when you trigger it, and a Dag
> >> author can't make them depend on a param or on the kind of run. Today
> >> people duplicate tasks and branch between them, keep two copies of a
> Dag,
> >> or ask their Deployment Manager for a cluster policy that reads
> >> dag_run.conf.
> >>
> >> The proposal has two parts.
> >>
> >> 1. Lookups, for Dag authors. A task field can take its value from a Dag
> >> param, the run type, or an Airflow Variable instead of a literal, for
> >> example pool=Lookup.param("warehouse_pool"). A param used this way must
> >> declare enum (or minimum and maximum for numbers), so whoever triggers
> the
> >> run can only pick values the author listed, and trigger permission is
> >> enough.
> >>
> >> 2. Trigger-time overrides, for operators. An optional "overrides" field
> on
> >> the trigger request sets these fields for specific tasks, TaskGroups or
> >> pools, for Dags whose author didn't prepare anything. It needs Dag edit
> >> permission and only works with pools and queues an administrator has
> >> nominated. Nothing is nominated by default, so nothing changes on
> upgrade.
> >>
> >> Values are resolved when the run is created and applied whenever task
> >> instances are created, cleared or retried. A task_instance_mutation_hook
> >> still runs last and wins. The scheduler never runs user code, and its
> >> critical section is unchanged.
> >>
> >> To make this concrete, here is roughly what the docs would say once it
> >> lands:
> >>
> >> - Params: "A task's pool, pool slots, priority weight or queue can come
> >> from a param, for example pool=Lookup.param("warehouse_pool") with
> >> Param(default="wh_small", enum=["wh_small", "wh_large"]). The trigger
> form
> >> shows it as a dropdown and notes which tasks it affects. Scheduled runs
> use
> >> the default. The same value can be passed as conf from the REST API, the
> >> CLI, TriggerDagRunOperator or a backfill."
> >>
> >> - Dag runs: "If you can edit a Dag, you can override these fields for
> one
> >> run without changing the Dag, for example {"overrides": {"tasks":
> >> {"extract": {"pool": "heavy_pool"}}}}. Keys can be task IDs or TaskGroup
> >> IDs, and "pools": {"etl_pool": "etl_overflow_pool"} moves every task of
> the
> >> run from one pool to another. Only pools and queues your administrator
> has
> >> nominated can be used."
> >>
> >> - Pools: "A pool can't be used for trigger-time overrides until it's
> >> marked as an override target. Treat that as letting anyone who can edit
> a
> >> Dag send its tasks to this pool. Removing the mark takes effect
> >> immediately, and runs already in progress fall back to their normal
> pool."
> >>
> >> - Cluster policies: "Lookups and overrides are applied before
> >> task_instance_mutation_hook, so the hook keeps the last word.
> task_policy
> >> runs at parse time and only sees defaults, so put rules that must hold
> for
> >> every run in task_instance_mutation_hook."
> >>
> >> I'd especially like feedback on:
> >>
> >> - The security model: authors bound what the person triggering can pick,
> >> and administrators nominate what Dag editors can override to.
> >> - The name "Lookup", which is a working name.
> >>
> >> Thanks,
> >> Ramit
> >>
>

Reply via email to