Hey Mathieu, I appreciate you contributing this back! I don't work very closely with Helm, so I'm not the best candidate to review this proposal, but from a multi team perspective, I'd really love to have this, especially for triggerers, where having one per team is critical. I think the proposal looks reasonable to me. If you haven't already, I'd start putting up some PRs if you're not getting much traction here.
Cheers, Niko On Mon, Oct 12, 2026 at 4:37 AM Mathieu Monet <[email protected]> wrote: > Hi all, > > I'd like to propose adding `sets` to the triggerer and dag processor in > the Helm chart, following the existing `workers.celery.sets` pattern, and > get direction before opening PRs. > > Why > --- > With `core.multi_team` enabled, each team needs a triggerer started with > `--team-name`, otherwise deferred tasks of that team's Dags never resume > (multi-team.rst, "Triggerer" section). The chart can only render one > triggerer today, so a multi-team deployment on the official chart is > not fully functional without patching templates. Running one dag > processor per team or bundle (`--bundle-name`) is the matching need on > the parsing side, for isolation and sizing. > > We run this in production (Airflow 3.3.2, several teams in one instance) > through a wrapper chart that copies the upstream triggerer and dag > processor templates in a loop. I'd rather contribute to it than keep that > copy in sync. > > Proposal > -------- > triggerer: > enableDefault: true # render the unsuffixed triggerer > teamName: ~ # -> --team-name > queues: ~ # -> --queues > extraArgs: [] # appended to the default args > sets: [] # each item overrides any triggerer.* key > dagProcessor: > enableDefault: true > bundleNames: [] # -> repeated --bundle-name > extraArgs: [] > sets: [] > > - A set `<n>` renders `<fullname>-<component>-<n>` with its own > ServiceAccount, PDB and ScaledObject, and a `<component>-set: <n>` > label. Values merge exactly like worker sets (shared helper). > - Sets reference bundles and teams by name only. Render-time validation > checks them against dagBundleConfigList, so there is no second list > to keep in sync (the concern raised on #61039). Users set small scope > fields instead of a full CLI in `args` (the concern raised on #61963). > - With every new field empty, the output is identical to today's. > > Planned as small PRs: shared set helpers (generalising > workersMergeValues), triggerer.sets, dagProcessor.sets, then optional > multi-team conveniences (deriving bundles from team_name, team-aware > KEDA query). > > Related: #26494, #61037, #61039, #61963, #64037. > > Questions > --------- > 1. For the dag processor, is a separate `sets` list (referencing > bundles by name) acceptable, or would you prefer per-bundle overrides > on the dagBundleConfigList entries (as discussed on #61039)? The > triggerer is scoped by team, not bundle, so `sets` seems the natural > fit there either way. > 2. Should this target 2.0.0? > 3. The default instance's selectors also match set pods (same as worker > sets today). Worth adding a `<component>-set: default` label to all > three components while the 2.0 window allows breaking changes? > > Thanks, > Mathieu Monet > > -- > Drafted-by: Claude Code (AI assistant) and > reviewed by me before sending. > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > >
