Hi Shahar,
I resonate with Jarek and Ash's performance concerns, trigger rule evaluation runs on every task candidate every scheduler cycle, so even a small per-call overhead will compound really fast at scale. One thing worth trying if you are exploring options (after you manage to handle the concerns above) -- Compile the expression into a plain Python function at DAG parse time. I do not promise that it will make things better but that way the scheduler just calls a function like it does today -- no expression parsing in the hot path at all. Might be worth benchmarking that approach too at some scale. Thanks & Regards, Amogh Desai On Tue, May 5, 2026 at 11:14 PM Shahar Epstein <[email protected]> wrote: > Thanks Jarek and Ash for your feedback. > Your concerns regarding performance are valid - I'll check if it can be > addressed. > > > Shahar > > On Tue, May 5, 2026, 18:13 Ash Berlin-Taylor <[email protected]> wrote: > > > +1 to everything Jarek said. On point A, I’d go so far as to ”not might, > > but almost certainly will cause performance regression”. > > > > Trigger rule evaluation is the hottest of hot paths in the scheduler, and > > any change to that area of the code needs to be done with extreme care > and > > attention. > > > > -ash > > > > > On 2 May 2026, at 18:03, Jarek Potiuk <[email protected]> wrote: > > > > > > Hi Shahar, > > > > > > I am not against this proposal, but I have three concerns regarding > > points > > > not fully covered in the AIP: > > > > > > a) Performance: Trigger evaluation occurs in the critical scheduler > loop. > > > Moving from hard-coded Python criteria to a generalized expression > model > > > might introduce a performance hit during the iteration of task > > candidates. > > > > > > b) Complexity and Reasoning: Allowing arbitrary combinations could lead > > to > > > contradicting or nonsensical conditions (e.g., requirements that exceed > > the > > > total number of upstream tasks). This complicates debugging for users > > > asking "why is my task not running?" and makes it harder for > maintainers > > to > > > discern the original author's intent compared to clear enums like > > > ALL_FAILED. > > > > > > c) Flexibility vs. Intention: The proposed primitives feel more > > > mathematical than business-logical. If a user doesn't care which > specific > > > upstream tasks succeed—only that a certain number do—those tasks are > > likely > > > "flavors" of the same work. Such logic might be better handled by > > extending > > > "mapped task" success criteria (e.g., a percentage threshold) rather > than > > > modifying trigger rules for distinct upstream nodes. > > > > > > I'd love to hear your thoughts and those of others on whether this > > > complexity aligns with what DAG authors actually need. > > > > > > Best, > > > Jarek Potiuk > > > > > > On Sat, May 2, 2026 at 11:40 AM Jens Scheffler <[email protected]> > > wrote: > > > > > >> Hi Shahar, > > >> > > >> ich think this is a good extension for cases needing this withou > adding > > >> too much complexity. Whereas I am not sure whether in our environment > we > > >> have a use case. Would be good to hear from others if there are any. > > >> Alternatives otherwise in my mind would only be adding more enum keys > > >> for other cases (which is not really preferrable) or modelling the Dag > > >> with a few EmptyOperators to collect pre-aggregate counts to leverage > > >> existing enums... which would in turn create artificial Dag complexity > > >> just because of limited enums. > > >> > > >> For me OK but would be cool to have other opinions as well. > > >> > > >> Jens > > >> > > >> On 02.05.26 11:16, Shahar Epstein wrote: > > >>> Bumping, I'll appreciate your feedback. > > >>> > > >>> > > >>> Shahar > > >>> > > >>> On Tue, Apr 21, 2026 at 2:00 PM Shahar Epstein <[email protected]> > > >> wrote: > > >>>> Hey everyone, > > >>>> > > >>>> I'd like to get your feedback on a new AIP proposal: > > >>>> > > >>>> AIP-106: Composable Trigger Rules > > >>>> https://cwiki.apache.org/confluence/x/64wmGQ > > >>>> > > >>>> tl;dr > > >>>> > > >>>> This proposal introduces a composable expression model for trigger > > >> rules, enabling conditions not expressible with the current enum-based > > >> system, while fully preserving existing trigger rules for simple > cases. > > >>>> For example, "All done, at least two succeeded" can be expressed as > > >> TR.expr(done="all", success=">=2"). > > >>>> > > >>>> Design Notes/Constraints: > > >>>> 1. The aggregate form is AND-only by design, to preserve determinism > > >> and keep evaluation predictable. > > >>>> 2. The composable model complements (rather than replaces) the enum. > > >> Some trigger rules (e.g. ALL_DONE_SETUP_SUCCESS, ONE_DONE, ALWAYS) > > remain > > >> intentionally out of scope, and enums remain the more readable option > > for > > >> simple cases. > > >>>> > > >>>> I'd appreciate your feedback on: > > >>>> > > >>>> Backward compatibility with existing trigger rules (see Appendix A) > > >>>> > > >>>> Semantics of eager vs complete evaluation > > >>>> > > >>>> Scope and limits of the expression model > > >>>> > > >>>> Potential edge cases and performance implications > > >>>> > > >>>> UI/observability considerations for debugging expressions > > >>>> > > >>>> Anything else > > >>>> > > >>>> > > >>>> Thank you! > > >>>> > > >>>> > > >>>> Shahar > > >>> --------------------------------------------------------------------- > > >>> To unsubscribe, e-mail: [email protected] > > >>> For additional commands, e-mail: [email protected] > > >>> > > >> > > >> --------------------------------------------------------------------- > > >> To unsubscribe, e-mail: [email protected] > > >> For additional commands, e-mail: [email protected] > > >> > > >> > > > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: [email protected] > > For additional commands, e-mail: [email protected] > > > > >
