Hi Shahar, I am not against this proposal, but I have three concerns regarding points not fully covered in the AIP:
a) Performance: Trigger evaluation occurs in the critical scheduler loop. Moving from hard-coded Python criteria to a generalized expression model might introduce a performance hit during the iteration of task candidates. b) Complexity and Reasoning: Allowing arbitrary combinations could lead to contradicting or nonsensical conditions (e.g., requirements that exceed the total number of upstream tasks). This complicates debugging for users asking "why is my task not running?" and makes it harder for maintainers to discern the original author's intent compared to clear enums like ALL_FAILED. c) Flexibility vs. Intention: The proposed primitives feel more mathematical than business-logical. If a user doesn't care which specific upstream tasks succeed—only that a certain number do—those tasks are likely "flavors" of the same work. Such logic might be better handled by extending "mapped task" success criteria (e.g., a percentage threshold) rather than modifying trigger rules for distinct upstream nodes. I'd love to hear your thoughts and those of others on whether this complexity aligns with what DAG authors actually need. Best, Jarek Potiuk On Sat, May 2, 2026 at 11:40 AM Jens Scheffler <[email protected]> wrote: > Hi Shahar, > > ich think this is a good extension for cases needing this withou adding > too much complexity. Whereas I am not sure whether in our environment we > have a use case. Would be good to hear from others if there are any. > Alternatives otherwise in my mind would only be adding more enum keys > for other cases (which is not really preferrable) or modelling the Dag > with a few EmptyOperators to collect pre-aggregate counts to leverage > existing enums... which would in turn create artificial Dag complexity > just because of limited enums. > > For me OK but would be cool to have other opinions as well. > > Jens > > On 02.05.26 11:16, Shahar Epstein wrote: > > Bumping, I'll appreciate your feedback. > > > > > > Shahar > > > > On Tue, Apr 21, 2026 at 2:00 PM Shahar Epstein <[email protected]> > wrote: > >> Hey everyone, > >> > >> I'd like to get your feedback on a new AIP proposal: > >> > >> AIP-106: Composable Trigger Rules > >> https://cwiki.apache.org/confluence/x/64wmGQ > >> > >> tl;dr > >> > >> This proposal introduces a composable expression model for trigger > rules, enabling conditions not expressible with the current enum-based > system, while fully preserving existing trigger rules for simple cases. > >> For example, "All done, at least two succeeded" can be expressed as > TR.expr(done="all", success=">=2"). > >> > >> Design Notes/Constraints: > >> 1. The aggregate form is AND-only by design, to preserve determinism > and keep evaluation predictable. > >> 2. The composable model complements (rather than replaces) the enum. > Some trigger rules (e.g. ALL_DONE_SETUP_SUCCESS, ONE_DONE, ALWAYS) remain > intentionally out of scope, and enums remain the more readable option for > simple cases. > >> > >> I'd appreciate your feedback on: > >> > >> Backward compatibility with existing trigger rules (see Appendix A) > >> > >> Semantics of eager vs complete evaluation > >> > >> Scope and limits of the expression model > >> > >> Potential edge cases and performance implications > >> > >> UI/observability considerations for debugging expressions > >> > >> Anything else > >> > >> > >> Thank you! > >> > >> > >> Shahar > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: [email protected] > > For additional commands, e-mail: [email protected] > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > >
