I would keep the semantics as simple and predictable as possible, especially from the perspective of the person receiving the alert.
To me, there are three distinct operational questions: A. Did the DAG run fail? → Self-mitigation or human intervention. B. Have we reached HH:mm and the DAG has not succeeded? → Act on the missed deadline. C. Is the DAG run likely to miss its SLA? → Proactive intervention. Each question represents a different signal and potentially a different operational response. If I configure a Deadline Alert to answer question B, I would find it confusing if it fired earlier simply because the DAG failed. I already have failure alerts for that. A DAG might fail at 09:10, get retried or recovered, and still succeed before the 10:00 deadline. I don't want the deadline alert to fire at 09:10, nor do I want the failure to cancel the deadline check. So option 3 from Dennis's list is the closest to my expectation, with one important distinction: at the deadline, I care whether the DAG has succeeded. If it has recovered and succeeded before then, there should be no alert. Ultimately, simplicity is key for me. I want each alert to answer one clear question and drive one clear operational decision. I think mixing failure detection, deadline breaches, and predictions would make operations harder. Thanks Alex Guglielmone Nemi On Wed, Oct 21, 2026, 04:54 Dev-iL <[email protected]> wrote: > "Why not both?" > > We have a retry policy for exceptions, why not a deadline policy? > > To me, a deadline always meant an answer to the **business** concern of > "data is n/a by time X". My equivalent for this in airflow2 is a > dataset+timetable schedule, where a timed run represents the deadline check > and always branches into a sensor that checks if the output is missing, > then immediately exits with the outcome. > > Under this framing, time measured from the start of a run and whether a > given dagrun stopped ahead of time are immaterial - it could've been rerun > (same or different dagrun), an external recovery processes could've kicked > in, etc. -- if the data is available by the end of this, I don't want to be > notified, and vice versa. > > If, on the other hand, we treat deadlines as "nonfatal" versions of > timeouts - then I think 2 - the callback should never fire and whatever > side-effect the deadline callback has should be included in the timeout > callback. I struggle to see the usefulness of 3 beyond what a failure > callback already gives. > > Best, > Dev-iL > > > On Wed, 21 Oct 2026, 1:05 Dennis Ferruzzi, <[email protected]> wrote: > > > We've circled this question a few times and I am trying to nail down an > > answer that we can implement. > > > > Given this situation: You create a Dag with a Deadline Alert for one hour > > from the Dag's queued_at time. Ten minutes in, the Dag fails. What do > you > > expect to happen? > > > > There are three clear options here: > > > > 1. The Deadline callback fires at ten minutes (on Dagrun failure). If > the > > Deadline Alert means "I expect this to succeed in under an hour," then > > after 10 minutes we know it can't succeed, so fire now. This is what PR > > #68961 implements (behind an opt-in flag). > > > > 2. The callback never fires. If the Deadline Alert means "I expect this > > to finish within an hour," then a failed run has finished, so the > deadline > > is satisfied and the pending alert is pruned, the same way we already > prune > > it on success. > > > > 3. The callback fires at the one-hour mark regardless. The deadline > timer > > runs to expiry independent of the run's state, so a failed run still > > triggers the alert 50 minutes after it died. This is the current > behaviour > > on main. > > > > What the docs say right now (emphasis mine, and keeping in mind we can > > change this based on the decision here): > > > > `core-concepts/dags.rst` says: "This example will send an email > > notification if the Dag **hasn't finished** 30 minutes after it was > queued." > > > > `howto/deadline-alerts.rst` says the DAGRUN_LOGICAL_DATE reference "would > > trigger the alert if the Dag **hasn't completed** 15 minutes after it was > > scheduled to start," and describes AVERAGE_RUNTIME as alerting when a > "Dag > > is **running longer** than expected." > > > > So the docs currently frame it as (not finished | not completed | still > > running), they do not say `not succeeded`. By that phrasing, a failed > run > > HAS finished, which points at option 2, and the user can use the existing > > `on_failure_callback` if that was what they wanted . > > > > But `howto/sla-to-deadlines.rst` describes the mechanism differently: > > "When a Dag run starts, calculate and store (DeadlineReference + > > interval). The scheduler loop then checks periodically ... **if any of > > those times have passed** then execute callback," and stresses the > callback > > "will execute 'immediately' ... and not wait until the Dag finishes > > first." That description has no step that cancels a pending deadline > when > > the run ends, which is the only place option 3 (today's behaviour) finds > > any support. > > > > So the documentation contradicts itself: the conceptual pages point to > > option 1 or option 2, depending on how you read it, and the migration > > guide's mechanism describes option 3. So I am asking: which of the three > > SHOULD be the behaviour, and should any of the others be an opt-in > > alternative? > > > > My own opinion is that option 1 was the intent, and I'd like to see it > > implemented as a bug fix rather than an optional flag: a deadline means > > "did not succeed in time," a failed run cannot succeed, so fire. If we > > keep a flag at all I'd make option 1 the default and allow an opt-out to > > prune on failure (option 2) for teams who already get failure signal from > > on_failure_callback and don't want a duplicate. The counter-argument > worth > > weighing, raised on #68961: an on-call who clears a failed run and > retries > > would, under option 1, lose the "still hasn't recovered by the deadline" > > signal, which is an argument for option 2 or for keeping timing-based > > firing available. > > > > Could you reply with the number you'd expect as a user, and whether you'd > > want the other behaviours available as an option? If there's rough > > consensus I'll make the docs and #68961 match it. > > > > Thanks, > > - ferruzzi > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: [email protected] > > For additional commands, e-mail: [email protected] > > > > >
