Re: generalization: Nice idea Daniel, but are you sure we can come up (and agree) with a generic enough approach and have it still done for Airflow 3. For me this is a bit YAGNI - and rather than generalising now, it would be better to hash it out, vote and see all the considerations in a new AIP - we are complicating the simple "deadline" feature to a more generic expectations that we might or might not need or not agree to and we don't even know what those would be. Any generalization makes it more complex to grasp by the users, so if we generalize - I think we should very good idea how to.
I'd say we stick with the deadline as agreed and approved in AIP-86 and we can have a more generic "expectations" discussion later. It's very easy later to change it and deprecate this one, otherwise we are risking that whatever we come up with now will be half-baked and we will have to change it later anyway, because we miss an important "generalization" feature. My 3 cents. J. On Tue, Feb 25, 2025 at 3:56 PM Daniel Standish <daniel.stand...@astronomer.io.invalid> wrote: > Thinking about this interface a little bit I have some thoughts to share. > > I think it might be good to generalize it very slightly to "expectations" > so that we can set multiple different kinds of expectations, and not just > deadlines. > > I also thought it might be helpful to make clear what is the reference > event and what is the target event, and to think of these as events. It > might make it a little easier to understand. > > Something like this > > with DAG( > dag_id="something", > expectations=[ > DagRunEventDeadline( # this is about interval betwween queued > and finished > reference_event=DagRunEvent.QUEUED, > target_event=DagRunEvent.END, > max_interval=datetime.timedelta(hour=1), > ), > DagRunDateDeadline( # this is about the interval between last > run and this one > reference_event=DagRunEvent.LAST_START, > target_event=DagRunEvent.START, > max_interval=datetime.timedelta(hour=1), > ), > RunFrequenceyExpectation( # this is about the interval > between last run and this one > expected_runs=5, > lookback_interval=timedelta(days=3), > ), > ], > ) as dag: > ... > > My "run frequency expectation" is completely made up but it's just there to > show, maybe there would be interesting things we could do re allowing users > to set expectations that are not specifically limited to deadlines. In > this example, the expectation that there should be 5 runs in last N time. > I'm sure we can all think of others. We could just implement some deadline > expectations, and maybe others could implement others. And maybe we could > allow users to define their own expectations, but these, if not known by > airflow, would just be stored as json, and users could roll their own > system to evaluate them or leave it to 3rd party service etc. > > WDYT!?!? > > > > > > > > On Tue, Feb 25, 2025 at 6:43 AM Daniel Standish < > daniel.stand...@astronomer.io> wrote: > > > also... curious can you confirm my understanding of what this means > > > > `landing_time = DeadlineReference.DAGRUN_QUEUED + timedelta(hours=1)` > > > > does this mean that the expectation will be missed if the *dag run does > > not complete* within 1 hr of dag run queued time? > > > > i.e. is it correct the event of interest is dag run completion? and the > > "anchor" is, dag run queued time? > > > > > > > > > > On Tue, Feb 25, 2025 at 6:33 AM Daniel Standish < > > daniel.stand...@astronomer.io> wrote: > > > >> or "expected_by" i like expected_by over need_by > >> > >> On Tue, Feb 25, 2025 at 6:26 AM Ash Berlin-Taylor <a...@apache.org> > wrote: > >> > >>> Breaking with the herd/answering a question you didn’t even ask > >>> (:evil-grin:) > >>> > >>> How about changing need_by to landing_time, i.e. > >>> > >>> `landing_time = DeadlineReference.DAGRUN_QUEUED + timedelta(hours=1)` > >>> > >>> Or some variation there of - `landed_by` etc. > >>> -ash > >>> > >>> > >>> > >>> > On 25 Feb 2025, at 05:15, Ankit Chaurasia <sunank...@gmail.com> > wrote: > >>> > > >>> > DeadlineReference +1. > >>> > > >>> > Regards, > >>> > *Ankit Chaurasia* > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > On Tue, 25 Feb 2025 at 10:05, Pavankumar Gopidesu < > >>> gopidesupa...@gmail.com> > >>> > wrote: > >>> > > >>> >> +1 `DeadlineReference` > >>> >> > >>> >> Regards > >>> >> Pavan Kumar > >>> >> > >>> >> On Tue, Feb 25, 2025, 04:11 Amogh Desai <amoghdesai....@gmail.com> > >>> wrote: > >>> >> > >>> >>> Late to the party, but I'd vote for `DeadlineReference` too. > Concise > >>> and > >>> >>> does the job well. > >>> >>> > >>> >>> Thanks & Regards, > >>> >>> Amogh Desai > >>> >>> > >>> >>> > >>> >>> On Tue, Feb 25, 2025 at 7:56 AM Wei Lee <weilee...@gmail.com> > wrote: > >>> >>> > >>> >>>> A bit late to the party, but DeadlineReference +1. > >>> >>>> > >>> >>>> Best, > >>> >>>> Wei > >>> >>>> > >>> >>>>> On Feb 25, 2025, at 4:02 AM, Daniel Standish > >>> >>>> <daniel.stand...@astronomer.io.invalid> wrote: > >>> >>>>> > >>> >>>>> Stewing presently thank you :) > >>> >>>>> > >>> >>>>> On Mon, Feb 24, 2025 at 11:53 AM Ferruzzi, Dennis > >>> >>>>> <ferru...@amazon.com.invalid> wrote: > >>> >>>>> > >>> >>>>>> Alright then. DeadlineReference gets the green light for now! > >>> If > >>> >>>> anyone > >>> >>>>>> has a suggestion they like more, please feel free to drop it in > >>> >> here. > >>> >>>> I'm > >>> >>>>>> working on some Ariflow3.0 stuff before I get back to this, so > >>> >> there's > >>> >>>>>> plenty of time before this is set down if anyone wants to stew > on > >>> >> it a > >>> >>>> bit. > >>> >>>>>> > >>> >>>>>> Thanks for the thoughts, all! > >>> >>>>>> > >>> >>>>>> > >>> >>>>>> - ferruzzi > >>> >>>>>> > >>> >>>>>> > >>> >>>>>> ________________________________ > >>> >>>>>> From: Jarek Potiuk <ja...@potiuk.com> > >>> >>>>>> Sent: Monday, February 24, 2025 11:11 AM > >>> >>>>>> To: dev@airflow.apache.org > >>> >>>>>> Subject: RE: [EXT] Two Hard Things: Deadline Alerts Edition > >>> >>>>>> > >>> >>>>>> CAUTION: This email originated from outside of the organization. > >>> Do > >>> >>> not > >>> >>>>>> click links or open attachments unless you can confirm the > sender > >>> >> and > >>> >>>> know > >>> >>>>>> the content is safe. > >>> >>>>>> > >>> >>>>>> > >>> >>>>>> > >>> >>>>>> AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur > >>> >>>> externe. > >>> >>>>>> Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si > vous > >>> ne > >>> >>>> pouvez > >>> >>>>>> pas confirmer l’identité de l’expéditeur et si vous n’êtes pas > >>> >> certain > >>> >>>> que > >>> >>>>>> le contenu ne présente aucun risque. > >>> >>>>>> > >>> >>>>>> > >>> >>>>>> > >>> >>>>>> Yep. DaeadlineReference is good . > >>> >>>>>> > >>> >>>>>> > >>> >>>>>> On Mon, Feb 24, 2025 at 8:07 PM Mehta, Shubham > >>> >>>> <shu...@amazon.com.invalid> > >>> >>>>>> wrote: > >>> >>>>>> > >>> >>>>>>> +1 to DeadlineReference. It is clear and allows flexibility to > >>> >> choose > >>> >>>> any > >>> >>>>>>> reference point. > >>> >>>>>>> > >>> >>>>>>> shubham > >>> >>>>>>> > >>> >>>>>>> On 2025-02-24, 10:00 AM, "Ferruzzi, Dennis" > >>> >>> <ferru...@amazon.com.inva > >>> >>>>>>> <mailto:ferru...@amazon.com.inva>LID> wrote: > >>> >>>>>>> > >>> >>>>>>> > >>> >>>>>>> CAUTION: This email originated from outside of the > organization. > >>> Do > >>> >>> not > >>> >>>>>>> click links or open attachments unless you can confirm the > sender > >>> >> and > >>> >>>>>> know > >>> >>>>>>> the content is safe. > >>> >>>>>>> > >>> >>>>>>> > >>> >>>>>>> > >>> >>>>>>> > >>> >>>>>>> > >>> >>>>>>> > >>> >>>>>>> AVERTISSEMENT: Ce courrier électronique provient d’un > expéditeur > >>> >>>> externe. > >>> >>>>>>> Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si > vous > >>> >> ne > >>> >>>>>> pouvez > >>> >>>>>>> pas confirmer l’identité de l’expéditeur et si vous n’êtes pas > >>> >>> certain > >>> >>>>>> que > >>> >>>>>>> le contenu ne présente aucun risque. > >>> >>>>>>> > >>> >>>>>>> > >>> >>>>>>> > >>> >>>>>>> > >>> >>>>>>> > >>> >>>>>>> > >>> >>>>>>> Hey folks. I need to narrow down the name for one of the > >>> parameters > >>> >>> on > >>> >>>>>> the > >>> >>>>>>> Deadline Alerts work and I'm fishing for suggestions. > >>> >>>>>>> > >>> >>>>>>> > >>> >>>>>>> TLDR: need_by_date = {some existing timestamp} + {a > user-defined > >>> >>>>>> timedelta} > >>> >>>>>>> > >>> >>>>>>> > >>> >>>>>>> The existing timestamp could be dynamic (like when the dagrun > is > >>> >>> queued > >>> >>>>>> or > >>> >>>>>>> when a specific task stars, etc), or fixed datetime. > >>> >>>>>>> > >>> >>>>>>> > >>> >>>>>>> So in practice this will look something like: > >>> >>>>>>> > >>> >>>>>>> > >>> >>>>>>> need_by = DeadlineTrigger.DAGRUN_QUEUED + timedelta(hours=1) > >>> >>>>>>> > >>> >>>>>>> > >>> >>>>>>> where the DeadlineTrigger is the part we are trying to rename. > >>> >>>>>>> > >>> >>>>>>> > >>> >>>>>>> I initially used Anchor in the AIP for lack of a better name > and > >>> it > >>> >>> was > >>> >>>>>>> universally hated. I like Trigger better, but that name is > >>> already > >>> >>>>>>> overloaded in Airflow so I don't want to reuse it. Maybe > >>> Reference? > >>> >>>>>>> > >>> >>>>>>> > >>> >>>>>>> need_by = DeadlineReference.DAGRUN_STARTED + > >>> timedelta(minutes=30? > >>> >>>>>>> > >>> >>>>>>> > >>> >>>>>>> Please throw some other suggestions or naming thoughts on the > >>> pile > >>> >>> and > >>> >>>>>>> maybe we can come up with something good. > >>> >>>>>>> > >>> >>>>>>> > >>> >>>>>>> > >>> >>>>>>> > >>> >>>>>>> - ferruzzi > >>> >>>>>>> > >>> >>>>>>> > >>> >>>>>>> > >>> >>>>>>> > >>> >>>>>> > >>> >>>> > >>> >>>> > >>> >>> > >>> >> > >>> > >>> > >>> --------------------------------------------------------------------- > >>> To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org > >>> For additional commands, e-mail: dev-h...@airflow.apache.org > >>> > >>> >
