I also don’t personally think it’s worth the pain (not to mention backcompat workaround) to rename DAG to Dag, so I’d be -0.5 on that.
-ash > On 18 Feb 2025, at 04:40, Daniel Imberman <daniel.imber...@gmail.com> wrote: > > I think my biggest concern is a marketing one and not a technical one. > > As has been mentioned on the thread the terms airflow and dag are kind of > synonymous and I certainly don’t want to give the impression that we are > breaking more than we are breaking. > > I wouldn’t die on this hill, but I’m slightly concerned. > > On Mon, Feb 17, 2025 at 6:00 PM Wei Lee <wei...@apache.org> wrote: > >> I’m not sure about adding ruff rules here 🤔 I think ruff rules are best >> suited for user-facing things but not the airflow code base itself. If what >> we mean is adding a rule to avoid users using "DAG" *after* we rename it, >> it's definitely a +1000. >> >> I just created GitHub issues for this removing "DAG" >> >> https://github.com/apache/airflow/issues/46842 >> https://github.com/apache/airflow/issues/46843 >> >> One thing I'm not sure about is whether we want to get rid of `from >> airflow import DAG` as well. 🤔 >> >> Best, >> Wei >> >> On 2025/02/17 19:50:57 Jarek Potiuk wrote: >>> Hard to say until it's looked at :) >>> >>> On Mon, Feb 17, 2025 at 8:45 PM Aritra Basu <aritrabasu1...@gmail.com> >>> wrote: >>> >>>> I can take it up, it's mostly just a doc update right? Or are we doing >> code >>>> files replacement too? >>>> -- >>>> Regards, >>>> Aritra Basu >>>> >>>> On Tue, 18 Feb 2025, 12:55 am Jarek Potiuk, <ja...@potiuk.com> wrote: >>>> >>>>> Sounds like another +1000 files big PR is coming :) ? Any volunteers >> to >>>>> make it ? It's fun. >>>>> >>>>> On Mon, Feb 17, 2025 at 6:52 PM Omkar P <droiddev5...@gmail.com> >> wrote: >>>>> >>>>>> +1 for ruff rules :) >>>>>> >>>>>> Also would be nice to introduce 'Dag' to replace 'DAG' in the >> Airflow >>>>>> docs, in line with the new UI changes and to make the renaming >>>>>> consistent across user-facing pages. >>>>>> >>>>>> Regards, >>>>>> Omkar >>>>>> >>>>>> On Mon, Feb 17, 2025 at 8:12 AM Jarek Potiuk <ja...@potiuk.com> >> wrote: >>>>>> >>>>>>> +1 :) . Maybe we could add a ruff rule for that :) >>>>>>> >>>>>>> On Mon, Feb 17, 2025 at 8:27 AM Wei Lee <wei...@apache.org> >> wrote: >>>>>>> >>>>>>>> It seems that our current conclusion is to use "dag" or "Dag" >>>> instead >>>>>> of >>>>>>>> "DAG" whenever possible. Should we replace all "DAG" in the >>>> codebase >>>>>> with >>>>>>>> "dag" or "Dag"? If it's too late for that (which it might be >> 🤔), >>>>>> should >>>>>>> we >>>>>>>> at least avoid introducing new "DAG" in the following PRs? >>>>>>>> >>>>>>>> Best, >>>>>>>> Wei >>>>>>>> >>>>>>>> On 2024/10/23 17:16:55 Brent Bovenzi wrote: >>>>>>>>> Here's a PR to use "dag" as a word in the new 3.0 UI: >>>>>>>>> https://github.com/apache/airflow/pull/43325 >>>>>>>>> Let me know if that's the direction we want to go. >>>>>>>>> >>>>>>>>> On Wed, Oct 23, 2024 at 11:06 AM Bishundeo, Rajeshwar >>>>>>>>> <rbish...@amazon.com.invalid> wrote: >>>>>>>>> >>>>>>>>>> I think Brent & Daniel summarized it best, "dag" is >> synonymous >>>>> with >>>>>>>>>> workflows in Airflow through the way we talk and explain >> what >>>>>> Airflow >>>>>>>> is >>>>>>>>>> all about. Although folks would ask, I don’t ever use the >>>>>>> mathematical >>>>>>>>>> definition of DAG. >>>>>>>>>> It will be challenging and possibly confusing for many >> users >>>>> making >>>>>>>> such a >>>>>>>>>> change - I would rather direct that energy to appending the >>>>> Oxford >>>>>>>>>> definition of "dag" to include a reference to workflows in >>>>> Airflow. >>>>>>> __ >>>>>>>>>> >>>>>>>>>> -- Rajesh >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On 2024-10-23, 3:37 AM, "Jarek Potiuk" <ja...@potiuk.com >>>>> <mailto: >>>>>>>>>> ja...@potiuk.com>> wrote: >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> CAUTION: This email originated from outside of the >>>> organization. >>>>> Do >>>>>>> not >>>>>>>>>> click links or open attachments unless you can confirm the >>>> sender >>>>>> and >>>>>>>> know >>>>>>>>>> the content is safe. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> AVERTISSEMENT: Ce courrier électronique provient d’un >>>> expéditeur >>>>>>>> externe. >>>>>>>>>> Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe >> si >>>> vous >>>>>> ne >>>>>>>> pouvez >>>>>>>>>> pas confirmer l’identité de l’expéditeur et si vous n’êtes >> pas >>>>>>> certain >>>>>>>> que >>>>>>>>>> le contenu ne présente aucun risque. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> From Guido's post: >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> "Around this time the renaming seems to have been renamed". >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Naming is hard. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> J. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Wed, Oct 23, 2024 at 7:11 AM Omkar P < >>>> droiddev5...@gmail.com >>>>>>>> <mailto: >>>>>>>>>> droiddev5...@gmail.com>> wrote: >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> Yes, completely agree with above comments, dag is an >> Airflow >>>>> term >>>>>>> now >>>>>>>>>>> rather >>>>>>>>>>> than just a "directed acyclic graph". Believe it or not, >> I've >>>>>>> worked >>>>>>>> with >>>>>>>>>>> work >>>>>>>>>>> folks who've been using dags in Airflow for years now >> and are >>>>> pro >>>>>>>> devs >>>>>>>>>> but >>>>>>>>>>> have trouble remembering the full form of a DAG! For >> them, >>>> dag >>>>> = >>>>>>>> Airflow. >>>>>>>>>>> >>>>>>>>>>> workflow, pipeline, flow are surely better terms but >> will be >>>> a >>>>>>> major >>>>>>>>>>> behavior >>>>>>>>>>> change for the developer community, so we'll need a solid >>>> plan >>>>> on >>>>>>>> how to >>>>>>>>>>> introduce it, when we do. >>>>>>>>>>> >>>>>>>>>>> While we discuss this, I'd like to share about the Great >>>>> Renaming >>>>>>> in >>>>>>>> core >>>>>>>>>>> Python started by Guido back in 2009. We could probably >> get >>>>> some >>>>>>>>>> learnings >>>>>>>>>>> from there? Who knows! >>>>>>>>>>> >>>>>>>>>>> Guido's blog (2009): >>>>>>>>>>> >>>>>>>> >>>>>> >>>> >> https://python-history.blogspot.com/2009/03/great-or-grand-renaming.html >>>>>>>>>> < >>>>>>>> >>>>>> >>>> >> https://python-history.blogspot.com/2009/03/great-or-grand-renaming.html >>>>>>>> >>>>>>>>>>> Follow-up discussion (2024): >>>>>>>>>>> >>>>> https://discuss.python.org/t/finishing-the-great-renaming/54082 >>>>>> < >>>>>>>>>> >>>> https://discuss.python.org/t/finishing-the-great-renaming/54082> >>>>>>>>>>> >>>>>>>>>>> Cheers, >>>>>>>>>>> Omkar >>>>>>>>>>> >>>>>>>>>>> On Wed, Oct 23, 2024 at 7:10 AM Wei Lee < >> weilee...@gmail.com >>>>>>>> <mailto: >>>>>>>>>> weilee...@gmail.com>> wrote: >>>>>>>>>>> >>>>>>>>>>>> I think we should probably just accept it as an airflow >>>> term. >>>>>> At >>>>>>>> least, >>>>>>>>>>>> that’s how I understood it when I first used Airflow. I >>>> feel >>>>>>>> renaming >>>>>>>>>> it >>>>>>>>>>> at >>>>>>>>>>>> this stage would require considerable effort from >>>> maintainers >>>>>> and >>>>>>>>>>> existing >>>>>>>>>>>> users without providing equivalent benefits. >>>>>>>>>>>> >>>>>>>>>>>> Best, >>>>>>>>>>>> Wei >>>>>>>>>>>> >>>>>>>>>>>>> On Oct 23, 2024, at 8:01 AM, Kaxil Naik < >>>>> kaxiln...@gmail.com >>>>>>>>>> <mailto:kaxiln...@gmail.com>> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>> Same agreed with Brent & Daniel -- maybe we re-kindle >>>> this >>>>>>>> discussion >>>>>>>>>>> for >>>>>>>>>>>>> Airflow 4 :) -- but right now it will cause too much >>>>>> disruption >>>>>>>>>>>>> >>>>>>>>>>>>> On Tue, 22 Oct 2024 at 21:27, Constance Martineau >>>>>>>>>>>>> <consta...@astronomer.io.inva <mailto: >>>>>>>> consta...@astronomer.io.inva>lid> >>>>>>>>>> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> In my experience, when you ask those with Airflow >>>>> experience >>>>>>>> what a >>>>>>>>>>> dag >>>>>>>>>>>> is, >>>>>>>>>>>>>> they'll start talking about workflow attributes - >> stuff >>>>> like >>>>>>>> dags >>>>>>>>>>> being >>>>>>>>>>>> a >>>>>>>>>>>>>> series of steps or tasks with owners. The structure >>>>> doesn't >>>>>>>> come up. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Echo-ing others, at this point, my vote is to >> embrace >>>> the >>>>>> name >>>>>>>> and >>>>>>>>>>>>>> de-emphasize the mathematical structure aspect. >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 3:47 PM Vikram Koka >>>>>>>>>>>> <vik...@astronomer.io.inva <mailto: >>>> vik...@astronomer.io.inva >>>>>>>> lid> >>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> It's an interesting discussion and I remember >>>> struggling >>>>>> with >>>>>>>> this >>>>>>>>>>>> when I >>>>>>>>>>>>>>> started working with Airflow. >>>>>>>>>>>>>>> But, I also agree with the viewpoint of it being an >>>>>>> established >>>>>>>>>>> concept >>>>>>>>>>>>>> now >>>>>>>>>>>>>>> regardless of the origin. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I am personally leaning towards the perspective >> best >>>>>>> expressed >>>>>>>> by >>>>>>>>>>>> Daniel >>>>>>>>>>>>>>> Standish and Brent of using Dag as a word, rather >> than >>>>> the >>>>>>>> computer >>>>>>>>>>>>>> science >>>>>>>>>>>>>>> concept. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Best regards, >>>>>>>>>>>>>>> Vikram >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 9:46 AM Oliveira, Niko >>>>>>>>>>>>>> <oniko...@amazon.com.inva <mailto: >>>>> oniko...@amazon.com.inva >>>>>>>> lid >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> I agree with the general sentiment of: You're >> right >>>>> Ryan, >>>>>>> DAG >>>>>>>>>> isn't >>>>>>>>>>>>>> great >>>>>>>>>>>>>>>> and I'd rather workflow, but changing it will >> cause >>>> much >>>>>>> more >>>>>>>>>>> wreckage >>>>>>>>>>>>>>> than >>>>>>>>>>>>>>>> it solves. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Also agree with the idea to just move away from >>>> defining >>>>>>> DAG. >>>>>>>> I >>>>>>>>>>> think >>>>>>>>>>>>>>>> we've been naturally doing that as a community >> for a >>>>> while >>>>>>> now >>>>>>>>>>> anyway, >>>>>>>>>>>>>> so >>>>>>>>>>>>>>>> that feels like a natural step. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Cheers, >>>>>>>>>>>>>>>> Niko >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> ________________________________ >>>>>>>>>>>>>>>> From: Ash Berlin-Taylor <a...@apache.org <mailto: >>>>>>>> a...@apache.org>> >>>>>>>>>>>>>>>> Sent: Tuesday, October 22, 2024 9:06:39 AM >>>>>>>>>>>>>>>> To: dev@airflow.apache.org <mailto: >>>>> dev@airflow.apache.org >>>>>>> >>>>>>>>>>>>>>>> Subject: RE: [EXT] Airflow should deprecate the >> term >>>>> "DAG" >>>>>>>> for end >>>>>>>>>>>>>> users >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> CAUTION: This email originated from outside of the >>>>>>>> organization. >>>>>>>>>> Do >>>>>>>>>>>> not >>>>>>>>>>>>>>>> click links or open attachments unless you can >> confirm >>>>> the >>>>>>>> sender >>>>>>>>>>> and >>>>>>>>>>>>>>> know >>>>>>>>>>>>>>>> the content is safe. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> AVERTISSEMENT: Ce courrier électronique provient >> d’un >>>>>>>> expéditeur >>>>>>>>>>>>>> externe. >>>>>>>>>>>>>>>> Ne cliquez sur aucun lien et n’ouvrez aucune pièce >>>>> jointe >>>>>> si >>>>>>>> vous >>>>>>>>>> ne >>>>>>>>>>>>>>> pouvez >>>>>>>>>>>>>>>> pas confirmer l’identité de l’expéditeur et si >> vous >>>>> n’êtes >>>>>>> pas >>>>>>>>>>> certain >>>>>>>>>>>>>>> que >>>>>>>>>>>>>>>> le contenu ne présente aucun risque. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Best argument in favour of keeping “dags” as a >> term — >>>>>>> getting >>>>>>>> to >>>>>>>>>>>> re-use >>>>>>>>>>>>>>>> puns like https://i.imgflip.com/1xhtwh.jpg < >>>>>>>>>> https://i.imgflip.com/1xhtwh.jpg> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> In all seriousness: I don’t mind either way, both >>>> sides >>>>>> have >>>>>>>> good >>>>>>>>>>>>>> reasons >>>>>>>>>>>>>>>> presented. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> -a >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> On 22 Oct 2024, at 17:03, Daniel Standish >>>>>>>>>>>>>>>> <daniel.stand...@astronomer.io.inva <mailto: >>>>>>>>>> daniel.stand...@astronomer.io.inva>LID> wrote: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Yeah just say, when asked where the name comes >> from, >>>>>> "well, >>>>>>>> no >>>>>>>>>> one >>>>>>>>>>>>>>>> actually >>>>>>>>>>>>>>>>> knows but..." and then make something up. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 8:31 AM Jarek Potiuk < >>>>>>>> ja...@potiuk.com >>>>>>>>>> <mailto:ja...@potiuk.com>> >>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Just to clarify - "directed acyclic graph" is >> the >>>>>>>>>> tongue-twister, >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 5:29 PM Jarek Potiuk < >>>>>>>> ja...@potiuk.com >>>>>>>>>> <mailto:ja...@potiuk.com>> >>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> I like what both Daniel and Brent wrote. I >> would >>>> very >>>>>>> much >>>>>>>> want >>>>>>>>>>> to >>>>>>>>>>>>>> be >>>>>>>>>>>>>>>>>> able >>>>>>>>>>>>>>>>>>> to say just "dag" without explaining it >> further. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> For me every time I explain "DAG" at a talk >> it's a >>>>>>>>>>> tongue-twister, >>>>>>>>>>>>>>> and >>>>>>>>>>>>>>>> I >>>>>>>>>>>>>>>>>>> almost stutter on trying to recall how to >> pronounce >>>>> it >>>>>>>>>> properly. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> J. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 5:27 PM Brent Bovenzi >>>>>>>>>>>>>>>>>> <br...@astronomer.io.inva <mailto: >>>>>>> br...@astronomer.io.inva >>>>>>>>> lid> >>>>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> I remember we explored renaming "DAG" when >>>> starting >>>>> on >>>>>>>> AIP-38 >>>>>>>>>> to >>>>>>>>>>>>>>>>>> modernize >>>>>>>>>>>>>>>>>>>> the UI. Both "pipeline" or "workflow" are more >>>>>>>> descriptive of >>>>>>>>>>> what >>>>>>>>>>>>>>> one >>>>>>>>>>>>>>>>>> is >>>>>>>>>>>>>>>>>>>> actually doing while Directed Acyclic Graph >> is an >>>>>>>>>> implementation >>>>>>>>>>>>>>>> detail. >>>>>>>>>>>>>>>>>>>> But I agree with Daniel Standish, at this >> point >>>>> "DAG" >>>>>>> has >>>>>>>>>> become >>>>>>>>>>>>>>> "dag" >>>>>>>>>>>>>>>>>> , a >>>>>>>>>>>>>>>>>>>> word in its own right. >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> Examples for "dag" are abound in community >>>>> discussion, >>>>>>>> Airflow >>>>>>>>>>>>>>> Summit >>>>>>>>>>>>>>>>>>>> talks, documentation and even in the UI. Let's >>>>> embrace >>>>>>>> "dag". >>>>>>>>>> A >>>>>>>>>>>>>> user >>>>>>>>>>>>>>>>>> just >>>>>>>>>>>>>>>>>>>> needs to learn one new word vs the technical >>>> concept >>>>>>>> behind >>>>>>>>>> that >>>>>>>>>>>>>>>> word. I >>>>>>>>>>>>>>>>>>>> think that is much less effort than >> refactoring so >>>>>> much >>>>>>>> code, >>>>>>>>>>>>>>>>>>>> documentation, blog posts, stack overflow >>>> questions, >>>>>>> etc. >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 10:51 AM Daniel >> Standish >>>>>>>>>>>>>>>>>>>> <daniel.stand...@astronomer.io.inva <mailto: >>>>>>>>>> daniel.stand...@astronomer.io.inva>lid> wrote: >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> I am skeptical. Seems like introducing a lot >> of >>>>> pain >>>>>>> for >>>>>>>>>>>>>>>> questionable >>>>>>>>>>>>>>>>>>>>> benefit. But, I am def sympathetic to the >> idea. I >>>>>> agree >>>>>>>> the >>>>>>>>>>>>>>>>>>>> association >>>>>>>>>>>>>>>>>>>>> with "directed acyclic graph" is not helpful. >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> And along those lines, I offer here some less >>>>>> invasive >>>>>>>>>>>>>> mitigations. >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> One thing we can do no matter what is to >>>>> de-emphasize >>>>>>> the >>>>>>>>>> math >>>>>>>>>>>>>> nerd >>>>>>>>>>>>>>>>>>>> origins >>>>>>>>>>>>>>>>>>>>> of the name. That is to say, in docs / >> website / >>>>> etc, >>>>>>>> *never >>>>>>>>>>>>>>> define* >>>>>>>>>>>>>>>>>>>>> airflow's "dag" concept as a directed acyclic >>>>> graph. >>>>>>>> Always >>>>>>>>>>>>>> define >>>>>>>>>>>>>>>> it >>>>>>>>>>>>>>>>>>>> as a >>>>>>>>>>>>>>>>>>>>> pipeline, collection of tasks, workflow etc. >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> The "directed acyclic graph" part of it is >> like a >>>>>>>> historical >>>>>>>>>>>>>>>> footnote, >>>>>>>>>>>>>>>>>>>> and >>>>>>>>>>>>>>>>>>>>> we could make one mention of it somewhere >> hidden. >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> We could also start using lowercase in the >> docs >>>> in >>>>>>>> general >>>>>>>>>> e.g. >>>>>>>>>>>>>>>>>> writing >>>>>>>>>>>>>>>>>>>>> "dag" / "dags" instead of writing "DAG" / >> "DAGs" >>>>> etc. >>>>>>> The >>>>>>>>>>> upper >>>>>>>>>>>>>>> case >>>>>>>>>>>>>>>>>>>> part >>>>>>>>>>>>>>>>>>>>> of it makes it look like an acronym; but >> "dag" in >>>>>>>> airlfow is >>>>>>>>>>> just >>>>>>>>>>>>>>> an >>>>>>>>>>>>>>>>>>>>> airflow concept and the association with >> "DAGs" >>>> is >>>>>> not >>>>>>>> really >>>>>>>>>>>>>>>>>> unhelpful. >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> In other words embrace that "dag" in airflow >> is >>>> its >>>>>> own >>>>>>>>>> thing, >>>>>>>>>>> is >>>>>>>>>>>>>>>>>>>>> *not* strictly >>>>>>>>>>>>>>>>>>>>> speaking a directed acyclic graph (which >> nobody >>>>> knows >>>>>>>> about >>>>>>>>>>>>>>> anyway), >>>>>>>>>>>>>>>>>> and >>>>>>>>>>>>>>>>>>>>> tell them what it is in simple terms that >> normal >>>>>> people >>>>>>>>>>>>>> understand. >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 7:27 AM Jarek Potiuk >> < >>>>>>>>>> ja...@potiuk.com <mailto:ja...@potiuk.com> >>>>>>>>>>>> >>>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> DAG is so embedded into what we do that it >> will >>>> be >>>>>>>> extremely >>>>>>>>>>>>>>>>>>>> difficult to >>>>>>>>>>>>>>>>>>>>>> get rid of it completely. Also I think it >> will >>>>> make >>>>>> a >>>>>>>> lot of >>>>>>>>>>>>>>>>>> "google" >>>>>>>>>>>>>>>>>>>>>> searches and "stack overflow" searches not >>>> finding >>>>>> the >>>>>>>> right >>>>>>>>>>>>>>>>>> answers. >>>>>>>>>>>>>>>>>>>>> This >>>>>>>>>>>>>>>>>>>>>> is one of the strengths of Airflow - >> besides the >>>>>>>> community >>>>>>>>>> and >>>>>>>>>>>>>>> ideas >>>>>>>>>>>>>>>>>>>> that >>>>>>>>>>>>>>>>>>>>>> Bernd mentioned - is the vast number of >>>> examples, >>>>>>>> problems >>>>>>>>>> and >>>>>>>>>>>>>>>>>>>> solutions >>>>>>>>>>>>>>>>>>>>>> you can so easily find (and we have to >> remember >>>>> that >>>>>>>> all the >>>>>>>>>>> AI >>>>>>>>>>>>>>>>>>>> trained >>>>>>>>>>>>>>>>>>>>> on >>>>>>>>>>>>>>>>>>>>>> past data will be also rather poorly >> matching >>>>>> queries >>>>>>> of >>>>>>>>>>> people. >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> I am not too attached to DAG. I could easily >>>>> switch. >>>>>>>> And if >>>>>>>>>> we >>>>>>>>>>>>>> do >>>>>>>>>>>>>>> - >>>>>>>>>>>>>>>>>> I >>>>>>>>>>>>>>>>>>>>>> would be for using workflow or pipeline >> instead >>>> of >>>>>>>> `dag` if >>>>>>>>>>> not >>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>> above >>>>>>>>>>>>>>>>>>>>>> reason, but I think I am here with Igor >> that it >>>>>> might >>>>>>>> cause >>>>>>>>>>> more >>>>>>>>>>>>>>>>>>>> problems >>>>>>>>>>>>>>>>>>>>>> than it solves. >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> But I am not 100% against - if others will >> think >>>>>> it's >>>>>>> a >>>>>>>> good >>>>>>>>>>>>>>> idea, I >>>>>>>>>>>>>>>>>>>> am >>>>>>>>>>>>>>>>>>>>> ok >>>>>>>>>>>>>>>>>>>>>> with it. >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> J, >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 3:12 PM Abhishek >> Bhakat >>>>>>>>>>>>>>>>>>>>>> <abhishek.bha...@astronomer.io.inva >> <mailto: >>>>>>>>>> abhishek.bha...@astronomer.io.inva>lid> wrote: >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> Agreed that the word DAG makes very less >> sense >>>> to >>>>>>>> someone >>>>>>>>>> new >>>>>>>>>>>>>> to >>>>>>>>>>>>>>>>>>>>> workflow >>>>>>>>>>>>>>>>>>>>>>> orchestration. But it does also show the >> nature >>>>> of >>>>>>>> being >>>>>>>>>>>>>> acyclic. >>>>>>>>>>>>>>>>>>>> Sure, >>>>>>>>>>>>>>>>>>>>>> as >>>>>>>>>>>>>>>>>>>>>>> Bas mentioned, there are ways to >> workaround it. >>>>>>> Still, >>>>>>>> in >>>>>>>>>> my >>>>>>>>>>>>>>>>>>>> opinion, >>>>>>>>>>>>>>>>>>>>>> there >>>>>>>>>>>>>>>>>>>>>>> is generally no need for cyclic behavior in >>>>>> workflow >>>>>>>>>>>>>>>>>> orchestration. >>>>>>>>>>>>>>>>>>>>> Most >>>>>>>>>>>>>>>>>>>>>>> (*if >>>>>>>>>>>>>>>>>>>>>>> not all*) cases can be in some way can be >>>> covered >>>>>>>> using an >>>>>>>>>>>>>>> acyclic >>>>>>>>>>>>>>>>>>>>> manner >>>>>>>>>>>>>>>>>>>>>>> with multiple runs. Hence, the >> idempotency. So >>>> I >>>>>>> would >>>>>>>> want >>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>> "acyclic" >>>>>>>>>>>>>>>>>>>>>>> word to stick. >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> Regards, >>>>>>>>>>>>>>>>>>>>>>> Avi >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 12:41 PM < >>>>>>>>>> bernd.stroe...@kosakya.de <mailto: >> bernd.stroe...@kosakya.de>> >>>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> Brilliant, I am on the way to become an >>>> Airflow >>>>>> Fan; >>>>>>>> so >>>>>>>>>> many >>>>>>>>>>>>>> new >>>>>>>>>>>>>>>>>>>>> ideas. >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> The Term DAG is misleading; it should be >>>>> replaced >>>>>> by >>>>>>>> the >>>>>>>>>>> more >>>>>>>>>>>>>>>>>>>> general >>>>>>>>>>>>>>>>>>>>>>> Term >>>>>>>>>>>>>>>>>>>>>>>> Airflow (Workflow) Graph (AFG) or Airflow >>>>> (Petri) >>>>>>> Net >>>>>>>>>> (AFN) >>>>>>>>>>>>>>>>>> (maybe >>>>>>>>>>>>>>>>>>>>>>> without >>>>>>>>>>>>>>>>>>>>>>>> a direction); >>>>>>>>>>>>>>>>>>>>>>>> and ... these Graphs should be stored in a >>>> Graph >>>>>>>> Database. >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> Every Node or Sup-Graph of an Airflow >> Graph >>>>> (AFG) >>>>>>>> might be >>>>>>>>>>>>>>>>>>>> assigned >>>>>>>>>>>>>>>>>>>>> to >>>>>>>>>>>>>>>>>>>>>> an >>>>>>>>>>>>>>>>>>>>>>>> executable (Python-, Rust-, ... ) member >> of a >>>>>>> library. >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> A running Graph might have a different >>>> structure >>>>>>> than >>>>>>>> a >>>>>>>>>>>>>>>>>>>> configuration >>>>>>>>>>>>>>>>>>>>>>>> Graph. >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> Forget that if you think it's bullshit. >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> Best Regards >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> Bernd Ströhle >>>>>>>>>>>>>>>>>>>>>>>> M: +49 171 5357916 >>>>>>>>>>>>>>>>>>>>>>>> E: bernd.stroe...@gmail.com <mailto: >>>>>>>>>> bernd.stroe...@gmail.com> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> -----Original Message----- >>>>>>>>>>>>>>>>>>>>>>>> From: Igor Kholopov >> <ikholo...@google.com.inva >>>>>>>> <mailto: >>>>>>>>>> ikholo...@google.com.inva>LID> >>>>>>>>>>>>>>>>>>>>>>>> Sent: Tuesday, October 22, 2024 12:02 PM >>>>>>>>>>>>>>>>>>>>>>>> To: dev@airflow.apache.org <mailto: >>>>>>>> dev@airflow.apache.org >>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> Subject: Re: Airflow should deprecate the >> term >>>>>> "DAG" >>>>>>>> for >>>>>>>>>> end >>>>>>>>>>>>>>>>>> users >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> Even though the term "DAG" is clearly >>>>> suboptimal, >>>>>> it >>>>>>>> is >>>>>>>>>> part >>>>>>>>>>>>>> of >>>>>>>>>>>>>>>>>>>>> Airflow >>>>>>>>>>>>>>>>>>>>>>>> DAG definition interface at so many >> levels, >>>> that >>>>>> any >>>>>>>>>> attempt >>>>>>>>>>>>>> to >>>>>>>>>>>>>>>>>>>>> change >>>>>>>>>>>>>>>>>>>>>> it >>>>>>>>>>>>>>>>>>>>>>>> will only introduce more chaos, not >> reduce it. >>>>> The >>>>>>>> only >>>>>>>>>>> thing >>>>>>>>>>>>>>>>>>>> that is >>>>>>>>>>>>>>>>>>>>>>> worse >>>>>>>>>>>>>>>>>>>>>>>> than a poorly chosen name in the code is >> when >>>>>> there >>>>>>>> are >>>>>>>>>> two >>>>>>>>>>>>>> ways >>>>>>>>>>>>>>>>>>>> to >>>>>>>>>>>>>>>>>>>>>>> define >>>>>>>>>>>>>>>>>>>>>>>> the same thing. Countless articles and >>>> tutorials >>>>>>> will >>>>>>>>>>> suddenly >>>>>>>>>>>>>>>>>>>> become >>>>>>>>>>>>>>>>>>>>>>>> confusing as they all refer to workflows >> as >>>>>> "DAG"s. >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> We are already at risk of scaring the >> users >>>> away >>>>>>> with >>>>>>>> a >>>>>>>>>>> number >>>>>>>>>>>>>>>>>> of >>>>>>>>>>>>>>>>>>>>>>> breaking >>>>>>>>>>>>>>>>>>>>>>>> changes in Airflow 3, promising even more >>>>> breaking >>>>>>>> changes >>>>>>>>>>> for >>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>> most >>>>>>>>>>>>>>>>>>>>>>>> basic things is not something that people >> are >>>>>>> looking >>>>>>>> for. >>>>>>>>>>>>>>>>>>>> Attempting >>>>>>>>>>>>>>>>>>>>>> to >>>>>>>>>>>>>>>>>>>>>>>> change the fundamental terms will be >>>> interpreted >>>>>> as >>>>>>> an >>>>>>>>>> even >>>>>>>>>>>>>>>>>>>> stronger >>>>>>>>>>>>>>>>>>>>>>> signal >>>>>>>>>>>>>>>>>>>>>>>> of project immaturity. >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> Given that, I oppose the idea of changing >> the >>>>> term >>>>>>> in >>>>>>>> the >>>>>>>>>>> long >>>>>>>>>>>>>>>>>>>> run. I >>>>>>>>>>>>>>>>>>>>>>> even >>>>>>>>>>>>>>>>>>>>>>>> stricter oppose the idea of deprecating >> it in >>>>> the >>>>>>> DAG >>>>>>>>>>>>>> definition >>>>>>>>>>>>>>>>>>>>>>> interface. >>>>>>>>>>>>>>>>>>>>>>>> We better put our time and efforts in >> other >>>>> places >>>>>>> in >>>>>>>>>>> Airflow, >>>>>>>>>>>>>>>>>> of >>>>>>>>>>>>>>>>>>>>> which >>>>>>>>>>>>>>>>>>>>>>>> there are plenty. >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> Kind regards, >>>>>>>>>>>>>>>>>>>>>>>> Igor >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 10:36 AM Bas >> Harenslak >>>>>>>>>>>>>>>>>>>>>> <b...@astronomer.io.inva <mailto: >>>>>>> b...@astronomer.io.inva >>>>>>>>> lid >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> Couple of thoughts: >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> 1. The boundaries/properties of “DAG” >> have >>>>>> already >>>>>>>> faded >>>>>>>>>>> over >>>>>>>>>>>>>>>>>>>> time, >>>>>>>>>>>>>>>>>>>>>>>>> for example there are now several ways to >>>>> create >>>>>>>> cyclic >>>>>>>>>>>>>>>>>> graphs, >>>>>>>>>>>>>>>>>>>>> e.g. >>>>>>>>>>>>>>>>>>>>>>>>> using the @continuous schedule. I imagine >>>> these >>>>>>>>>> properties >>>>>>>>>>>>>>>>>>>>> vanishing >>>>>>>>>>>>>>>>>>>>>>>>> even more in the future, so from that >>>>>> perspective I >>>>>>>>>> support >>>>>>>>>>>>>>>>>>>>> changing >>>>>>>>>>>>>>>>>>>>>>>>> “DAG" to a more generic name. >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> 2. How other orchestration frameworks do >>>>> naming: >>>>>>>>>>>>>>>>>>>>>>>>> Dagster: pipeline >>>>>>>>>>>>>>>>>>>>>>>>> Prefect: flow >>>>>>>>>>>>>>>>>>>>>>>>> Flyte: workflow >>>>>>>>>>>>>>>>>>>>>>>>> Temporal: workflow >>>>>>>>>>>>>>>>>>>>>>>>> Kestra: flow >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> I think “workflow” is the most fitting >> name. >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> 3. Given the large impact of this >> change, I >>>>>> suggest >>>>>>>>>>> defining >>>>>>>>>>>>>> a >>>>>>>>>>>>>>>>>>>>> clear >>>>>>>>>>>>>>>>>>>>>>>>> path forward. Would we first introduce >> the >>>>>>>> deprecation in >>>>>>>>>>>>>>>>>>>> Airflow >>>>>>>>>>>>>>>>>>>>> 3, >>>>>>>>>>>>>>>>>>>>>>>>> and remove “DAG” in Airflow 4? >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> Bas >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> On 22 Oct 2024, at 09:22, Neil < >>>>>>> neil4r...@gmail.com >>>>>>>>>> <mailto:neil4r...@gmail.com>> >>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> I don't see a problem with the term DAG, >>>>>>> especially >>>>>>>> when >>>>>>>>>>>>>>>>>> most >>>>>>>>>>>>>>>>>>>>> other >>>>>>>>>>>>>>>>>>>>>>>>>> platforms embrace the term >> wholeheartedly. >>>>>>>>>>>>>>>>>>>>>>>>>> I don't see anything intimidating or >>>> confusing >>>>>>>> about it >>>>>>>>>> at >>>>>>>>>>>>>>>>>>>> all, >>>>>>>>>>>>>>>>>>>>>>>>>> changing the term though would be fairly >>>>>> confusing >>>>>>>> to >>>>>>>>>> most >>>>>>>>>>>>>>>>>>>> users >>>>>>>>>>>>>>>>>>>>>> who >>>>>>>>>>>>>>>>>>>>>>>>>> have been >>>>>>>>>>>>>>>>>>>>>>>>> using >>>>>>>>>>>>>>>>>>>>>>>>>> the term for years. >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 1:18 AM Tzu-ping >>>> Chung >>>>>>>>>>>>>>>>>>>>>>>>>> <t...@astronomer.io.inva <mailto: >>>>>>>> t...@astronomer.io.inva >>>>>>>>>>> lid >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> I totally agree with doing away with >> the >>>> term >>>>>>> DAG. >>>>>>>> The >>>>>>>>>>> only >>>>>>>>>>>>>>>>>>>>>> problem >>>>>>>>>>>>>>>>>>>>>>>>> (aside >>>>>>>>>>>>>>>>>>>>>>>>>>> from actually telling people—including >>>>>> myself—to >>>>>>>> stop >>>>>>>>>>> using >>>>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>>>>> term) >>>>>>>>>>>>>>>>>>>>>>>>> is to >>>>>>>>>>>>>>>>>>>>>>>>>>> come up with a reasonable alternative. >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> I can’t recall who, but someone >> mentioned >>>>>>>> “workflow” is >>>>>>>>>>> not >>>>>>>>>>>>>>>>>>>> very >>>>>>>>>>>>>>>>>>>>>>>>> accurate >>>>>>>>>>>>>>>>>>>>>>>>>>> for Airflow. The term “definition” was >>>>>> proposed, >>>>>>>> but >>>>>>>>>>> it’s a >>>>>>>>>>>>>>>>>>>> bit >>>>>>>>>>>>>>>>>>>>>>>>>>> broad; I tried to use it in a few >> places >>>> and >>>>>> kept >>>>>>>>>> finding >>>>>>>>>>>>>>>>>>>> myself >>>>>>>>>>>>>>>>>>>>>>>>>>> doubting “what definition?” and >> wanting to >>>>>>> clarify >>>>>>>> “DAG >>>>>>>>>>>>>>>>>>>>>> definition” >>>>>>>>>>>>>>>>>>>>>>>>>>> (defeating the purpose). >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> TP >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> On 22 Oct 2024, at 13:07, Jens >> Scheffler >>>>>>>>>>>>>>>>>>>>>>>>>>>> <j_scheff...@gmx.de.inva <mailto: >>>>>>>>>> j_scheff...@gmx.de.inva>LID> >>>>>>>>>>>>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Ryan, >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for posting. I share the >> exactly >>>> same >>>>>>>>>>> observation, >>>>>>>>>>>>>>>>>>>> had a >>>>>>>>>>>>>>>>>>>>>>>>>>>> short >>>>>>>>>>>>>>>>>>>>>>>>>>> laight because the DAG question is >> always >>>> an >>>>>>>>>> introduction >>>>>>>>>>>>>>>>>> if >>>>>>>>>>>>>>>>>>>>>>>>>>> someone >>>>>>>>>>>>>>>>>>>>>>>>> joins >>>>>>>>>>>>>>>>>>>>>>>>>>> the party. I think a global renaming >> makes >>>>>> sense. >>>>>>>>>>>>>>>>>> Especially >>>>>>>>>>>>>>>>>>>>> when >>>>>>>>>>>>>>>>>>>>>>>>>>> we >>>>>>>>>>>>>>>>>>>>>>>>> also >>>>>>>>>>>>>>>>>>>>>>>>>>> rename Dataset to Asset this is also a >>>>>> reasonable >>>>>>>> step. >>>>>>>>>>>>>>>>>>>> Concepts >>>>>>>>>>>>>>>>>>>>>>>>>>> still >>>>>>>>>>>>>>>>>>>>>>>>> can >>>>>>>>>>>>>>>>>>>>>>>>>>> stay the same. >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> So I hope I don‘t need to join hiding >>>> below >>>>>> the >>>>>>>> desk >>>>>>>>>>> with >>>>>>>>>>>>>>>>>>>> you >>>>>>>>>>>>>>>>>>>>> and >>>>>>>>>>>>>>>>>>>>>>>>>>>> +1 >>>>>>>>>>>>>>>>>>>>>>>>> for >>>>>>>>>>>>>>>>>>>>>>>>>>> raising the discussion. >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Technically we can still think if we >> keep >>>>>>> details >>>>>>>> of >>>>>>>>>>>>>>>>>> python >>>>>>>>>>>>>>>>>>>>> names >>>>>>>>>>>>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>>>>> same because the execution is still a >> DAG… >>>>> but >>>>>>> user >>>>>>>>>>> facing >>>>>>>>>>>>>>>>>> it >>>>>>>>>>>>>>>>>>>>> is a >>>>>>>>>>>>>>>>>>>>>>>>> workflow. >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Jens >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Sent from my Smartphone >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 21. Oct 2024, at 23:56, Ryan >> Hatter < >>>>>>>>>>>>>>>>>>>>>> ryan.hat...@astronomer.io <mailto: >>>>>>>> ryan.hat...@astronomer.io >>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> .invalid> >>>>>>>>>>>>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Everyone please sheathe your >> swords... at >>>>>> least >>>>>>>> for >>>>>>>>>>> now. >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> The term "DAG" has very little >> meaning to >>>>>>> Airflow >>>>>>>>>>> users. >>>>>>>>>>>>>>>>>>>>> Indeed, >>>>>>>>>>>>>>>>>>>>>>>>>>>>> it >>>>>>>>>>>>>>>>>>>>>>>>> has >>>>>>>>>>>>>>>>>>>>>>>>>>>>> little meaning outside of some >>>>> mathematicians >>>>>>> and >>>>>>>>>>>>>>>>>> software >>>>>>>>>>>>>>>>>>>>>>>>>>>>> engineers >>>>>>>>>>>>>>>>>>>>>>>>> for >>>>>>>>>>>>>>>>>>>>>>>>>>>>> whom the properties of a DAG actually >>>>> matter. >>>>>>> For >>>>>>>>>>> someone >>>>>>>>>>>>>>>>>>>> new >>>>>>>>>>>>>>>>>>>>> to >>>>>>>>>>>>>>>>>>>>>>>>>>>>> data engineering or workflow >>>> orchestration, >>>>>> one >>>>>>>> of >>>>>>>>>> the >>>>>>>>>>>>>>>>>>>> first >>>>>>>>>>>>>>>>>>>>>>>>>>>>> questions they >>>>>>>>>>>>>>>>>>>>>>>>>>> will >>>>>>>>>>>>>>>>>>>>>>>>>>>>> likely have is, "what on earth is a >> DAG?" >>>>> The >>>>>>>> answer >>>>>>>>>> is >>>>>>>>>>>>>>>>>>>> almost >>>>>>>>>>>>>>>>>>>>>>>>>>>>> always, "It's a directed acyclic >> graph. >>>> You >>>>>>> don't >>>>>>>>>> need >>>>>>>>>>> to >>>>>>>>>>>>>>>>>>>>> worry >>>>>>>>>>>>>>>>>>>>>>>>>>>>> about what >>>>>>>>>>>>>>>>>>>>>>>>> that >>>>>>>>>>>>>>>>>>>>>>>>>>>>> means; it's just a term for your >>>> workflow." >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> The term "DAG" is problematic for at >>>> least >>>>> a >>>>>>>> couple >>>>>>>>>>>>>>>>>>>> important >>>>>>>>>>>>>>>>>>>>>>>> reasons: >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> *Complexity for New Users*: As >> mentioned >>>>>> above, >>>>>>>> "DAG" >>>>>>>>>>> is >>>>>>>>>>>>>>>>>>>>>>>>>>>>> unnecessarily intimidating and >> confusing. >>>>> We >>>>>>> want >>>>>>>>>>> Airflow >>>>>>>>>>>>>>>>>>>> to >>>>>>>>>>>>>>>>>>>>> be >>>>>>>>>>>>>>>>>>>>>>>>>>>>> approachable, and >>>>>>>>>>>>>>>>>>>>>>>>>>> using >>>>>>>>>>>>>>>>>>>>>>>>>>>>> technical jargon like "DAG" right >> off the >>>>> bat >>>>>>>> creates >>>>>>>>>>> an >>>>>>>>>>>>>>>>>>>>> initial >>>>>>>>>>>>>>>>>>>>>>>>>>> barrier to >>>>>>>>>>>>>>>>>>>>>>>>>>>>> understanding. >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> *Disconnect Between DAG and Workflow >>>>>> Concepts*: >>>>>>>> The >>>>>>>>>> DAG >>>>>>>>>>>>>>>>>> is >>>>>>>>>>>>>>>>>>>>> just >>>>>>>>>>>>>>>>>>>>>>>>>>>>> one component of an Airflow >> workflow. The >>>>>>>> workflow >>>>>>>>>>>>>>>>>> includes >>>>>>>>>>>>>>>>>>>>> its >>>>>>>>>>>>>>>>>>>>>>>>>>>>> schedule, retries, timeouts, a dozen >>>> other >>>>>>>>>> parameters, >>>>>>>>>>>>>>>>>> and >>>>>>>>>>>>>>>>>>>>> other >>>>>>>>>>>>>>>>>>>>>>>>>>>>> metadata that >>>>>>>>>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>>>>>>> DAG component doesn’t account for. >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Consider the following from the >> Airflow >>>>>>> homepage >>>>>>>>>>>>>>>>>>>>>>>>>>>>> <https://airflow.apache.org/> < >>>>>>>>>> https://airflow.apache.org/>>. >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Apache Airflow® is a platform >> created by >>>>> the >>>>>>>>>> community >>>>>>>>>>> to >>>>>>>>>>>>>>>>>>>>>>>>>>> programmatically >>>>>>>>>>>>>>>>>>>>>>>>>>>>> author, schedule and monitor >> workflows. >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Then, if we look at the "What is >>>> Airflow?" >>>>>> docs >>>>>>>> page >>>>>>>>>>>>>>>>>>>>>>>>>>>>> < >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>> >> https://airflow.apache.org/docs/apache-airflow/stable/index.html < >>>>>>>>>> >>>> https://airflow.apache.org/docs/apache-airflow/stable/index.html >>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> , >>>>>>>>>>>>>>>>>>>>>>>>> we >>>>>>>>>>>>>>>>>>>>>>>>>>> can >>>>>>>>>>>>>>>>>>>>>>>>>>>>> see that the docs explain what >> Airflow is >>>>>>> without >>>>>>>>>> using >>>>>>>>>>>>>>>>>>>> "DAG." >>>>>>>>>>>>>>>>>>>>>>>>>>>>> It's >>>>>>>>>>>>>>>>>>>>>>>>>>> only in >>>>>>>>>>>>>>>>>>>>>>>>>>>>> the *workflow* Python code that the >> term >>>> is >>>>>>>>>> introduced >>>>>>>>>>>>>>>>>> out >>>>>>>>>>>>>>>>>>>> of >>>>>>>>>>>>>>>>>>>>>>>>>>>>> nowhere >>>>>>>>>>>>>>>>>>>>>>>>>>> as a >>>>>>>>>>>>>>>>>>>>>>>>>>>>> comment that awkwardly tries to >> explain >>>> it: >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> # A DAG represents a workflow, a >>>> collection >>>>>> of >>>>>>>> tasks >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> It makes sense to not refer to DAGs >> in >>>>> these >>>>>>>>>>>>>>>>>> introductions >>>>>>>>>>>>>>>>>>>> to >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Airflow, because *Airflow doesn't >>>>> orchestrate >>>>>>>> DAGs; >>>>>>>>>> it >>>>>>>>>>>>>>>>>>>>>>> orchestrates >>>>>>>>>>>>>>>>>>>>>>>> workflows*. >>>>>>>>>>>>>>>>>>>>>>>>>>> The >>>>>>>>>>>>>>>>>>>>>>>>>>>>> DAG is the model that, for reasons >>>>> irrelevant >>>>>>> to >>>>>>>>>> almost >>>>>>>>>>>>>>>>>>>> every >>>>>>>>>>>>>>>>>>>>>>>>>>>>> user, workflows must adhere to. >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> So, I propose at least adding an >> alias >>>> for >>>>>> the >>>>>>>> term >>>>>>>>>>> "DAG" >>>>>>>>>>>>>>>>>>>> and >>>>>>>>>>>>>>>>>>>>>>>>>>>>> updating documentation to replace >> "DAG" >>>>> with >>>>>>>>>>> "workflow". >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> For example, instead of... >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> @dag( >>>>>>>>>>>>>>>>>>>>>>>>>>>>> schedule="@daily", >>>>>>>>>>>>>>>>>>>>>>>>>>>>> ... >>>>>>>>>>>>>>>>>>>>>>>>>>>>> dagrun_timeout=timedelta(hours=1) >>>>>>>>>>>>>>>>>>>>>>>>>>>>> ) >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Users could do... >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> @workflow( >>>>>>>>>>>>>>>>>>>>>>>>>>>>> schedule="@daily", >>>>>>>>>>>>>>>>>>>>>>>>>>>>> ... >>>>>>>>>>>>>>>>>>>>>>>>>>>>> run_timeout=timedelta(hours=1) >>>>>>>>>>>>>>>>>>>>>>>>>>>>> ) >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> And with that... I will start running >>>> away. >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>> >> ------------------------------------------------------------------ >>>>>>>>>>>>>>>>>>>>>>>>>>>> --- To unsubscribe, e-mail: >>>>>>>>>>>>>>>>>>>> dev-unsubscr...@airflow.apache.org <mailto: >>>>>>>>>> dev-unsubscr...@airflow.apache.org> >>>>>>>>>>>>>>>>>>>>>>>>>>>> For additional commands, e-mail: >>>>>>>>>>>>>>>>>>>> dev-h...@airflow.apache.org <mailto: >>>>>>>>>> dev-h...@airflow.apache.org> >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org For additional commands, e-mail: dev-h...@airflow.apache.org
