+1 for ruff rules :) Also would be nice to introduce 'Dag' to replace 'DAG' in the Airflow docs, in line with the new UI changes and to make the renaming consistent across user-facing pages.
Regards, Omkar On Mon, Feb 17, 2025 at 8:12 AM Jarek Potiuk <ja...@potiuk.com> wrote: > +1 :) . Maybe we could add a ruff rule for that :) > > On Mon, Feb 17, 2025 at 8:27 AM Wei Lee <wei...@apache.org> wrote: > > > It seems that our current conclusion is to use "dag" or "Dag" instead of > > "DAG" whenever possible. Should we replace all "DAG" in the codebase with > > "dag" or "Dag"? If it's too late for that (which it might be 🤔), should > we > > at least avoid introducing new "DAG" in the following PRs? > > > > Best, > > Wei > > > > On 2024/10/23 17:16:55 Brent Bovenzi wrote: > > > Here's a PR to use "dag" as a word in the new 3.0 UI: > > > https://github.com/apache/airflow/pull/43325 > > > Let me know if that's the direction we want to go. > > > > > > On Wed, Oct 23, 2024 at 11:06 AM Bishundeo, Rajeshwar > > > <rbish...@amazon.com.invalid> wrote: > > > > > > > I think Brent & Daniel summarized it best, "dag" is synonymous with > > > > workflows in Airflow through the way we talk and explain what Airflow > > is > > > > all about. Although folks would ask, I don’t ever use the > mathematical > > > > definition of DAG. > > > > It will be challenging and possibly confusing for many users making > > such a > > > > change - I would rather direct that energy to appending the Oxford > > > > definition of "dag" to include a reference to workflows in Airflow. > __ > > > > > > > > -- Rajesh > > > > > > > > > > > > > > > > > > > > > > > > > > > > On 2024-10-23, 3:37 AM, "Jarek Potiuk" <ja...@potiuk.com <mailto: > > > > ja...@potiuk.com>> wrote: > > > > > > > > > > > > CAUTION: This email originated from outside of the organization. Do > not > > > > click links or open attachments unless you can confirm the sender and > > know > > > > the content is safe. > > > > > > > > > > > > > > > > > > > > > > > > > > > > AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur > > externe. > > > > Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne > > pouvez > > > > pas confirmer l’identité de l’expéditeur et si vous n’êtes pas > certain > > que > > > > le contenu ne présente aucun risque. > > > > > > > > > > > > > > > > > > > > > > > > > > > > From Guido's post: > > > > > > > > > > > > "Around this time the renaming seems to have been renamed". > > > > > > > > > > > > Naming is hard. > > > > > > > > > > > > J. > > > > > > > > > > > > > > > > > > > > On Wed, Oct 23, 2024 at 7:11 AM Omkar P <droiddev5...@gmail.com > > <mailto: > > > > droiddev5...@gmail.com>> wrote: > > > > > > > > > > > > > Yes, completely agree with above comments, dag is an Airflow term > now > > > > > rather > > > > > than just a "directed acyclic graph". Believe it or not, I've > worked > > with > > > > > work > > > > > folks who've been using dags in Airflow for years now and are pro > > devs > > > > but > > > > > have trouble remembering the full form of a DAG! For them, dag = > > Airflow. > > > > > > > > > > workflow, pipeline, flow are surely better terms but will be a > major > > > > > behavior > > > > > change for the developer community, so we'll need a solid plan on > > how to > > > > > introduce it, when we do. > > > > > > > > > > While we discuss this, I'd like to share about the Great Renaming > in > > core > > > > > Python started by Guido back in 2009. We could probably get some > > > > learnings > > > > > from there? Who knows! > > > > > > > > > > Guido's blog (2009): > > > > > > > https://python-history.blogspot.com/2009/03/great-or-grand-renaming.html > > > > < > > https://python-history.blogspot.com/2009/03/great-or-grand-renaming.html > > > > > > > Follow-up discussion (2024): > > > > > https://discuss.python.org/t/finishing-the-great-renaming/54082 < > > > > https://discuss.python.org/t/finishing-the-great-renaming/54082> > > > > > > > > > > Cheers, > > > > > Omkar > > > > > > > > > > On Wed, Oct 23, 2024 at 7:10 AM Wei Lee <weilee...@gmail.com > > <mailto: > > > > weilee...@gmail.com>> wrote: > > > > > > > > > > > I think we should probably just accept it as an airflow term. At > > least, > > > > > > that’s how I understood it when I first used Airflow. I feel > > renaming > > > > it > > > > > at > > > > > > this stage would require considerable effort from maintainers and > > > > > existing > > > > > > users without providing equivalent benefits. > > > > > > > > > > > > Best, > > > > > > Wei > > > > > > > > > > > > > On Oct 23, 2024, at 8:01 AM, Kaxil Naik <kaxiln...@gmail.com > > > > <mailto:kaxiln...@gmail.com>> wrote: > > > > > > > > > > > > > > Same agreed with Brent & Daniel -- maybe we re-kindle this > > discussion > > > > > for > > > > > > > Airflow 4 :) -- but right now it will cause too much disruption > > > > > > > > > > > > > > On Tue, 22 Oct 2024 at 21:27, Constance Martineau > > > > > > > <consta...@astronomer.io.inva <mailto: > > consta...@astronomer.io.inva>lid> > > > > wrote: > > > > > > > > > > > > > >> In my experience, when you ask those with Airflow experience > > what a > > > > > dag > > > > > > is, > > > > > > >> they'll start talking about workflow attributes - stuff like > > dags > > > > > being > > > > > > a > > > > > > >> series of steps or tasks with owners. The structure doesn't > > come up. > > > > > > >> > > > > > > >> Echo-ing others, at this point, my vote is to embrace the name > > and > > > > > > >> de-emphasize the mathematical structure aspect. > > > > > > >> > > > > > > >> On Tue, Oct 22, 2024 at 3:47 PM Vikram Koka > > > > > > <vik...@astronomer.io.inva <mailto:vik...@astronomer.io.inva > >lid> > > > > > > >> wrote: > > > > > > >> > > > > > > >>> It's an interesting discussion and I remember struggling with > > this > > > > > > when I > > > > > > >>> started working with Airflow. > > > > > > >>> But, I also agree with the viewpoint of it being an > established > > > > > concept > > > > > > >> now > > > > > > >>> regardless of the origin. > > > > > > >>> > > > > > > >>> I am personally leaning towards the perspective best > expressed > > by > > > > > > Daniel > > > > > > >>> Standish and Brent of using Dag as a word, rather than the > > computer > > > > > > >> science > > > > > > >>> concept. > > > > > > >>> > > > > > > >>> Best regards, > > > > > > >>> Vikram > > > > > > >>> > > > > > > >>> > > > > > > >>> On Tue, Oct 22, 2024 at 9:46 AM Oliveira, Niko > > > > > > >> <oniko...@amazon.com.inva <mailto:oniko...@amazon.com.inva > >lid > > > > > > >>>> > > > > > > >>> wrote: > > > > > > >>> > > > > > > >>>> I agree with the general sentiment of: You're right Ryan, > DAG > > > > isn't > > > > > > >> great > > > > > > >>>> and I'd rather workflow, but changing it will cause much > more > > > > > wreckage > > > > > > >>> than > > > > > > >>>> it solves. > > > > > > >>>> > > > > > > >>>> Also agree with the idea to just move away from defining > DAG. > > I > > > > > think > > > > > > >>>> we've been naturally doing that as a community for a while > now > > > > > anyway, > > > > > > >> so > > > > > > >>>> that feels like a natural step. > > > > > > >>>> > > > > > > >>>> Cheers, > > > > > > >>>> Niko > > > > > > >>>> > > > > > > >>>> ________________________________ > > > > > > >>>> From: Ash Berlin-Taylor <a...@apache.org <mailto: > > a...@apache.org>> > > > > > > >>>> Sent: Tuesday, October 22, 2024 9:06:39 AM > > > > > > >>>> To: dev@airflow.apache.org <mailto:dev@airflow.apache.org> > > > > > > >>>> Subject: RE: [EXT] Airflow should deprecate the term "DAG" > > for end > > > > > > >> users > > > > > > >>>> > > > > > > >>>> CAUTION: This email originated from outside of the > > organization. > > > > Do > > > > > > not > > > > > > >>>> click links or open attachments unless you can confirm the > > sender > > > > > and > > > > > > >>> know > > > > > > >>>> the content is safe. > > > > > > >>>> > > > > > > >>>> > > > > > > >>>> > > > > > > >>>> AVERTISSEMENT: Ce courrier électronique provient d’un > > expéditeur > > > > > > >> externe. > > > > > > >>>> Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si > > vous > > > > ne > > > > > > >>> pouvez > > > > > > >>>> pas confirmer l’identité de l’expéditeur et si vous n’êtes > pas > > > > > certain > > > > > > >>> que > > > > > > >>>> le contenu ne présente aucun risque. > > > > > > >>>> > > > > > > >>>> > > > > > > >>>> > > > > > > >>>> Best argument in favour of keeping “dags” as a term — > getting > > to > > > > > > re-use > > > > > > >>>> puns like https://i.imgflip.com/1xhtwh.jpg < > > > > https://i.imgflip.com/1xhtwh.jpg> > > > > > > >>>> > > > > > > >>>> In all seriousness: I don’t mind either way, both sides have > > good > > > > > > >> reasons > > > > > > >>>> presented. > > > > > > >>>> > > > > > > >>>> -a > > > > > > >>>> > > > > > > >>>>> On 22 Oct 2024, at 17:03, Daniel Standish > > > > > > >>>> <daniel.stand...@astronomer.io.inva <mailto: > > > > daniel.stand...@astronomer.io.inva>LID> wrote: > > > > > > >>>>> > > > > > > >>>>> Yeah just say, when asked where the name comes from, "well, > > no > > > > one > > > > > > >>>> actually > > > > > > >>>>> knows but..." and then make something up. > > > > > > >>>>> > > > > > > >>>>> On Tue, Oct 22, 2024 at 8:31 AM Jarek Potiuk < > > ja...@potiuk.com > > > > <mailto:ja...@potiuk.com>> > > > > > > >> wrote: > > > > > > >>>>> > > > > > > >>>>>> Just to clarify - "directed acyclic graph" is the > > > > tongue-twister, > > > > > > >>>>>> > > > > > > >>>>>> On Tue, Oct 22, 2024 at 5:29 PM Jarek Potiuk < > > ja...@potiuk.com > > > > <mailto:ja...@potiuk.com>> > > > > > > >>> wrote: > > > > > > >>>>>> > > > > > > >>>>>>> I like what both Daniel and Brent wrote. I would very > much > > want > > > > > to > > > > > > >> be > > > > > > >>>>>> able > > > > > > >>>>>>> to say just "dag" without explaining it further. > > > > > > >>>>>>> > > > > > > >>>>>>> For me every time I explain "DAG" at a talk it's a > > > > > tongue-twister, > > > > > > >>> and > > > > > > >>>> I > > > > > > >>>>>>> almost stutter on trying to recall how to pronounce it > > > > properly. > > > > > > >>>>>>> > > > > > > >>>>>>> J. > > > > > > >>>>>>> > > > > > > >>>>>>> > > > > > > >>>>>>> On Tue, Oct 22, 2024 at 5:27 PM Brent Bovenzi > > > > > > >>>>>> <br...@astronomer.io.inva <mailto: > br...@astronomer.io.inva > > >lid> > > > > > > >>>>>>> wrote: > > > > > > >>>>>>> > > > > > > >>>>>>>> I remember we explored renaming "DAG" when starting on > > AIP-38 > > > > to > > > > > > >>>>>> modernize > > > > > > >>>>>>>> the UI. Both "pipeline" or "workflow" are more > > descriptive of > > > > > what > > > > > > >>> one > > > > > > >>>>>> is > > > > > > >>>>>>>> actually doing while Directed Acyclic Graph is an > > > > implementation > > > > > > >>>> detail. > > > > > > >>>>>>>> But I agree with Daniel Standish, at this point "DAG" > has > > > > become > > > > > > >>> "dag" > > > > > > >>>>>> , a > > > > > > >>>>>>>> word in its own right. > > > > > > >>>>>>>> > > > > > > >>>>>>>> Examples for "dag" are abound in community discussion, > > Airflow > > > > > > >>> Summit > > > > > > >>>>>>>> talks, documentation and even in the UI. Let's embrace > > "dag". > > > > A > > > > > > >> user > > > > > > >>>>>> just > > > > > > >>>>>>>> needs to learn one new word vs the technical concept > > behind > > > > that > > > > > > >>>> word. I > > > > > > >>>>>>>> think that is much less effort than refactoring so much > > code, > > > > > > >>>>>>>> documentation, blog posts, stack overflow questions, > etc. > > > > > > >>>>>>>> > > > > > > >>>>>>>> On Tue, Oct 22, 2024 at 10:51 AM Daniel Standish > > > > > > >>>>>>>> <daniel.stand...@astronomer.io.inva <mailto: > > > > daniel.stand...@astronomer.io.inva>lid> wrote: > > > > > > >>>>>>>> > > > > > > >>>>>>>>> I am skeptical. Seems like introducing a lot of pain > for > > > > > > >>>> questionable > > > > > > >>>>>>>>> benefit. But, I am def sympathetic to the idea. I agree > > the > > > > > > >>>>>>>> association > > > > > > >>>>>>>>> with "directed acyclic graph" is not helpful. > > > > > > >>>>>>>>> > > > > > > >>>>>>>>> And along those lines, I offer here some less invasive > > > > > > >> mitigations. > > > > > > >>>>>>>>> > > > > > > >>>>>>>>> One thing we can do no matter what is to de-emphasize > the > > > > math > > > > > > >> nerd > > > > > > >>>>>>>> origins > > > > > > >>>>>>>>> of the name. That is to say, in docs / website / etc, > > *never > > > > > > >>> define* > > > > > > >>>>>>>>> airflow's "dag" concept as a directed acyclic graph. > > Always > > > > > > >> define > > > > > > >>>> it > > > > > > >>>>>>>> as a > > > > > > >>>>>>>>> pipeline, collection of tasks, workflow etc. > > > > > > >>>>>>>>> > > > > > > >>>>>>>>> The "directed acyclic graph" part of it is like a > > historical > > > > > > >>>> footnote, > > > > > > >>>>>>>> and > > > > > > >>>>>>>>> we could make one mention of it somewhere hidden. > > > > > > >>>>>>>>> > > > > > > >>>>>>>>> We could also start using lowercase in the docs in > > general > > > > e.g. > > > > > > >>>>>> writing > > > > > > >>>>>>>>> "dag" / "dags" instead of writing "DAG" / "DAGs" etc. > The > > > > > upper > > > > > > >>> case > > > > > > >>>>>>>> part > > > > > > >>>>>>>>> of it makes it look like an acronym; but "dag" in > > airlfow is > > > > > just > > > > > > >>> an > > > > > > >>>>>>>>> airflow concept and the association with "DAGs" is not > > really > > > > > > >>>>>> unhelpful. > > > > > > >>>>>>>>> > > > > > > >>>>>>>>> In other words embrace that "dag" in airflow is its own > > > > thing, > > > > > is > > > > > > >>>>>>>>> *not* strictly > > > > > > >>>>>>>>> speaking a directed acyclic graph (which nobody knows > > about > > > > > > >>> anyway), > > > > > > >>>>>> and > > > > > > >>>>>>>>> tell them what it is in simple terms that normal people > > > > > > >> understand. > > > > > > >>>>>>>>> > > > > > > >>>>>>>>> > > > > > > >>>>>>>>> On Tue, Oct 22, 2024 at 7:27 AM Jarek Potiuk < > > > > ja...@potiuk.com <mailto:ja...@potiuk.com> > > > > > > > > > > > > >>>>>> wrote: > > > > > > >>>>>>>>> > > > > > > >>>>>>>>>> DAG is so embedded into what we do that it will be > > extremely > > > > > > >>>>>>>> difficult to > > > > > > >>>>>>>>>> get rid of it completely. Also I think it will make a > > lot of > > > > > > >>>>>> "google" > > > > > > >>>>>>>>>> searches and "stack overflow" searches not finding the > > right > > > > > > >>>>>> answers. > > > > > > >>>>>>>>> This > > > > > > >>>>>>>>>> is one of the strengths of Airflow - besides the > > community > > > > and > > > > > > >>> ideas > > > > > > >>>>>>>> that > > > > > > >>>>>>>>>> Bernd mentioned - is the vast number of examples, > > problems > > > > and > > > > > > >>>>>>>> solutions > > > > > > >>>>>>>>>> you can so easily find (and we have to remember that > > all the > > > > > AI > > > > > > >>>>>>>> trained > > > > > > >>>>>>>>> on > > > > > > >>>>>>>>>> past data will be also rather poorly matching queries > of > > > > > people. > > > > > > >>>>>>>>>> > > > > > > >>>>>>>>>> I am not too attached to DAG. I could easily switch. > > And if > > > > we > > > > > > >> do > > > > > > >>> - > > > > > > >>>>>> I > > > > > > >>>>>>>>>> would be for using workflow or pipeline instead of > > `dag` if > > > > > not > > > > > > >>> the > > > > > > >>>>>>>> above > > > > > > >>>>>>>>>> reason, but I think I am here with Igor that it might > > cause > > > > > more > > > > > > >>>>>>>> problems > > > > > > >>>>>>>>>> than it solves. > > > > > > >>>>>>>>>> > > > > > > >>>>>>>>>> But I am not 100% against - if others will think it's > a > > good > > > > > > >>> idea, I > > > > > > >>>>>>>> am > > > > > > >>>>>>>>> ok > > > > > > >>>>>>>>>> with it. > > > > > > >>>>>>>>>> > > > > > > >>>>>>>>>> J, > > > > > > >>>>>>>>>> > > > > > > >>>>>>>>>> > > > > > > >>>>>>>>>> On Tue, Oct 22, 2024 at 3:12 PM Abhishek Bhakat > > > > > > >>>>>>>>>> <abhishek.bha...@astronomer.io.inva <mailto: > > > > abhishek.bha...@astronomer.io.inva>lid> wrote: > > > > > > >>>>>>>>>> > > > > > > >>>>>>>>>>> Agreed that the word DAG makes very less sense to > > someone > > > > new > > > > > > >> to > > > > > > >>>>>>>>> workflow > > > > > > >>>>>>>>>>> orchestration. But it does also show the nature of > > being > > > > > > >> acyclic. > > > > > > >>>>>>>> Sure, > > > > > > >>>>>>>>>> as > > > > > > >>>>>>>>>>> Bas mentioned, there are ways to workaround it. > Still, > > in > > > > my > > > > > > >>>>>>>> opinion, > > > > > > >>>>>>>>>> there > > > > > > >>>>>>>>>>> is generally no need for cyclic behavior in workflow > > > > > > >>>>>> orchestration. > > > > > > >>>>>>>>> Most > > > > > > >>>>>>>>>>> (*if > > > > > > >>>>>>>>>>> not all*) cases can be in some way can be covered > > using an > > > > > > >>> acyclic > > > > > > >>>>>>>>> manner > > > > > > >>>>>>>>>>> with multiple runs. Hence, the idempotency. So I > would > > want > > > > > the > > > > > > >>>>>>>>> "acyclic" > > > > > > >>>>>>>>>>> word to stick. > > > > > > >>>>>>>>>>> > > > > > > >>>>>>>>>>> Regards, > > > > > > >>>>>>>>>>> Avi > > > > > > >>>>>>>>>>> > > > > > > >>>>>>>>>>> On Tue, Oct 22, 2024 at 12:41 PM < > > > > bernd.stroe...@kosakya.de <mailto:bernd.stroe...@kosakya.de>> > > > > > > >>>>>> wrote: > > > > > > >>>>>>>>>>> > > > > > > >>>>>>>>>>>> Brilliant, I am on the way to become an Airflow Fan; > > so > > > > many > > > > > > >> new > > > > > > >>>>>>>>> ideas. > > > > > > >>>>>>>>>>>> > > > > > > >>>>>>>>>>>> The Term DAG is misleading; it should be replaced by > > the > > > > > more > > > > > > >>>>>>>> general > > > > > > >>>>>>>>>>> Term > > > > > > >>>>>>>>>>>> Airflow (Workflow) Graph (AFG) or Airflow (Petri) > Net > > > > (AFN) > > > > > > >>>>>> (maybe > > > > > > >>>>>>>>>>> without > > > > > > >>>>>>>>>>>> a direction); > > > > > > >>>>>>>>>>>> and ... these Graphs should be stored in a Graph > > Database. > > > > > > >>>>>>>>>>>> > > > > > > >>>>>>>>>>>> Every Node or Sup-Graph of an Airflow Graph (AFG) > > might be > > > > > > >>>>>>>> assigned > > > > > > >>>>>>>>> to > > > > > > >>>>>>>>>> an > > > > > > >>>>>>>>>>>> executable (Python-, Rust-, ... ) member of a > library. > > > > > > >>>>>>>>>>>> > > > > > > >>>>>>>>>>>> A running Graph might have a different structure > than > > a > > > > > > >>>>>>>> configuration > > > > > > >>>>>>>>>>>> Graph. > > > > > > >>>>>>>>>>>> > > > > > > >>>>>>>>>>>> Forget that if you think it's bullshit. > > > > > > >>>>>>>>>>>> > > > > > > >>>>>>>>>>>> Best Regards > > > > > > >>>>>>>>>>>> > > > > > > >>>>>>>>>>>> Bernd Ströhle > > > > > > >>>>>>>>>>>> M: +49 171 5357916 > > > > > > >>>>>>>>>>>> E: bernd.stroe...@gmail.com <mailto: > > > > bernd.stroe...@gmail.com> > > > > > > >>>>>>>>>>>> > > > > > > >>>>>>>>>>>> > > > > > > >>>>>>>>>>>> -----Original Message----- > > > > > > >>>>>>>>>>>> From: Igor Kholopov <ikholo...@google.com.inva > > <mailto: > > > > ikholo...@google.com.inva>LID> > > > > > > >>>>>>>>>>>> Sent: Tuesday, October 22, 2024 12:02 PM > > > > > > >>>>>>>>>>>> To: dev@airflow.apache.org <mailto: > > dev@airflow.apache.org > > > > > > > > > > > >>>>>>>>>>>> Subject: Re: Airflow should deprecate the term "DAG" > > for > > > > end > > > > > > >>>>>> users > > > > > > >>>>>>>>>>>> > > > > > > >>>>>>>>>>>> Even though the term "DAG" is clearly suboptimal, it > > is > > > > part > > > > > > >> of > > > > > > >>>>>>>>> Airflow > > > > > > >>>>>>>>>>>> DAG definition interface at so many levels, that any > > > > attempt > > > > > > >> to > > > > > > >>>>>>>>> change > > > > > > >>>>>>>>>> it > > > > > > >>>>>>>>>>>> will only introduce more chaos, not reduce it. The > > only > > > > > thing > > > > > > >>>>>>>> that is > > > > > > >>>>>>>>>>> worse > > > > > > >>>>>>>>>>>> than a poorly chosen name in the code is when there > > are > > > > two > > > > > > >> ways > > > > > > >>>>>>>> to > > > > > > >>>>>>>>>>> define > > > > > > >>>>>>>>>>>> the same thing. Countless articles and tutorials > will > > > > > suddenly > > > > > > >>>>>>>> become > > > > > > >>>>>>>>>>>> confusing as they all refer to workflows as "DAG"s. > > > > > > >>>>>>>>>>>> > > > > > > >>>>>>>>>>>> We are already at risk of scaring the users away > with > > a > > > > > number > > > > > > >>>>>> of > > > > > > >>>>>>>>>>> breaking > > > > > > >>>>>>>>>>>> changes in Airflow 3, promising even more breaking > > changes > > > > > for > > > > > > >>>>>> the > > > > > > >>>>>>>>> most > > > > > > >>>>>>>>>>>> basic things is not something that people are > looking > > for. > > > > > > >>>>>>>> Attempting > > > > > > >>>>>>>>>> to > > > > > > >>>>>>>>>>>> change the fundamental terms will be interpreted as > an > > > > even > > > > > > >>>>>>>> stronger > > > > > > >>>>>>>>>>> signal > > > > > > >>>>>>>>>>>> of project immaturity. > > > > > > >>>>>>>>>>>> > > > > > > >>>>>>>>>>>> Given that, I oppose the idea of changing the term > in > > the > > > > > long > > > > > > >>>>>>>> run. I > > > > > > >>>>>>>>>>> even > > > > > > >>>>>>>>>>>> stricter oppose the idea of deprecating it in the > DAG > > > > > > >> definition > > > > > > >>>>>>>>>>> interface. > > > > > > >>>>>>>>>>>> We better put our time and efforts in other places > in > > > > > Airflow, > > > > > > >>>>>> of > > > > > > >>>>>>>>> which > > > > > > >>>>>>>>>>>> there are plenty. > > > > > > >>>>>>>>>>>> > > > > > > >>>>>>>>>>>> Kind regards, > > > > > > >>>>>>>>>>>> Igor > > > > > > >>>>>>>>>>>> > > > > > > >>>>>>>>>>>> On Tue, Oct 22, 2024 at 10:36 AM Bas Harenslak > > > > > > >>>>>>>>>> <b...@astronomer.io.inva <mailto: > b...@astronomer.io.inva > > >lid > > > > > > >>>>>>>>>>>> > > > > > > >>>>>>>>>>>> wrote: > > > > > > >>>>>>>>>>>> > > > > > > >>>>>>>>>>>>> Couple of thoughts: > > > > > > >>>>>>>>>>>>> > > > > > > >>>>>>>>>>>>> 1. The boundaries/properties of “DAG” have already > > faded > > > > > over > > > > > > >>>>>>>> time, > > > > > > >>>>>>>>>>>>> for example there are now several ways to create > > cyclic > > > > > > >>>>>> graphs, > > > > > > >>>>>>>>> e.g. > > > > > > >>>>>>>>>>>>> using the @continuous schedule. I imagine these > > > > properties > > > > > > >>>>>>>>> vanishing > > > > > > >>>>>>>>>>>>> even more in the future, so from that perspective I > > > > support > > > > > > >>>>>>>>> changing > > > > > > >>>>>>>>>>>>> “DAG" to a more generic name. > > > > > > >>>>>>>>>>>>> > > > > > > >>>>>>>>>>>>> 2. How other orchestration frameworks do naming: > > > > > > >>>>>>>>>>>>> Dagster: pipeline > > > > > > >>>>>>>>>>>>> Prefect: flow > > > > > > >>>>>>>>>>>>> Flyte: workflow > > > > > > >>>>>>>>>>>>> Temporal: workflow > > > > > > >>>>>>>>>>>>> Kestra: flow > > > > > > >>>>>>>>>>>>> > > > > > > >>>>>>>>>>>>> I think “workflow” is the most fitting name. > > > > > > >>>>>>>>>>>>> > > > > > > >>>>>>>>>>>>> 3. Given the large impact of this change, I suggest > > > > > defining > > > > > > >> a > > > > > > >>>>>>>>> clear > > > > > > >>>>>>>>>>>>> path forward. Would we first introduce the > > deprecation in > > > > > > >>>>>>>> Airflow > > > > > > >>>>>>>>> 3, > > > > > > >>>>>>>>>>>>> and remove “DAG” in Airflow 4? > > > > > > >>>>>>>>>>>>> > > > > > > >>>>>>>>>>>>> Bas > > > > > > >>>>>>>>>>>>> > > > > > > >>>>>>>>>>>>>> On 22 Oct 2024, at 09:22, Neil < > neil4r...@gmail.com > > > > <mailto:neil4r...@gmail.com>> > > > > > wrote: > > > > > > >>>>>>>>>>>>>> > > > > > > >>>>>>>>>>>>>> I don't see a problem with the term DAG, > especially > > when > > > > > > >>>>>> most > > > > > > >>>>>>>>> other > > > > > > >>>>>>>>>>>>>> platforms embrace the term wholeheartedly. > > > > > > >>>>>>>>>>>>>> I don't see anything intimidating or confusing > > about it > > > > at > > > > > > >>>>>>>> all, > > > > > > >>>>>>>>>>>>>> changing the term though would be fairly confusing > > to > > > > most > > > > > > >>>>>>>> users > > > > > > >>>>>>>>>> who > > > > > > >>>>>>>>>>>>>> have been > > > > > > >>>>>>>>>>>>> using > > > > > > >>>>>>>>>>>>>> the term for years. > > > > > > >>>>>>>>>>>>>> > > > > > > >>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 1:18 AM Tzu-ping Chung > > > > > > >>>>>>>>>>>>>> <t...@astronomer.io.inva <mailto: > > t...@astronomer.io.inva > > > > >lid > > > > > > >>>>>>>>>>>>>> > > > > > > >>>>>>>>>>>>>> wrote: > > > > > > >>>>>>>>>>>>>> > > > > > > >>>>>>>>>>>>>>> I totally agree with doing away with the term > DAG. > > The > > > > > only > > > > > > >>>>>>>>>> problem > > > > > > >>>>>>>>>>>>> (aside > > > > > > >>>>>>>>>>>>>>> from actually telling people—including myself—to > > stop > > > > > using > > > > > > >>>>>>>> the > > > > > > >>>>>>>>>>>>>>> term) > > > > > > >>>>>>>>>>>>> is to > > > > > > >>>>>>>>>>>>>>> come up with a reasonable alternative. > > > > > > >>>>>>>>>>>>>>> > > > > > > >>>>>>>>>>>>>>> I can’t recall who, but someone mentioned > > “workflow” is > > > > > not > > > > > > >>>>>>>> very > > > > > > >>>>>>>>>>>>> accurate > > > > > > >>>>>>>>>>>>>>> for Airflow. The term “definition” was proposed, > > but > > > > > it’s a > > > > > > >>>>>>>> bit > > > > > > >>>>>>>>>>>>>>> broad; I tried to use it in a few places and kept > > > > finding > > > > > > >>>>>>>> myself > > > > > > >>>>>>>>>>>>>>> doubting “what definition?” and wanting to > clarify > > “DAG > > > > > > >>>>>>>>>> definition” > > > > > > >>>>>>>>>>>>>>> (defeating the purpose). > > > > > > >>>>>>>>>>>>>>> > > > > > > >>>>>>>>>>>>>>> TP > > > > > > >>>>>>>>>>>>>>> > > > > > > >>>>>>>>>>>>>>> > > > > > > >>>>>>>>>>>>>>>> On 22 Oct 2024, at 13:07, Jens Scheffler > > > > > > >>>>>>>>>>>>>>>> <j_scheff...@gmx.de.inva <mailto: > > > > j_scheff...@gmx.de.inva>LID> > > > > > > >>>>>>>>>>>>>>> wrote: > > > > > > >>>>>>>>>>>>>>>> > > > > > > >>>>>>>>>>>>>>>> Hi Ryan, > > > > > > >>>>>>>>>>>>>>>> > > > > > > >>>>>>>>>>>>>>>> Thanks for posting. I share the exactly same > > > > > observation, > > > > > > >>>>>>>> had a > > > > > > >>>>>>>>>>>>>>>> short > > > > > > >>>>>>>>>>>>>>> laight because the DAG question is always an > > > > introduction > > > > > > >>>>>> if > > > > > > >>>>>>>>>>>>>>> someone > > > > > > >>>>>>>>>>>>> joins > > > > > > >>>>>>>>>>>>>>> the party. I think a global renaming makes sense. > > > > > > >>>>>> Especially > > > > > > >>>>>>>>> when > > > > > > >>>>>>>>>>>>>>> we > > > > > > >>>>>>>>>>>>> also > > > > > > >>>>>>>>>>>>>>> rename Dataset to Asset this is also a reasonable > > step. > > > > > > >>>>>>>> Concepts > > > > > > >>>>>>>>>>>>>>> still > > > > > > >>>>>>>>>>>>> can > > > > > > >>>>>>>>>>>>>>> stay the same. > > > > > > >>>>>>>>>>>>>>>> > > > > > > >>>>>>>>>>>>>>>> So I hope I don‘t need to join hiding below the > > desk > > > > > with > > > > > > >>>>>>>> you > > > > > > >>>>>>>>> and > > > > > > >>>>>>>>>>>>>>>> +1 > > > > > > >>>>>>>>>>>>> for > > > > > > >>>>>>>>>>>>>>> raising the discussion. > > > > > > >>>>>>>>>>>>>>>> > > > > > > >>>>>>>>>>>>>>>> Technically we can still think if we keep > details > > of > > > > > > >>>>>> python > > > > > > >>>>>>>>> names > > > > > > >>>>>>>>>>>>>>>> the > > > > > > >>>>>>>>>>>>>>> same because the execution is still a DAG… but > user > > > > > facing > > > > > > >>>>>> it > > > > > > >>>>>>>>> is a > > > > > > >>>>>>>>>>>>> workflow. > > > > > > >>>>>>>>>>>>>>>> > > > > > > >>>>>>>>>>>>>>>> Jens > > > > > > >>>>>>>>>>>>>>>> > > > > > > >>>>>>>>>>>>>>>> Sent from my Smartphone > > > > > > >>>>>>>>>>>>>>>> > > > > > > >>>>>>>>>>>>>>>>> On 21. Oct 2024, at 23:56, Ryan Hatter < > > > > > > >>>>>>>>>> ryan.hat...@astronomer.io <mailto: > > ryan.hat...@astronomer.io > > > > > > > > > > > >>>>>>>>>>>>> .invalid> > > > > > > >>>>>>>>>>>>>>> wrote: > > > > > > >>>>>>>>>>>>>>>>> > > > > > > >>>>>>>>>>>>>>>>> Everyone please sheathe your swords... at least > > for > > > > > now. > > > > > > >>>>>>>>>>>>>>>>> > > > > > > >>>>>>>>>>>>>>>>> The term "DAG" has very little meaning to > Airflow > > > > > users. > > > > > > >>>>>>>>> Indeed, > > > > > > >>>>>>>>>>>>>>>>> it > > > > > > >>>>>>>>>>>>> has > > > > > > >>>>>>>>>>>>>>>>> little meaning outside of some mathematicians > and > > > > > > >>>>>> software > > > > > > >>>>>>>>>>>>>>>>> engineers > > > > > > >>>>>>>>>>>>> for > > > > > > >>>>>>>>>>>>>>>>> whom the properties of a DAG actually matter. > For > > > > > someone > > > > > > >>>>>>>> new > > > > > > >>>>>>>>> to > > > > > > >>>>>>>>>>>>>>>>> data engineering or workflow orchestration, one > > of > > > > the > > > > > > >>>>>>>> first > > > > > > >>>>>>>>>>>>>>>>> questions they > > > > > > >>>>>>>>>>>>>>> will > > > > > > >>>>>>>>>>>>>>>>> likely have is, "what on earth is a DAG?" The > > answer > > > > is > > > > > > >>>>>>>> almost > > > > > > >>>>>>>>>>>>>>>>> always, "It's a directed acyclic graph. You > don't > > > > need > > > > > to > > > > > > >>>>>>>>> worry > > > > > > >>>>>>>>>>>>>>>>> about what > > > > > > >>>>>>>>>>>>> that > > > > > > >>>>>>>>>>>>>>>>> means; it's just a term for your workflow." > > > > > > >>>>>>>>>>>>>>>>> > > > > > > >>>>>>>>>>>>>>>>> The term "DAG" is problematic for at least a > > couple > > > > > > >>>>>>>> important > > > > > > >>>>>>>>>>>> reasons: > > > > > > >>>>>>>>>>>>>>>>> > > > > > > >>>>>>>>>>>>>>>>> *Complexity for New Users*: As mentioned above, > > "DAG" > > > > > is > > > > > > >>>>>>>>>>>>>>>>> unnecessarily intimidating and confusing. We > want > > > > > Airflow > > > > > > >>>>>>>> to > > > > > > >>>>>>>>> be > > > > > > >>>>>>>>>>>>>>>>> approachable, and > > > > > > >>>>>>>>>>>>>>> using > > > > > > >>>>>>>>>>>>>>>>> technical jargon like "DAG" right off the bat > > creates > > > > > an > > > > > > >>>>>>>>> initial > > > > > > >>>>>>>>>>>>>>> barrier to > > > > > > >>>>>>>>>>>>>>>>> understanding. > > > > > > >>>>>>>>>>>>>>>>> > > > > > > >>>>>>>>>>>>>>>>> *Disconnect Between DAG and Workflow Concepts*: > > The > > > > DAG > > > > > > >>>>>> is > > > > > > >>>>>>>>> just > > > > > > >>>>>>>>>>>>>>>>> one component of an Airflow workflow. The > > workflow > > > > > > >>>>>> includes > > > > > > >>>>>>>>> its > > > > > > >>>>>>>>>>>>>>>>> schedule, retries, timeouts, a dozen other > > > > parameters, > > > > > > >>>>>> and > > > > > > >>>>>>>>> other > > > > > > >>>>>>>>>>>>>>>>> metadata that > > > > > > >>>>>>>>>>>>> the > > > > > > >>>>>>>>>>>>>>>>> DAG component doesn’t account for. > > > > > > >>>>>>>>>>>>>>>>> > > > > > > >>>>>>>>>>>>>>>>> Consider the following from the Airflow > homepage > > > > > > >>>>>>>>>>>>>>>>> <https://airflow.apache.org/> < > > > > https://airflow.apache.org/>>. > > > > > > >>>>>>>>>>>>>>>>> > > > > > > >>>>>>>>>>>>>>>>> Apache Airflow® is a platform created by the > > > > community > > > > > to > > > > > > >>>>>>>>>>>>>>> programmatically > > > > > > >>>>>>>>>>>>>>>>> author, schedule and monitor workflows. > > > > > > >>>>>>>>>>>>>>>>> Then, if we look at the "What is Airflow?" docs > > page > > > > > > >>>>>>>>>>>>>>>>> < > > > > > > >>>>>>>>>> > > > > > > >> > > https://airflow.apache.org/docs/apache-airflow/stable/index.html < > > > > https://airflow.apache.org/docs/apache-airflow/stable/index.html> > > > > > > >>>>>>>>>>>>>>>>>> , > > > > > > >>>>>>>>>>>>> we > > > > > > >>>>>>>>>>>>>>> can > > > > > > >>>>>>>>>>>>>>>>> see that the docs explain what Airflow is > without > > > > using > > > > > > >>>>>>>> "DAG." > > > > > > >>>>>>>>>>>>>>>>> It's > > > > > > >>>>>>>>>>>>>>> only in > > > > > > >>>>>>>>>>>>>>>>> the *workflow* Python code that the term is > > > > introduced > > > > > > >>>>>> out > > > > > > >>>>>>>> of > > > > > > >>>>>>>>>>>>>>>>> nowhere > > > > > > >>>>>>>>>>>>>>> as a > > > > > > >>>>>>>>>>>>>>>>> comment that awkwardly tries to explain it: > > > > > > >>>>>>>>>>>>>>>>> > > > > > > >>>>>>>>>>>>>>>>> # A DAG represents a workflow, a collection of > > tasks > > > > > > >>>>>>>>>>>>>>>>> > > > > > > >>>>>>>>>>>>>>>>> It makes sense to not refer to DAGs in these > > > > > > >>>>>> introductions > > > > > > >>>>>>>> to > > > > > > >>>>>>>>>>>>>>>>> Airflow, because *Airflow doesn't orchestrate > > DAGs; > > > > it > > > > > > >>>>>>>>>>> orchestrates > > > > > > >>>>>>>>>>>> workflows*. > > > > > > >>>>>>>>>>>>>>> The > > > > > > >>>>>>>>>>>>>>>>> DAG is the model that, for reasons irrelevant > to > > > > almost > > > > > > >>>>>>>> every > > > > > > >>>>>>>>>>>>>>>>> user, workflows must adhere to. > > > > > > >>>>>>>>>>>>>>>>> > > > > > > >>>>>>>>>>>>>>>>> So, I propose at least adding an alias for the > > term > > > > > "DAG" > > > > > > >>>>>>>> and > > > > > > >>>>>>>>>>>>>>>>> updating documentation to replace "DAG" with > > > > > "workflow". > > > > > > >>>>>>>>>>>>>>>>> > > > > > > >>>>>>>>>>>>>>>>> For example, instead of... > > > > > > >>>>>>>>>>>>>>>>> > > > > > > >>>>>>>>>>>>>>>>> @dag( > > > > > > >>>>>>>>>>>>>>>>> schedule="@daily", > > > > > > >>>>>>>>>>>>>>>>> ... > > > > > > >>>>>>>>>>>>>>>>> dagrun_timeout=timedelta(hours=1) > > > > > > >>>>>>>>>>>>>>>>> ) > > > > > > >>>>>>>>>>>>>>>>> > > > > > > >>>>>>>>>>>>>>>>> Users could do... > > > > > > >>>>>>>>>>>>>>>>> > > > > > > >>>>>>>>>>>>>>>>> @workflow( > > > > > > >>>>>>>>>>>>>>>>> schedule="@daily", > > > > > > >>>>>>>>>>>>>>>>> ... > > > > > > >>>>>>>>>>>>>>>>> run_timeout=timedelta(hours=1) > > > > > > >>>>>>>>>>>>>>>>> ) > > > > > > >>>>>>>>>>>>>>>>> > > > > > > >>>>>>>>>>>>>>>>> > > > > > > >>>>>>>>>>>>>>>>> And with that... I will start running away. > > > > > > >>>>>>>>>>>>>>>> > > > > > > >>>>>>>>>>>>>>>> > > > > > > >>>>>>>>>>>>>>>> > > > > > > >>>>>>>>>> > > > > > > >> > > ------------------------------------------------------------------ > > > > > > >>>>>>>>>>>>>>>> --- To unsubscribe, e-mail: > > > > > > >>>>>>>> dev-unsubscr...@airflow.apache.org <mailto: > > > > dev-unsubscr...@airflow.apache.org> > > > > > > >>>>>>>>>>>>>>>> For additional commands, e-mail: > > > > > > >>>>>>>> dev-h...@airflow.apache.org <mailto: > > > > dev-h...@airflow.apache.org> > > > > > > >>>>>>>>>>>>>>>> > > > > > > >>>>>>>>>>>>>>> > > > > > > >>>>>>>>>>>>>>> > > > > > > >>>>>>>>>>>>>>> > > > > > > >>>>>>>>>> > > > > > > >>> > > ------------------------------------------------------------------- > > > > > > >>>>>>>>>>>>>>> -- To unsubscribe, e-mail: > > > > > > >>>>>>>> dev-unsubscr...@airflow.apache.org <mailto: > > > > dev-unsubscr...@airflow.apache.org> > > > > > > >>>>>>>>>>>>>>> For additional commands, e-mail: > > > > > > >>>>>> dev-h...@airflow.apache.org <mailto: > > dev-h...@airflow.apache.org > > > > > > > > > > > >>>>>>>>>>>>>>> > > > > > > >>>>>>>>>>>>>>> > > > > > > >>>>>>>>>>>>> > > > > > > >>>>>>>>>>>>> > > > > > > >>>>>>>>>>>> > > > > > > >>>>>>>>>>>> > > > > > > >>>>>>>>>>>> > > > > > > >>>>>>>> > > > > > > >>> > > > > --------------------------------------------------------------------- > > > > > > >>>>>>>>>>>> To unsubscribe, e-mail: > > > > dev-unsubscr...@airflow.apache.org <mailto: > > > > dev-unsubscr...@airflow.apache.org> > > > > > > >>>>>>>>>>>> For additional commands, e-mail: > > > > > dev-h...@airflow.apache.org <mailto:dev-h...@airflow.apache.org> > > > > > > >>>>>>>>>>>> > > > > > > >>>>>>>>>>>> > > > > > > >>>>>>>>>>> > > > > > > >>>>>>>>>> > > > > > > >>>>>>>>> > > > > > > >>>>>>>> > > > > > > >>>>>>> > > > > > > >>>>>> > > > > > > >>>> > > > > > > >>>> > > > > > > >>> > > > > > > >> > > > > > > > > > > > > > > > > > > > > --------------------------------------------------------------------- > > > > > > To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org > > <mailto: > > > > dev-unsubscr...@airflow.apache.org> > > > > > > For additional commands, e-mail: dev-h...@airflow.apache.org > > <mailto: > > > > dev-h...@airflow.apache.org> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org > > For additional commands, e-mail: dev-h...@airflow.apache.org > > > > >
