Sorry for the few typos ... My slight dyslexia did not help and autocorrect did not help either this time : dept -> debt, routed -> rooted and a few other small typos :) .
On Tue, Feb 18, 2025 at 4:28 PM Jarek Potiuk <ja...@potiuk.com> wrote: > First of all - yes, agree with Lee and Ash, I think we do not need to > change the public interface, I'd also be -0 on that, but docs, internal > strings, yes we could. > > But .... After thinking a bit and looking at the discussion here, where > people have concerns, I have an alternative take. And I know this might be > controversial, and some people might find it confusing, but why don't we > come up with a new acronym for DAG, one that will be more airflow and less > "mathematics/algorithm". And I have, I think a good reason for it, and I > even have a proposal. Bear with me, > > DAG => *Data Airflow Graph* > > As many of us mentioned, DAG and Airflow are almost 100% 1-1 now (with > exception of some niche products - you know which one I speak about - which > are unfortunate enough to use the "Dag" in their name - I will explain why > this is unfortunate, IMHO). > > We are lucky to have Airflow for "A" in the acronym and we can make it > even more "synonymous" -> Airflow <> DAG with (A) being Airflow. > > Now, why I think "Directed Acyclic Graph" is unfortunate. > > Not only because "Directed" and "Acyclic" are mostly not in the vocabulary > of most of our users - as Ryan, starting this thread explained yes, I agree > with it, but also I think there is a more important reason - I simply > believe in the future our DAGs might not (and likely will not) be "acyclic". > > I know for some it might be herezy and we have not discussed any of that > yet, but more and more I keep on hearing that Machine Learning Workflows of > the future are often somewhat cyclic. In our case there is one type of > repeat-in-cycle behaviour - we have "task retry on failure" - but this is > only on failure and it basically mens "idempotent" retry - we want to redo > something and get single result out of that and basically discard the > previous result. But machine learning/AI workflows are different. I had a > lot of conversations recently (I even co-organized "AI low-level > engineering and hacking" devroom to learn more about those) - and I keep on > hearing that "Repeat a sequence of things several times and refine" is > precisely what AI / ML workflows will need - especially with Agentic > workflows - often we have cooperating agents which **could** be part of the > same DAG, but currently in Airflow where our DAGs are "Acyclic" we can > only express "repeat several time and refine" as a single task that will do > such refining. If we would like to split such an "agent" doing their own > work in "cycles" and communicating with other "agents" doing the same - > represented as a multi-task-entity, we have no way to express it using our > current "acyclic" semantics. > > But - other than our implementation makes that assumption currently - > there is nothing fundamental with Airflow being able to run such "partially > cyclic workflows". We have all the pieces, we would have to change > dependency calculation to allow for cycles, update UI to handle cycles > representation - and we could in theory quite seamlessly, support "cyclic" > workflows. > > I know, I know, that's the first time we hear it and that's huge change - > of course, I do not want to start discussing any details of it, not before > Airflow 3.0 is out - but I think we should be open that at some point of > time the "Acyclic" property is going to go away, And my personal intuition > tells me that a year from now we will have acyclic workflows (assuming that > all the foundational work we do on Airflow 3 will really help in decreasing > all the technical dept, and will open our minds to new ideas). > > That's why I think - the sooner we get rid of "Directed Acyclic" - the > better. that will - even for us, maintainer free us from being routed in > the acyclic nature of Airflow. > > How about we change to "Data Airflow Graph" (or any other good acronym > which has (A)irflow instead of (A)cyclic). The name is just a proposal. > > Now.. I go hide somewhere as I believe that might cause a "flamewar" :), > unless on request of Ryan you already sheathed your swords in the > discussion (which seems everyone did). > > But honestly - I would really love what you think of "let's just break > with Acyclic NOW" and shape our future by using a different acronym. > > J. > > > > > > > > > > > On Tue, Feb 18, 2025 at 10:54 AM Ephraim Anierobi < > ephraimanier...@gmail.com> wrote: > >> When I started contributing to Airflow, I had to read up on DAG, and now I >> know what it means. I wonder if we are about to have users who use the >> term >> Dag without knowing that there's a DAG. >> >> Slightly concerned. >> >> -ephraim >> >> On Tue, 18 Feb 2025 at 10:26, Wei Lee <weilee...@gmail.com> wrote: >> >> > For user-facing things (e.g., `from airflow import DAG`), I’m more like >> > -0. But for documentation, docstring, internal things and etc., we >> probably >> > could still change most of them? >> > >> > Best, >> > Wei >> > >> > > On Feb 18, 2025, at 5:10 PM, Ash Berlin-Taylor <a...@apache.org> >> wrote: >> > > >> > > I also don’t personally think it’s worth the pain (not to mention >> > backcompat workaround) to rename DAG to Dag, so I’d be -0.5 on that. >> > > >> > > -ash >> > > >> > >> On 18 Feb 2025, at 04:40, Daniel Imberman <daniel.imber...@gmail.com >> > >> > wrote: >> > >> >> > >> I think my biggest concern is a marketing one and not a technical >> one. >> > >> >> > >> As has been mentioned on the thread the terms airflow and dag are >> kind >> > of >> > >> synonymous and I certainly don’t want to give the impression that we >> are >> > >> breaking more than we are breaking. >> > >> >> > >> I wouldn’t die on this hill, but I’m slightly concerned. >> > >> >> > >> On Mon, Feb 17, 2025 at 6:00 PM Wei Lee <wei...@apache.org> wrote: >> > >> >> > >>> I’m not sure about adding ruff rules here 🤔 I think ruff rules are >> > best >> > >>> suited for user-facing things but not the airflow code base itself. >> If >> > what >> > >>> we mean is adding a rule to avoid users using "DAG" *after* we >> rename >> > it, >> > >>> it's definitely a +1000. >> > >>> >> > >>> I just created GitHub issues for this removing "DAG" >> > >>> >> > >>> https://github.com/apache/airflow/issues/46842 >> > >>> https://github.com/apache/airflow/issues/46843 >> > >>> >> > >>> One thing I'm not sure about is whether we want to get rid of `from >> > >>> airflow import DAG` as well. 🤔 >> > >>> >> > >>> Best, >> > >>> Wei >> > >>> >> > >>> On 2025/02/17 19:50:57 Jarek Potiuk wrote: >> > >>>> Hard to say until it's looked at :) >> > >>>> >> > >>>> On Mon, Feb 17, 2025 at 8:45 PM Aritra Basu < >> aritrabasu1...@gmail.com >> > > >> > >>>> wrote: >> > >>>> >> > >>>>> I can take it up, it's mostly just a doc update right? Or are we >> > doing >> > >>> code >> > >>>>> files replacement too? >> > >>>>> -- >> > >>>>> Regards, >> > >>>>> Aritra Basu >> > >>>>> >> > >>>>> On Tue, 18 Feb 2025, 12:55 am Jarek Potiuk, <ja...@potiuk.com> >> > wrote: >> > >>>>> >> > >>>>>> Sounds like another +1000 files big PR is coming :) ? Any >> volunteers >> > >>> to >> > >>>>>> make it ? It's fun. >> > >>>>>> >> > >>>>>> On Mon, Feb 17, 2025 at 6:52 PM Omkar P <droiddev5...@gmail.com> >> > >>> wrote: >> > >>>>>> >> > >>>>>>> +1 for ruff rules :) >> > >>>>>>> >> > >>>>>>> Also would be nice to introduce 'Dag' to replace 'DAG' in the >> > >>> Airflow >> > >>>>>>> docs, in line with the new UI changes and to make the renaming >> > >>>>>>> consistent across user-facing pages. >> > >>>>>>> >> > >>>>>>> Regards, >> > >>>>>>> Omkar >> > >>>>>>> >> > >>>>>>> On Mon, Feb 17, 2025 at 8:12 AM Jarek Potiuk <ja...@potiuk.com> >> > >>> wrote: >> > >>>>>>> >> > >>>>>>>> +1 :) . Maybe we could add a ruff rule for that :) >> > >>>>>>>> >> > >>>>>>>> On Mon, Feb 17, 2025 at 8:27 AM Wei Lee <wei...@apache.org> >> > >>> wrote: >> > >>>>>>>> >> > >>>>>>>>> It seems that our current conclusion is to use "dag" or "Dag" >> > >>>>> instead >> > >>>>>>> of >> > >>>>>>>>> "DAG" whenever possible. Should we replace all "DAG" in the >> > >>>>> codebase >> > >>>>>>> with >> > >>>>>>>>> "dag" or "Dag"? If it's too late for that (which it might be >> > >>> 🤔), >> > >>>>>>> should >> > >>>>>>>> we >> > >>>>>>>>> at least avoid introducing new "DAG" in the following PRs? >> > >>>>>>>>> >> > >>>>>>>>> Best, >> > >>>>>>>>> Wei >> > >>>>>>>>> >> > >>>>>>>>> On 2024/10/23 17:16:55 Brent Bovenzi wrote: >> > >>>>>>>>>> Here's a PR to use "dag" as a word in the new 3.0 UI: >> > >>>>>>>>>> https://github.com/apache/airflow/pull/43325 >> > >>>>>>>>>> Let me know if that's the direction we want to go. >> > >>>>>>>>>> >> > >>>>>>>>>> On Wed, Oct 23, 2024 at 11:06 AM Bishundeo, Rajeshwar >> > >>>>>>>>>> <rbish...@amazon.com.invalid> wrote: >> > >>>>>>>>>> >> > >>>>>>>>>>> I think Brent & Daniel summarized it best, "dag" is >> > >>> synonymous >> > >>>>>> with >> > >>>>>>>>>>> workflows in Airflow through the way we talk and explain >> > >>> what >> > >>>>>>> Airflow >> > >>>>>>>>> is >> > >>>>>>>>>>> all about. Although folks would ask, I don’t ever use the >> > >>>>>>>> mathematical >> > >>>>>>>>>>> definition of DAG. >> > >>>>>>>>>>> It will be challenging and possibly confusing for many >> > >>> users >> > >>>>>> making >> > >>>>>>>>> such a >> > >>>>>>>>>>> change - I would rather direct that energy to appending the >> > >>>>>> Oxford >> > >>>>>>>>>>> definition of "dag" to include a reference to workflows in >> > >>>>>> Airflow. >> > >>>>>>>> __ >> > >>>>>>>>>>> >> > >>>>>>>>>>> -- Rajesh >> > >>>>>>>>>>> >> > >>>>>>>>>>> >> > >>>>>>>>>>> >> > >>>>>>>>>>> >> > >>>>>>>>>>> >> > >>>>>>>>>>> >> > >>>>>>>>>>> On 2024-10-23, 3:37 AM, "Jarek Potiuk" <ja...@potiuk.com >> > >>>>>> <mailto: >> > >>>>>>>>>>> ja...@potiuk.com>> wrote: >> > >>>>>>>>>>> >> > >>>>>>>>>>> >> > >>>>>>>>>>> CAUTION: This email originated from outside of the >> > >>>>> organization. >> > >>>>>> Do >> > >>>>>>>> not >> > >>>>>>>>>>> click links or open attachments unless you can confirm the >> > >>>>> sender >> > >>>>>>> and >> > >>>>>>>>> know >> > >>>>>>>>>>> the content is safe. >> > >>>>>>>>>>> >> > >>>>>>>>>>> >> > >>>>>>>>>>> >> > >>>>>>>>>>> >> > >>>>>>>>>>> >> > >>>>>>>>>>> >> > >>>>>>>>>>> AVERTISSEMENT: Ce courrier électronique provient d’un >> > >>>>> expéditeur >> > >>>>>>>>> externe. >> > >>>>>>>>>>> Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe >> > >>> si >> > >>>>> vous >> > >>>>>>> ne >> > >>>>>>>>> pouvez >> > >>>>>>>>>>> pas confirmer l’identité de l’expéditeur et si vous n’êtes >> > >>> pas >> > >>>>>>>> certain >> > >>>>>>>>> que >> > >>>>>>>>>>> le contenu ne présente aucun risque. >> > >>>>>>>>>>> >> > >>>>>>>>>>> >> > >>>>>>>>>>> >> > >>>>>>>>>>> >> > >>>>>>>>>>> >> > >>>>>>>>>>> >> > >>>>>>>>>>> From Guido's post: >> > >>>>>>>>>>> >> > >>>>>>>>>>> >> > >>>>>>>>>>> "Around this time the renaming seems to have been renamed". >> > >>>>>>>>>>> >> > >>>>>>>>>>> >> > >>>>>>>>>>> Naming is hard. >> > >>>>>>>>>>> >> > >>>>>>>>>>> >> > >>>>>>>>>>> J. >> > >>>>>>>>>>> >> > >>>>>>>>>>> >> > >>>>>>>>>>> >> > >>>>>>>>>>> >> > >>>>>>>>>>> On Wed, Oct 23, 2024 at 7:11 AM Omkar P < >> > >>>>> droiddev5...@gmail.com >> > >>>>>>>>> <mailto: >> > >>>>>>>>>>> droiddev5...@gmail.com>> wrote: >> > >>>>>>>>>>> >> > >>>>>>>>>>> >> > >>>>>>>>>>>> Yes, completely agree with above comments, dag is an >> > >>> Airflow >> > >>>>>> term >> > >>>>>>>> now >> > >>>>>>>>>>>> rather >> > >>>>>>>>>>>> than just a "directed acyclic graph". Believe it or not, >> > >>> I've >> > >>>>>>>> worked >> > >>>>>>>>> with >> > >>>>>>>>>>>> work >> > >>>>>>>>>>>> folks who've been using dags in Airflow for years now >> > >>> and are >> > >>>>>> pro >> > >>>>>>>>> devs >> > >>>>>>>>>>> but >> > >>>>>>>>>>>> have trouble remembering the full form of a DAG! For >> > >>> them, >> > >>>>> dag >> > >>>>>> = >> > >>>>>>>>> Airflow. >> > >>>>>>>>>>>> >> > >>>>>>>>>>>> workflow, pipeline, flow are surely better terms but >> > >>> will be >> > >>>>> a >> > >>>>>>>> major >> > >>>>>>>>>>>> behavior >> > >>>>>>>>>>>> change for the developer community, so we'll need a solid >> > >>>>> plan >> > >>>>>> on >> > >>>>>>>>> how to >> > >>>>>>>>>>>> introduce it, when we do. >> > >>>>>>>>>>>> >> > >>>>>>>>>>>> While we discuss this, I'd like to share about the Great >> > >>>>>> Renaming >> > >>>>>>>> in >> > >>>>>>>>> core >> > >>>>>>>>>>>> Python started by Guido back in 2009. We could probably >> > >>> get >> > >>>>>> some >> > >>>>>>>>>>> learnings >> > >>>>>>>>>>>> from there? Who knows! >> > >>>>>>>>>>>> >> > >>>>>>>>>>>> Guido's blog (2009): >> > >>>>>>>>>>>> >> > >>>>>>>>> >> > >>>>>>> >> > >>>>> >> > >>> >> > >> https://python-history.blogspot.com/2009/03/great-or-grand-renaming.html >> > >>>>>>>>>>> < >> > >>>>>>>>> >> > >>>>>>> >> > >>>>> >> > >>> >> > >> https://python-history.blogspot.com/2009/03/great-or-grand-renaming.html >> > >>>>>>>>> >> > >>>>>>>>>>>> Follow-up discussion (2024): >> > >>>>>>>>>>>> >> > >>>>>> https://discuss.python.org/t/finishing-the-great-renaming/54082 >> > >>>>>>> < >> > >>>>>>>>>>> >> > >>>>> https://discuss.python.org/t/finishing-the-great-renaming/54082> >> > >>>>>>>>>>>> >> > >>>>>>>>>>>> Cheers, >> > >>>>>>>>>>>> Omkar >> > >>>>>>>>>>>> >> > >>>>>>>>>>>> On Wed, Oct 23, 2024 at 7:10 AM Wei Lee < >> > >>> weilee...@gmail.com >> > >>>>>>>>> <mailto: >> > >>>>>>>>>>> weilee...@gmail.com>> wrote: >> > >>>>>>>>>>>> >> > >>>>>>>>>>>>> I think we should probably just accept it as an airflow >> > >>>>> term. >> > >>>>>>> At >> > >>>>>>>>> least, >> > >>>>>>>>>>>>> that’s how I understood it when I first used Airflow. I >> > >>>>> feel >> > >>>>>>>>> renaming >> > >>>>>>>>>>> it >> > >>>>>>>>>>>> at >> > >>>>>>>>>>>>> this stage would require considerable effort from >> > >>>>> maintainers >> > >>>>>>> and >> > >>>>>>>>>>>> existing >> > >>>>>>>>>>>>> users without providing equivalent benefits. >> > >>>>>>>>>>>>> >> > >>>>>>>>>>>>> Best, >> > >>>>>>>>>>>>> Wei >> > >>>>>>>>>>>>> >> > >>>>>>>>>>>>>> On Oct 23, 2024, at 8:01 AM, Kaxil Naik < >> > >>>>>> kaxiln...@gmail.com >> > >>>>>>>>>>> <mailto:kaxiln...@gmail.com>> wrote: >> > >>>>>>>>>>>>>> >> > >>>>>>>>>>>>>> Same agreed with Brent & Daniel -- maybe we re-kindle >> > >>>>> this >> > >>>>>>>>> discussion >> > >>>>>>>>>>>> for >> > >>>>>>>>>>>>>> Airflow 4 :) -- but right now it will cause too much >> > >>>>>>> disruption >> > >>>>>>>>>>>>>> >> > >>>>>>>>>>>>>> On Tue, 22 Oct 2024 at 21:27, Constance Martineau >> > >>>>>>>>>>>>>> <consta...@astronomer.io.inva <mailto: >> > >>>>>>>>> consta...@astronomer.io.inva>lid> >> > >>>>>>>>>>> wrote: >> > >>>>>>>>>>>>>> >> > >>>>>>>>>>>>>>> In my experience, when you ask those with Airflow >> > >>>>>> experience >> > >>>>>>>>> what a >> > >>>>>>>>>>>> dag >> > >>>>>>>>>>>>> is, >> > >>>>>>>>>>>>>>> they'll start talking about workflow attributes - >> > >>> stuff >> > >>>>>> like >> > >>>>>>>>> dags >> > >>>>>>>>>>>> being >> > >>>>>>>>>>>>> a >> > >>>>>>>>>>>>>>> series of steps or tasks with owners. The structure >> > >>>>>> doesn't >> > >>>>>>>>> come up. >> > >>>>>>>>>>>>>>> >> > >>>>>>>>>>>>>>> Echo-ing others, at this point, my vote is to >> > >>> embrace >> > >>>>> the >> > >>>>>>> name >> > >>>>>>>>> and >> > >>>>>>>>>>>>>>> de-emphasize the mathematical structure aspect. >> > >>>>>>>>>>>>>>> >> > >>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 3:47 PM Vikram Koka >> > >>>>>>>>>>>>> <vik...@astronomer.io.inva <mailto: >> > >>>>> vik...@astronomer.io.inva >> > >>>>>>>>> lid> >> > >>>>>>>>>>>>>>> wrote: >> > >>>>>>>>>>>>>>> >> > >>>>>>>>>>>>>>>> It's an interesting discussion and I remember >> > >>>>> struggling >> > >>>>>>> with >> > >>>>>>>>> this >> > >>>>>>>>>>>>> when I >> > >>>>>>>>>>>>>>>> started working with Airflow. >> > >>>>>>>>>>>>>>>> But, I also agree with the viewpoint of it being an >> > >>>>>>>> established >> > >>>>>>>>>>>> concept >> > >>>>>>>>>>>>>>> now >> > >>>>>>>>>>>>>>>> regardless of the origin. >> > >>>>>>>>>>>>>>>> >> > >>>>>>>>>>>>>>>> I am personally leaning towards the perspective >> > >>> best >> > >>>>>>>> expressed >> > >>>>>>>>> by >> > >>>>>>>>>>>>> Daniel >> > >>>>>>>>>>>>>>>> Standish and Brent of using Dag as a word, rather >> > >>> than >> > >>>>>> the >> > >>>>>>>>> computer >> > >>>>>>>>>>>>>>> science >> > >>>>>>>>>>>>>>>> concept. >> > >>>>>>>>>>>>>>>> >> > >>>>>>>>>>>>>>>> Best regards, >> > >>>>>>>>>>>>>>>> Vikram >> > >>>>>>>>>>>>>>>> >> > >>>>>>>>>>>>>>>> >> > >>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 9:46 AM Oliveira, Niko >> > >>>>>>>>>>>>>>> <oniko...@amazon.com.inva <mailto: >> > >>>>>> oniko...@amazon.com.inva >> > >>>>>>>>> lid >> > >>>>>>>>>>>>>>>>> >> > >>>>>>>>>>>>>>>> wrote: >> > >>>>>>>>>>>>>>>> >> > >>>>>>>>>>>>>>>>> I agree with the general sentiment of: You're >> > >>> right >> > >>>>>> Ryan, >> > >>>>>>>> DAG >> > >>>>>>>>>>> isn't >> > >>>>>>>>>>>>>>> great >> > >>>>>>>>>>>>>>>>> and I'd rather workflow, but changing it will >> > >>> cause >> > >>>>> much >> > >>>>>>>> more >> > >>>>>>>>>>>> wreckage >> > >>>>>>>>>>>>>>>> than >> > >>>>>>>>>>>>>>>>> it solves. >> > >>>>>>>>>>>>>>>>> >> > >>>>>>>>>>>>>>>>> Also agree with the idea to just move away from >> > >>>>> defining >> > >>>>>>>> DAG. >> > >>>>>>>>> I >> > >>>>>>>>>>>> think >> > >>>>>>>>>>>>>>>>> we've been naturally doing that as a community >> > >>> for a >> > >>>>>> while >> > >>>>>>>> now >> > >>>>>>>>>>>> anyway, >> > >>>>>>>>>>>>>>> so >> > >>>>>>>>>>>>>>>>> that feels like a natural step. >> > >>>>>>>>>>>>>>>>> >> > >>>>>>>>>>>>>>>>> Cheers, >> > >>>>>>>>>>>>>>>>> Niko >> > >>>>>>>>>>>>>>>>> >> > >>>>>>>>>>>>>>>>> ________________________________ >> > >>>>>>>>>>>>>>>>> From: Ash Berlin-Taylor <a...@apache.org <mailto: >> > >>>>>>>>> a...@apache.org>> >> > >>>>>>>>>>>>>>>>> Sent: Tuesday, October 22, 2024 9:06:39 AM >> > >>>>>>>>>>>>>>>>> To: dev@airflow.apache.org <mailto: >> > >>>>>> dev@airflow.apache.org >> > >>>>>>>> >> > >>>>>>>>>>>>>>>>> Subject: RE: [EXT] Airflow should deprecate the >> > >>> term >> > >>>>>> "DAG" >> > >>>>>>>>> for end >> > >>>>>>>>>>>>>>> users >> > >>>>>>>>>>>>>>>>> >> > >>>>>>>>>>>>>>>>> CAUTION: This email originated from outside of the >> > >>>>>>>>> organization. >> > >>>>>>>>>>> Do >> > >>>>>>>>>>>>> not >> > >>>>>>>>>>>>>>>>> click links or open attachments unless you can >> > >>> confirm >> > >>>>>> the >> > >>>>>>>>> sender >> > >>>>>>>>>>>> and >> > >>>>>>>>>>>>>>>> know >> > >>>>>>>>>>>>>>>>> the content is safe. >> > >>>>>>>>>>>>>>>>> >> > >>>>>>>>>>>>>>>>> >> > >>>>>>>>>>>>>>>>> >> > >>>>>>>>>>>>>>>>> AVERTISSEMENT: Ce courrier électronique provient >> > >>> d’un >> > >>>>>>>>> expéditeur >> > >>>>>>>>>>>>>>> externe. >> > >>>>>>>>>>>>>>>>> Ne cliquez sur aucun lien et n’ouvrez aucune pièce >> > >>>>>> jointe >> > >>>>>>> si >> > >>>>>>>>> vous >> > >>>>>>>>>>> ne >> > >>>>>>>>>>>>>>>> pouvez >> > >>>>>>>>>>>>>>>>> pas confirmer l’identité de l’expéditeur et si >> > >>> vous >> > >>>>>> n’êtes >> > >>>>>>>> pas >> > >>>>>>>>>>>> certain >> > >>>>>>>>>>>>>>>> que >> > >>>>>>>>>>>>>>>>> le contenu ne présente aucun risque. >> > >>>>>>>>>>>>>>>>> >> > >>>>>>>>>>>>>>>>> >> > >>>>>>>>>>>>>>>>> >> > >>>>>>>>>>>>>>>>> Best argument in favour of keeping “dags” as a >> > >>> term — >> > >>>>>>>> getting >> > >>>>>>>>> to >> > >>>>>>>>>>>>> re-use >> > >>>>>>>>>>>>>>>>> puns like https://i.imgflip.com/1xhtwh.jpg < >> > >>>>>>>>>>> https://i.imgflip.com/1xhtwh.jpg> >> > >>>>>>>>>>>>>>>>> >> > >>>>>>>>>>>>>>>>> In all seriousness: I don’t mind either way, both >> > >>>>> sides >> > >>>>>>> have >> > >>>>>>>>> good >> > >>>>>>>>>>>>>>> reasons >> > >>>>>>>>>>>>>>>>> presented. >> > >>>>>>>>>>>>>>>>> >> > >>>>>>>>>>>>>>>>> -a >> > >>>>>>>>>>>>>>>>> >> > >>>>>>>>>>>>>>>>>> On 22 Oct 2024, at 17:03, Daniel Standish >> > >>>>>>>>>>>>>>>>> <daniel.stand...@astronomer.io.inva <mailto: >> > >>>>>>>>>>> daniel.stand...@astronomer.io.inva>LID> wrote: >> > >>>>>>>>>>>>>>>>>> >> > >>>>>>>>>>>>>>>>>> Yeah just say, when asked where the name comes >> > >>> from, >> > >>>>>>> "well, >> > >>>>>>>>> no >> > >>>>>>>>>>> one >> > >>>>>>>>>>>>>>>>> actually >> > >>>>>>>>>>>>>>>>>> knows but..." and then make something up. >> > >>>>>>>>>>>>>>>>>> >> > >>>>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 8:31 AM Jarek Potiuk < >> > >>>>>>>>> ja...@potiuk.com >> > >>>>>>>>>>> <mailto:ja...@potiuk.com>> >> > >>>>>>>>>>>>>>> wrote: >> > >>>>>>>>>>>>>>>>>> >> > >>>>>>>>>>>>>>>>>>> Just to clarify - "directed acyclic graph" is >> > >>> the >> > >>>>>>>>>>> tongue-twister, >> > >>>>>>>>>>>>>>>>>>> >> > >>>>>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 5:29 PM Jarek Potiuk < >> > >>>>>>>>> ja...@potiuk.com >> > >>>>>>>>>>> <mailto:ja...@potiuk.com>> >> > >>>>>>>>>>>>>>>> wrote: >> > >>>>>>>>>>>>>>>>>>> >> > >>>>>>>>>>>>>>>>>>>> I like what both Daniel and Brent wrote. I >> > >>> would >> > >>>>> very >> > >>>>>>>> much >> > >>>>>>>>> want >> > >>>>>>>>>>>> to >> > >>>>>>>>>>>>>>> be >> > >>>>>>>>>>>>>>>>>>> able >> > >>>>>>>>>>>>>>>>>>>> to say just "dag" without explaining it >> > >>> further. >> > >>>>>>>>>>>>>>>>>>>> >> > >>>>>>>>>>>>>>>>>>>> For me every time I explain "DAG" at a talk >> > >>> it's a >> > >>>>>>>>>>>> tongue-twister, >> > >>>>>>>>>>>>>>>> and >> > >>>>>>>>>>>>>>>>> I >> > >>>>>>>>>>>>>>>>>>>> almost stutter on trying to recall how to >> > >>> pronounce >> > >>>>>> it >> > >>>>>>>>>>> properly. >> > >>>>>>>>>>>>>>>>>>>> >> > >>>>>>>>>>>>>>>>>>>> J. >> > >>>>>>>>>>>>>>>>>>>> >> > >>>>>>>>>>>>>>>>>>>> >> > >>>>>>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 5:27 PM Brent Bovenzi >> > >>>>>>>>>>>>>>>>>>> <br...@astronomer.io.inva <mailto: >> > >>>>>>>> br...@astronomer.io.inva >> > >>>>>>>>>> lid> >> > >>>>>>>>>>>>>>>>>>>> wrote: >> > >>>>>>>>>>>>>>>>>>>> >> > >>>>>>>>>>>>>>>>>>>>> I remember we explored renaming "DAG" when >> > >>>>> starting >> > >>>>>> on >> > >>>>>>>>> AIP-38 >> > >>>>>>>>>>> to >> > >>>>>>>>>>>>>>>>>>> modernize >> > >>>>>>>>>>>>>>>>>>>>> the UI. Both "pipeline" or "workflow" are more >> > >>>>>>>>> descriptive of >> > >>>>>>>>>>>> what >> > >>>>>>>>>>>>>>>> one >> > >>>>>>>>>>>>>>>>>>> is >> > >>>>>>>>>>>>>>>>>>>>> actually doing while Directed Acyclic Graph >> > >>> is an >> > >>>>>>>>>>> implementation >> > >>>>>>>>>>>>>>>>> detail. >> > >>>>>>>>>>>>>>>>>>>>> But I agree with Daniel Standish, at this >> > >>> point >> > >>>>>> "DAG" >> > >>>>>>>> has >> > >>>>>>>>>>> become >> > >>>>>>>>>>>>>>>> "dag" >> > >>>>>>>>>>>>>>>>>>> , a >> > >>>>>>>>>>>>>>>>>>>>> word in its own right. >> > >>>>>>>>>>>>>>>>>>>>> >> > >>>>>>>>>>>>>>>>>>>>> Examples for "dag" are abound in community >> > >>>>>> discussion, >> > >>>>>>>>> Airflow >> > >>>>>>>>>>>>>>>> Summit >> > >>>>>>>>>>>>>>>>>>>>> talks, documentation and even in the UI. Let's >> > >>>>>> embrace >> > >>>>>>>>> "dag". >> > >>>>>>>>>>> A >> > >>>>>>>>>>>>>>> user >> > >>>>>>>>>>>>>>>>>>> just >> > >>>>>>>>>>>>>>>>>>>>> needs to learn one new word vs the technical >> > >>>>> concept >> > >>>>>>>>> behind >> > >>>>>>>>>>> that >> > >>>>>>>>>>>>>>>>> word. I >> > >>>>>>>>>>>>>>>>>>>>> think that is much less effort than >> > >>> refactoring so >> > >>>>>>> much >> > >>>>>>>>> code, >> > >>>>>>>>>>>>>>>>>>>>> documentation, blog posts, stack overflow >> > >>>>> questions, >> > >>>>>>>> etc. >> > >>>>>>>>>>>>>>>>>>>>> >> > >>>>>>>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 10:51 AM Daniel >> > >>> Standish >> > >>>>>>>>>>>>>>>>>>>>> <daniel.stand...@astronomer.io.inva <mailto: >> > >>>>>>>>>>> daniel.stand...@astronomer.io.inva>lid> wrote: >> > >>>>>>>>>>>>>>>>>>>>> >> > >>>>>>>>>>>>>>>>>>>>>> I am skeptical. Seems like introducing a lot >> > >>> of >> > >>>>>> pain >> > >>>>>>>> for >> > >>>>>>>>>>>>>>>>> questionable >> > >>>>>>>>>>>>>>>>>>>>>> benefit. But, I am def sympathetic to the >> > >>> idea. I >> > >>>>>>> agree >> > >>>>>>>>> the >> > >>>>>>>>>>>>>>>>>>>>> association >> > >>>>>>>>>>>>>>>>>>>>>> with "directed acyclic graph" is not helpful. >> > >>>>>>>>>>>>>>>>>>>>>> >> > >>>>>>>>>>>>>>>>>>>>>> And along those lines, I offer here some less >> > >>>>>>> invasive >> > >>>>>>>>>>>>>>> mitigations. >> > >>>>>>>>>>>>>>>>>>>>>> >> > >>>>>>>>>>>>>>>>>>>>>> One thing we can do no matter what is to >> > >>>>>> de-emphasize >> > >>>>>>>> the >> > >>>>>>>>>>> math >> > >>>>>>>>>>>>>>> nerd >> > >>>>>>>>>>>>>>>>>>>>> origins >> > >>>>>>>>>>>>>>>>>>>>>> of the name. That is to say, in docs / >> > >>> website / >> > >>>>>> etc, >> > >>>>>>>>> *never >> > >>>>>>>>>>>>>>>> define* >> > >>>>>>>>>>>>>>>>>>>>>> airflow's "dag" concept as a directed acyclic >> > >>>>>> graph. >> > >>>>>>>>> Always >> > >>>>>>>>>>>>>>> define >> > >>>>>>>>>>>>>>>>> it >> > >>>>>>>>>>>>>>>>>>>>> as a >> > >>>>>>>>>>>>>>>>>>>>>> pipeline, collection of tasks, workflow etc. >> > >>>>>>>>>>>>>>>>>>>>>> >> > >>>>>>>>>>>>>>>>>>>>>> The "directed acyclic graph" part of it is >> > >>> like a >> > >>>>>>>>> historical >> > >>>>>>>>>>>>>>>>> footnote, >> > >>>>>>>>>>>>>>>>>>>>> and >> > >>>>>>>>>>>>>>>>>>>>>> we could make one mention of it somewhere >> > >>> hidden. >> > >>>>>>>>>>>>>>>>>>>>>> >> > >>>>>>>>>>>>>>>>>>>>>> We could also start using lowercase in the >> > >>> docs >> > >>>>> in >> > >>>>>>>>> general >> > >>>>>>>>>>> e.g. >> > >>>>>>>>>>>>>>>>>>> writing >> > >>>>>>>>>>>>>>>>>>>>>> "dag" / "dags" instead of writing "DAG" / >> > >>> "DAGs" >> > >>>>>> etc. >> > >>>>>>>> The >> > >>>>>>>>>>>> upper >> > >>>>>>>>>>>>>>>> case >> > >>>>>>>>>>>>>>>>>>>>> part >> > >>>>>>>>>>>>>>>>>>>>>> of it makes it look like an acronym; but >> > >>> "dag" in >> > >>>>>>>>> airlfow is >> > >>>>>>>>>>>> just >> > >>>>>>>>>>>>>>>> an >> > >>>>>>>>>>>>>>>>>>>>>> airflow concept and the association with >> > >>> "DAGs" >> > >>>>> is >> > >>>>>>> not >> > >>>>>>>>> really >> > >>>>>>>>>>>>>>>>>>> unhelpful. >> > >>>>>>>>>>>>>>>>>>>>>> >> > >>>>>>>>>>>>>>>>>>>>>> In other words embrace that "dag" in airflow >> > >>> is >> > >>>>> its >> > >>>>>>> own >> > >>>>>>>>>>> thing, >> > >>>>>>>>>>>> is >> > >>>>>>>>>>>>>>>>>>>>>> *not* strictly >> > >>>>>>>>>>>>>>>>>>>>>> speaking a directed acyclic graph (which >> > >>> nobody >> > >>>>>> knows >> > >>>>>>>>> about >> > >>>>>>>>>>>>>>>> anyway), >> > >>>>>>>>>>>>>>>>>>> and >> > >>>>>>>>>>>>>>>>>>>>>> tell them what it is in simple terms that >> > >>> normal >> > >>>>>>> people >> > >>>>>>>>>>>>>>> understand. >> > >>>>>>>>>>>>>>>>>>>>>> >> > >>>>>>>>>>>>>>>>>>>>>> >> > >>>>>>>>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 7:27 AM Jarek Potiuk >> > >>> < >> > >>>>>>>>>>> ja...@potiuk.com <mailto:ja...@potiuk.com> >> > >>>>>>>>>>>>> >> > >>>>>>>>>>>>>>>>>>> wrote: >> > >>>>>>>>>>>>>>>>>>>>>> >> > >>>>>>>>>>>>>>>>>>>>>>> DAG is so embedded into what we do that it >> > >>> will >> > >>>>> be >> > >>>>>>>>> extremely >> > >>>>>>>>>>>>>>>>>>>>> difficult to >> > >>>>>>>>>>>>>>>>>>>>>>> get rid of it completely. Also I think it >> > >>> will >> > >>>>>> make >> > >>>>>>> a >> > >>>>>>>>> lot of >> > >>>>>>>>>>>>>>>>>>> "google" >> > >>>>>>>>>>>>>>>>>>>>>>> searches and "stack overflow" searches not >> > >>>>> finding >> > >>>>>>> the >> > >>>>>>>>> right >> > >>>>>>>>>>>>>>>>>>> answers. >> > >>>>>>>>>>>>>>>>>>>>>> This >> > >>>>>>>>>>>>>>>>>>>>>>> is one of the strengths of Airflow - >> > >>> besides the >> > >>>>>>>>> community >> > >>>>>>>>>>> and >> > >>>>>>>>>>>>>>>> ideas >> > >>>>>>>>>>>>>>>>>>>>> that >> > >>>>>>>>>>>>>>>>>>>>>>> Bernd mentioned - is the vast number of >> > >>>>> examples, >> > >>>>>>>>> problems >> > >>>>>>>>>>> and >> > >>>>>>>>>>>>>>>>>>>>> solutions >> > >>>>>>>>>>>>>>>>>>>>>>> you can so easily find (and we have to >> > >>> remember >> > >>>>>> that >> > >>>>>>>>> all the >> > >>>>>>>>>>>> AI >> > >>>>>>>>>>>>>>>>>>>>> trained >> > >>>>>>>>>>>>>>>>>>>>>> on >> > >>>>>>>>>>>>>>>>>>>>>>> past data will be also rather poorly >> > >>> matching >> > >>>>>>> queries >> > >>>>>>>> of >> > >>>>>>>>>>>> people. >> > >>>>>>>>>>>>>>>>>>>>>>> >> > >>>>>>>>>>>>>>>>>>>>>>> I am not too attached to DAG. I could easily >> > >>>>>> switch. >> > >>>>>>>>> And if >> > >>>>>>>>>>> we >> > >>>>>>>>>>>>>>> do >> > >>>>>>>>>>>>>>>> - >> > >>>>>>>>>>>>>>>>>>> I >> > >>>>>>>>>>>>>>>>>>>>>>> would be for using workflow or pipeline >> > >>> instead >> > >>>>> of >> > >>>>>>>>> `dag` if >> > >>>>>>>>>>>> not >> > >>>>>>>>>>>>>>>> the >> > >>>>>>>>>>>>>>>>>>>>> above >> > >>>>>>>>>>>>>>>>>>>>>>> reason, but I think I am here with Igor >> > >>> that it >> > >>>>>>> might >> > >>>>>>>>> cause >> > >>>>>>>>>>>> more >> > >>>>>>>>>>>>>>>>>>>>> problems >> > >>>>>>>>>>>>>>>>>>>>>>> than it solves. >> > >>>>>>>>>>>>>>>>>>>>>>> >> > >>>>>>>>>>>>>>>>>>>>>>> But I am not 100% against - if others will >> > >>> think >> > >>>>>>> it's >> > >>>>>>>> a >> > >>>>>>>>> good >> > >>>>>>>>>>>>>>>> idea, I >> > >>>>>>>>>>>>>>>>>>>>> am >> > >>>>>>>>>>>>>>>>>>>>>> ok >> > >>>>>>>>>>>>>>>>>>>>>>> with it. >> > >>>>>>>>>>>>>>>>>>>>>>> >> > >>>>>>>>>>>>>>>>>>>>>>> J, >> > >>>>>>>>>>>>>>>>>>>>>>> >> > >>>>>>>>>>>>>>>>>>>>>>> >> > >>>>>>>>>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 3:12 PM Abhishek >> > >>> Bhakat >> > >>>>>>>>>>>>>>>>>>>>>>> <abhishek.bha...@astronomer.io.inva >> > >>> <mailto: >> > >>>>>>>>>>> abhishek.bha...@astronomer.io.inva>lid> wrote: >> > >>>>>>>>>>>>>>>>>>>>>>> >> > >>>>>>>>>>>>>>>>>>>>>>>> Agreed that the word DAG makes very less >> > >>> sense >> > >>>>> to >> > >>>>>>>>> someone >> > >>>>>>>>>>> new >> > >>>>>>>>>>>>>>> to >> > >>>>>>>>>>>>>>>>>>>>>> workflow >> > >>>>>>>>>>>>>>>>>>>>>>>> orchestration. But it does also show the >> > >>> nature >> > >>>>>> of >> > >>>>>>>>> being >> > >>>>>>>>>>>>>>> acyclic. >> > >>>>>>>>>>>>>>>>>>>>> Sure, >> > >>>>>>>>>>>>>>>>>>>>>>> as >> > >>>>>>>>>>>>>>>>>>>>>>>> Bas mentioned, there are ways to >> > >>> workaround it. >> > >>>>>>>> Still, >> > >>>>>>>>> in >> > >>>>>>>>>>> my >> > >>>>>>>>>>>>>>>>>>>>> opinion, >> > >>>>>>>>>>>>>>>>>>>>>>> there >> > >>>>>>>>>>>>>>>>>>>>>>>> is generally no need for cyclic behavior in >> > >>>>>>> workflow >> > >>>>>>>>>>>>>>>>>>> orchestration. >> > >>>>>>>>>>>>>>>>>>>>>> Most >> > >>>>>>>>>>>>>>>>>>>>>>>> (*if >> > >>>>>>>>>>>>>>>>>>>>>>>> not all*) cases can be in some way can be >> > >>>>> covered >> > >>>>>>>>> using an >> > >>>>>>>>>>>>>>>> acyclic >> > >>>>>>>>>>>>>>>>>>>>>> manner >> > >>>>>>>>>>>>>>>>>>>>>>>> with multiple runs. Hence, the >> > >>> idempotency. So >> > >>>>> I >> > >>>>>>>> would >> > >>>>>>>>> want >> > >>>>>>>>>>>> the >> > >>>>>>>>>>>>>>>>>>>>>> "acyclic" >> > >>>>>>>>>>>>>>>>>>>>>>>> word to stick. >> > >>>>>>>>>>>>>>>>>>>>>>>> >> > >>>>>>>>>>>>>>>>>>>>>>>> Regards, >> > >>>>>>>>>>>>>>>>>>>>>>>> Avi >> > >>>>>>>>>>>>>>>>>>>>>>>> >> > >>>>>>>>>>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 12:41 PM < >> > >>>>>>>>>>> bernd.stroe...@kosakya.de <mailto: >> > >>> bernd.stroe...@kosakya.de>> >> > >>>>>>>>>>>>>>>>>>> wrote: >> > >>>>>>>>>>>>>>>>>>>>>>>> >> > >>>>>>>>>>>>>>>>>>>>>>>>> Brilliant, I am on the way to become an >> > >>>>> Airflow >> > >>>>>>> Fan; >> > >>>>>>>>> so >> > >>>>>>>>>>> many >> > >>>>>>>>>>>>>>> new >> > >>>>>>>>>>>>>>>>>>>>>> ideas. >> > >>>>>>>>>>>>>>>>>>>>>>>>> >> > >>>>>>>>>>>>>>>>>>>>>>>>> The Term DAG is misleading; it should be >> > >>>>>> replaced >> > >>>>>>> by >> > >>>>>>>>> the >> > >>>>>>>>>>>> more >> > >>>>>>>>>>>>>>>>>>>>> general >> > >>>>>>>>>>>>>>>>>>>>>>>> Term >> > >>>>>>>>>>>>>>>>>>>>>>>>> Airflow (Workflow) Graph (AFG) or Airflow >> > >>>>>> (Petri) >> > >>>>>>>> Net >> > >>>>>>>>>>> (AFN) >> > >>>>>>>>>>>>>>>>>>> (maybe >> > >>>>>>>>>>>>>>>>>>>>>>>> without >> > >>>>>>>>>>>>>>>>>>>>>>>>> a direction); >> > >>>>>>>>>>>>>>>>>>>>>>>>> and ... these Graphs should be stored in a >> > >>>>> Graph >> > >>>>>>>>> Database. >> > >>>>>>>>>>>>>>>>>>>>>>>>> >> > >>>>>>>>>>>>>>>>>>>>>>>>> Every Node or Sup-Graph of an Airflow >> > >>> Graph >> > >>>>>> (AFG) >> > >>>>>>>>> might be >> > >>>>>>>>>>>>>>>>>>>>> assigned >> > >>>>>>>>>>>>>>>>>>>>>> to >> > >>>>>>>>>>>>>>>>>>>>>>> an >> > >>>>>>>>>>>>>>>>>>>>>>>>> executable (Python-, Rust-, ... ) member >> > >>> of a >> > >>>>>>>> library. >> > >>>>>>>>>>>>>>>>>>>>>>>>> >> > >>>>>>>>>>>>>>>>>>>>>>>>> A running Graph might have a different >> > >>>>> structure >> > >>>>>>>> than >> > >>>>>>>>> a >> > >>>>>>>>>>>>>>>>>>>>> configuration >> > >>>>>>>>>>>>>>>>>>>>>>>>> Graph. >> > >>>>>>>>>>>>>>>>>>>>>>>>> >> > >>>>>>>>>>>>>>>>>>>>>>>>> Forget that if you think it's bullshit. >> > >>>>>>>>>>>>>>>>>>>>>>>>> >> > >>>>>>>>>>>>>>>>>>>>>>>>> Best Regards >> > >>>>>>>>>>>>>>>>>>>>>>>>> >> > >>>>>>>>>>>>>>>>>>>>>>>>> Bernd Ströhle >> > >>>>>>>>>>>>>>>>>>>>>>>>> M: +49 171 5357916 >> > >>>>>>>>>>>>>>>>>>>>>>>>> E: bernd.stroe...@gmail.com <mailto: >> > >>>>>>>>>>> bernd.stroe...@gmail.com> >> > >>>>>>>>>>>>>>>>>>>>>>>>> >> > >>>>>>>>>>>>>>>>>>>>>>>>> >> > >>>>>>>>>>>>>>>>>>>>>>>>> -----Original Message----- >> > >>>>>>>>>>>>>>>>>>>>>>>>> From: Igor Kholopov >> > >>> <ikholo...@google.com.inva >> > >>>>>>>>> <mailto: >> > >>>>>>>>>>> ikholo...@google.com.inva>LID> >> > >>>>>>>>>>>>>>>>>>>>>>>>> Sent: Tuesday, October 22, 2024 12:02 PM >> > >>>>>>>>>>>>>>>>>>>>>>>>> To: dev@airflow.apache.org <mailto: >> > >>>>>>>>> dev@airflow.apache.org >> > >>>>>>>>>>>> >> > >>>>>>>>>>>>>>>>>>>>>>>>> Subject: Re: Airflow should deprecate the >> > >>> term >> > >>>>>>> "DAG" >> > >>>>>>>>> for >> > >>>>>>>>>>> end >> > >>>>>>>>>>>>>>>>>>> users >> > >>>>>>>>>>>>>>>>>>>>>>>>> >> > >>>>>>>>>>>>>>>>>>>>>>>>> Even though the term "DAG" is clearly >> > >>>>>> suboptimal, >> > >>>>>>> it >> > >>>>>>>>> is >> > >>>>>>>>>>> part >> > >>>>>>>>>>>>>>> of >> > >>>>>>>>>>>>>>>>>>>>>> Airflow >> > >>>>>>>>>>>>>>>>>>>>>>>>> DAG definition interface at so many >> > >>> levels, >> > >>>>> that >> > >>>>>>> any >> > >>>>>>>>>>> attempt >> > >>>>>>>>>>>>>>> to >> > >>>>>>>>>>>>>>>>>>>>>> change >> > >>>>>>>>>>>>>>>>>>>>>>> it >> > >>>>>>>>>>>>>>>>>>>>>>>>> will only introduce more chaos, not >> > >>> reduce it. >> > >>>>>> The >> > >>>>>>>>> only >> > >>>>>>>>>>>> thing >> > >>>>>>>>>>>>>>>>>>>>> that is >> > >>>>>>>>>>>>>>>>>>>>>>>> worse >> > >>>>>>>>>>>>>>>>>>>>>>>>> than a poorly chosen name in the code is >> > >>> when >> > >>>>>>> there >> > >>>>>>>>> are >> > >>>>>>>>>>> two >> > >>>>>>>>>>>>>>> ways >> > >>>>>>>>>>>>>>>>>>>>> to >> > >>>>>>>>>>>>>>>>>>>>>>>> define >> > >>>>>>>>>>>>>>>>>>>>>>>>> the same thing. Countless articles and >> > >>>>> tutorials >> > >>>>>>>> will >> > >>>>>>>>>>>> suddenly >> > >>>>>>>>>>>>>>>>>>>>> become >> > >>>>>>>>>>>>>>>>>>>>>>>>> confusing as they all refer to workflows >> > >>> as >> > >>>>>>> "DAG"s. >> > >>>>>>>>>>>>>>>>>>>>>>>>> >> > >>>>>>>>>>>>>>>>>>>>>>>>> We are already at risk of scaring the >> > >>> users >> > >>>>> away >> > >>>>>>>> with >> > >>>>>>>>> a >> > >>>>>>>>>>>> number >> > >>>>>>>>>>>>>>>>>>> of >> > >>>>>>>>>>>>>>>>>>>>>>>> breaking >> > >>>>>>>>>>>>>>>>>>>>>>>>> changes in Airflow 3, promising even more >> > >>>>>> breaking >> > >>>>>>>>> changes >> > >>>>>>>>>>>> for >> > >>>>>>>>>>>>>>>>>>> the >> > >>>>>>>>>>>>>>>>>>>>>> most >> > >>>>>>>>>>>>>>>>>>>>>>>>> basic things is not something that people >> > >>> are >> > >>>>>>>> looking >> > >>>>>>>>> for. >> > >>>>>>>>>>>>>>>>>>>>> Attempting >> > >>>>>>>>>>>>>>>>>>>>>>> to >> > >>>>>>>>>>>>>>>>>>>>>>>>> change the fundamental terms will be >> > >>>>> interpreted >> > >>>>>>> as >> > >>>>>>>> an >> > >>>>>>>>>>> even >> > >>>>>>>>>>>>>>>>>>>>> stronger >> > >>>>>>>>>>>>>>>>>>>>>>>> signal >> > >>>>>>>>>>>>>>>>>>>>>>>>> of project immaturity. >> > >>>>>>>>>>>>>>>>>>>>>>>>> >> > >>>>>>>>>>>>>>>>>>>>>>>>> Given that, I oppose the idea of changing >> > >>> the >> > >>>>>> term >> > >>>>>>>> in >> > >>>>>>>>> the >> > >>>>>>>>>>>> long >> > >>>>>>>>>>>>>>>>>>>>> run. I >> > >>>>>>>>>>>>>>>>>>>>>>>> even >> > >>>>>>>>>>>>>>>>>>>>>>>>> stricter oppose the idea of deprecating >> > >>> it in >> > >>>>>> the >> > >>>>>>>> DAG >> > >>>>>>>>>>>>>>> definition >> > >>>>>>>>>>>>>>>>>>>>>>>> interface. >> > >>>>>>>>>>>>>>>>>>>>>>>>> We better put our time and efforts in >> > >>> other >> > >>>>>> places >> > >>>>>>>> in >> > >>>>>>>>>>>> Airflow, >> > >>>>>>>>>>>>>>>>>>> of >> > >>>>>>>>>>>>>>>>>>>>>> which >> > >>>>>>>>>>>>>>>>>>>>>>>>> there are plenty. >> > >>>>>>>>>>>>>>>>>>>>>>>>> >> > >>>>>>>>>>>>>>>>>>>>>>>>> Kind regards, >> > >>>>>>>>>>>>>>>>>>>>>>>>> Igor >> > >>>>>>>>>>>>>>>>>>>>>>>>> >> > >>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 10:36 AM Bas >> > >>> Harenslak >> > >>>>>>>>>>>>>>>>>>>>>>> <b...@astronomer.io.inva <mailto: >> > >>>>>>>> b...@astronomer.io.inva >> > >>>>>>>>>> lid >> > >>>>>>>>>>>>>>>>>>>>>>>>> >> > >>>>>>>>>>>>>>>>>>>>>>>>> wrote: >> > >>>>>>>>>>>>>>>>>>>>>>>>> >> > >>>>>>>>>>>>>>>>>>>>>>>>>> Couple of thoughts: >> > >>>>>>>>>>>>>>>>>>>>>>>>>> >> > >>>>>>>>>>>>>>>>>>>>>>>>>> 1. The boundaries/properties of “DAG” >> > >>> have >> > >>>>>>> already >> > >>>>>>>>> faded >> > >>>>>>>>>>>> over >> > >>>>>>>>>>>>>>>>>>>>> time, >> > >>>>>>>>>>>>>>>>>>>>>>>>>> for example there are now several ways to >> > >>>>>> create >> > >>>>>>>>> cyclic >> > >>>>>>>>>>>>>>>>>>> graphs, >> > >>>>>>>>>>>>>>>>>>>>>> e.g. >> > >>>>>>>>>>>>>>>>>>>>>>>>>> using the @continuous schedule. I imagine >> > >>>>> these >> > >>>>>>>>>>> properties >> > >>>>>>>>>>>>>>>>>>>>>> vanishing >> > >>>>>>>>>>>>>>>>>>>>>>>>>> even more in the future, so from that >> > >>>>>>> perspective I >> > >>>>>>>>>>> support >> > >>>>>>>>>>>>>>>>>>>>>> changing >> > >>>>>>>>>>>>>>>>>>>>>>>>>> “DAG" to a more generic name. >> > >>>>>>>>>>>>>>>>>>>>>>>>>> >> > >>>>>>>>>>>>>>>>>>>>>>>>>> 2. How other orchestration frameworks do >> > >>>>>> naming: >> > >>>>>>>>>>>>>>>>>>>>>>>>>> Dagster: pipeline >> > >>>>>>>>>>>>>>>>>>>>>>>>>> Prefect: flow >> > >>>>>>>>>>>>>>>>>>>>>>>>>> Flyte: workflow >> > >>>>>>>>>>>>>>>>>>>>>>>>>> Temporal: workflow >> > >>>>>>>>>>>>>>>>>>>>>>>>>> Kestra: flow >> > >>>>>>>>>>>>>>>>>>>>>>>>>> >> > >>>>>>>>>>>>>>>>>>>>>>>>>> I think “workflow” is the most fitting >> > >>> name. >> > >>>>>>>>>>>>>>>>>>>>>>>>>> >> > >>>>>>>>>>>>>>>>>>>>>>>>>> 3. Given the large impact of this >> > >>> change, I >> > >>>>>>> suggest >> > >>>>>>>>>>>> defining >> > >>>>>>>>>>>>>>> a >> > >>>>>>>>>>>>>>>>>>>>>> clear >> > >>>>>>>>>>>>>>>>>>>>>>>>>> path forward. Would we first introduce >> > >>> the >> > >>>>>>>>> deprecation in >> > >>>>>>>>>>>>>>>>>>>>> Airflow >> > >>>>>>>>>>>>>>>>>>>>>> 3, >> > >>>>>>>>>>>>>>>>>>>>>>>>>> and remove “DAG” in Airflow 4? >> > >>>>>>>>>>>>>>>>>>>>>>>>>> >> > >>>>>>>>>>>>>>>>>>>>>>>>>> Bas >> > >>>>>>>>>>>>>>>>>>>>>>>>>> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>> On 22 Oct 2024, at 09:22, Neil < >> > >>>>>>>> neil4r...@gmail.com >> > >>>>>>>>>>> <mailto:neil4r...@gmail.com>> >> > >>>>>>>>>>>> wrote: >> > >>>>>>>>>>>>>>>>>>>>>>>>>>> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>> I don't see a problem with the term DAG, >> > >>>>>>>> especially >> > >>>>>>>>> when >> > >>>>>>>>>>>>>>>>>>> most >> > >>>>>>>>>>>>>>>>>>>>>> other >> > >>>>>>>>>>>>>>>>>>>>>>>>>>> platforms embrace the term >> > >>> wholeheartedly. >> > >>>>>>>>>>>>>>>>>>>>>>>>>>> I don't see anything intimidating or >> > >>>>> confusing >> > >>>>>>>>> about it >> > >>>>>>>>>>> at >> > >>>>>>>>>>>>>>>>>>>>> all, >> > >>>>>>>>>>>>>>>>>>>>>>>>>>> changing the term though would be fairly >> > >>>>>>> confusing >> > >>>>>>>>> to >> > >>>>>>>>>>> most >> > >>>>>>>>>>>>>>>>>>>>> users >> > >>>>>>>>>>>>>>>>>>>>>>> who >> > >>>>>>>>>>>>>>>>>>>>>>>>>>> have been >> > >>>>>>>>>>>>>>>>>>>>>>>>>> using >> > >>>>>>>>>>>>>>>>>>>>>>>>>>> the term for years. >> > >>>>>>>>>>>>>>>>>>>>>>>>>>> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 1:18 AM Tzu-ping >> > >>>>> Chung >> > >>>>>>>>>>>>>>>>>>>>>>>>>>> <t...@astronomer.io.inva <mailto: >> > >>>>>>>>> t...@astronomer.io.inva >> > >>>>>>>>>>>> lid >> > >>>>>>>>>>>>>>>>>>>>>>>>>>> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>> wrote: >> > >>>>>>>>>>>>>>>>>>>>>>>>>>> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> I totally agree with doing away with >> > >>> the >> > >>>>> term >> > >>>>>>>> DAG. >> > >>>>>>>>> The >> > >>>>>>>>>>>> only >> > >>>>>>>>>>>>>>>>>>>>>>> problem >> > >>>>>>>>>>>>>>>>>>>>>>>>>> (aside >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> from actually telling people—including >> > >>>>>>> myself—to >> > >>>>>>>>> stop >> > >>>>>>>>>>>> using >> > >>>>>>>>>>>>>>>>>>>>> the >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> term) >> > >>>>>>>>>>>>>>>>>>>>>>>>>> is to >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> come up with a reasonable alternative. >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> I can’t recall who, but someone >> > >>> mentioned >> > >>>>>>>>> “workflow” is >> > >>>>>>>>>>>> not >> > >>>>>>>>>>>>>>>>>>>>> very >> > >>>>>>>>>>>>>>>>>>>>>>>>>> accurate >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> for Airflow. The term “definition” was >> > >>>>>>> proposed, >> > >>>>>>>>> but >> > >>>>>>>>>>>> it’s a >> > >>>>>>>>>>>>>>>>>>>>> bit >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> broad; I tried to use it in a few >> > >>> places >> > >>>>> and >> > >>>>>>> kept >> > >>>>>>>>>>> finding >> > >>>>>>>>>>>>>>>>>>>>> myself >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> doubting “what definition?” and >> > >>> wanting to >> > >>>>>>>> clarify >> > >>>>>>>>> “DAG >> > >>>>>>>>>>>>>>>>>>>>>>> definition” >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> (defeating the purpose). >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> TP >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 22 Oct 2024, at 13:07, Jens >> > >>> Scheffler >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> <j_scheff...@gmx.de.inva <mailto: >> > >>>>>>>>>>> j_scheff...@gmx.de.inva>LID> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote: >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Ryan, >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for posting. I share the >> > >>> exactly >> > >>>>> same >> > >>>>>>>>>>>> observation, >> > >>>>>>>>>>>>>>>>>>>>> had a >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> short >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> laight because the DAG question is >> > >>> always >> > >>>>> an >> > >>>>>>>>>>> introduction >> > >>>>>>>>>>>>>>>>>>> if >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> someone >> > >>>>>>>>>>>>>>>>>>>>>>>>>> joins >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> the party. I think a global renaming >> > >>> makes >> > >>>>>>> sense. >> > >>>>>>>>>>>>>>>>>>> Especially >> > >>>>>>>>>>>>>>>>>>>>>> when >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> we >> > >>>>>>>>>>>>>>>>>>>>>>>>>> also >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> rename Dataset to Asset this is also a >> > >>>>>>> reasonable >> > >>>>>>>>> step. >> > >>>>>>>>>>>>>>>>>>>>> Concepts >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> still >> > >>>>>>>>>>>>>>>>>>>>>>>>>> can >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> stay the same. >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> So I hope I don‘t need to join hiding >> > >>>>> below >> > >>>>>>> the >> > >>>>>>>>> desk >> > >>>>>>>>>>>> with >> > >>>>>>>>>>>>>>>>>>>>> you >> > >>>>>>>>>>>>>>>>>>>>>> and >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> +1 >> > >>>>>>>>>>>>>>>>>>>>>>>>>> for >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> raising the discussion. >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Technically we can still think if we >> > >>> keep >> > >>>>>>>> details >> > >>>>>>>>> of >> > >>>>>>>>>>>>>>>>>>> python >> > >>>>>>>>>>>>>>>>>>>>>> names >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> the >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> same because the execution is still a >> > >>> DAG… >> > >>>>>> but >> > >>>>>>>> user >> > >>>>>>>>>>>> facing >> > >>>>>>>>>>>>>>>>>>> it >> > >>>>>>>>>>>>>>>>>>>>>> is a >> > >>>>>>>>>>>>>>>>>>>>>>>>>> workflow. >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jens >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Sent from my Smartphone >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 21. Oct 2024, at 23:56, Ryan >> > >>> Hatter < >> > >>>>>>>>>>>>>>>>>>>>>>> ryan.hat...@astronomer.io <mailto: >> > >>>>>>>>> ryan.hat...@astronomer.io >> > >>>>>>>>>>>> >> > >>>>>>>>>>>>>>>>>>>>>>>>>> .invalid> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote: >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Everyone please sheathe your >> > >>> swords... at >> > >>>>>>> least >> > >>>>>>>>> for >> > >>>>>>>>>>>> now. >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The term "DAG" has very little >> > >>> meaning to >> > >>>>>>>> Airflow >> > >>>>>>>>>>>> users. >> > >>>>>>>>>>>>>>>>>>>>>> Indeed, >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> it >> > >>>>>>>>>>>>>>>>>>>>>>>>>> has >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> little meaning outside of some >> > >>>>>> mathematicians >> > >>>>>>>> and >> > >>>>>>>>>>>>>>>>>>> software >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> engineers >> > >>>>>>>>>>>>>>>>>>>>>>>>>> for >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> whom the properties of a DAG actually >> > >>>>>> matter. >> > >>>>>>>> For >> > >>>>>>>>>>>> someone >> > >>>>>>>>>>>>>>>>>>>>> new >> > >>>>>>>>>>>>>>>>>>>>>> to >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> data engineering or workflow >> > >>>>> orchestration, >> > >>>>>>> one >> > >>>>>>>>> of >> > >>>>>>>>>>> the >> > >>>>>>>>>>>>>>>>>>>>> first >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> questions they >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> will >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> likely have is, "what on earth is a >> > >>> DAG?" >> > >>>>>> The >> > >>>>>>>>> answer >> > >>>>>>>>>>> is >> > >>>>>>>>>>>>>>>>>>>>> almost >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> always, "It's a directed acyclic >> > >>> graph. >> > >>>>> You >> > >>>>>>>> don't >> > >>>>>>>>>>> need >> > >>>>>>>>>>>> to >> > >>>>>>>>>>>>>>>>>>>>>> worry >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> about what >> > >>>>>>>>>>>>>>>>>>>>>>>>>> that >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> means; it's just a term for your >> > >>>>> workflow." >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The term "DAG" is problematic for at >> > >>>>> least >> > >>>>>> a >> > >>>>>>>>> couple >> > >>>>>>>>>>>>>>>>>>>>> important >> > >>>>>>>>>>>>>>>>>>>>>>>>> reasons: >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> *Complexity for New Users*: As >> > >>> mentioned >> > >>>>>>> above, >> > >>>>>>>>> "DAG" >> > >>>>>>>>>>>> is >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> unnecessarily intimidating and >> > >>> confusing. >> > >>>>>> We >> > >>>>>>>> want >> > >>>>>>>>>>>> Airflow >> > >>>>>>>>>>>>>>>>>>>>> to >> > >>>>>>>>>>>>>>>>>>>>>> be >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> approachable, and >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> using >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> technical jargon like "DAG" right >> > >>> off the >> > >>>>>> bat >> > >>>>>>>>> creates >> > >>>>>>>>>>>> an >> > >>>>>>>>>>>>>>>>>>>>>> initial >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> barrier to >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> understanding. >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> *Disconnect Between DAG and Workflow >> > >>>>>>> Concepts*: >> > >>>>>>>>> The >> > >>>>>>>>>>> DAG >> > >>>>>>>>>>>>>>>>>>> is >> > >>>>>>>>>>>>>>>>>>>>>> just >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> one component of an Airflow >> > >>> workflow. The >> > >>>>>>>>> workflow >> > >>>>>>>>>>>>>>>>>>> includes >> > >>>>>>>>>>>>>>>>>>>>>> its >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> schedule, retries, timeouts, a dozen >> > >>>>> other >> > >>>>>>>>>>> parameters, >> > >>>>>>>>>>>>>>>>>>> and >> > >>>>>>>>>>>>>>>>>>>>>> other >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> metadata that >> > >>>>>>>>>>>>>>>>>>>>>>>>>> the >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> DAG component doesn’t account for. >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Consider the following from the >> > >>> Airflow >> > >>>>>>>> homepage >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <https://airflow.apache.org/> < >> > >>>>>>>>>>> https://airflow.apache.org/>>. >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Apache Airflow® is a platform >> > >>> created by >> > >>>>>> the >> > >>>>>>>>>>> community >> > >>>>>>>>>>>> to >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> programmatically >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> author, schedule and monitor >> > >>> workflows. >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Then, if we look at the "What is >> > >>>>> Airflow?" >> > >>>>>>> docs >> > >>>>>>>>> page >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> < >> > >>>>>>>>>>>>>>>>>>>>>>> >> > >>>>>>>>>>>>>>> >> > >>>>>>>>> >> > >>> https://airflow.apache.org/docs/apache-airflow/stable/index.html < >> > >>>>>>>>>>> >> > >>>>> https://airflow.apache.org/docs/apache-airflow/stable/index.html >> > >>>>>>> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> , >> > >>>>>>>>>>>>>>>>>>>>>>>>>> we >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> can >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> see that the docs explain what >> > >>> Airflow is >> > >>>>>>>> without >> > >>>>>>>>>>> using >> > >>>>>>>>>>>>>>>>>>>>> "DAG." >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> It's >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> only in >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the *workflow* Python code that the >> > >>> term >> > >>>>> is >> > >>>>>>>>>>> introduced >> > >>>>>>>>>>>>>>>>>>> out >> > >>>>>>>>>>>>>>>>>>>>> of >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nowhere >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> as a >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> comment that awkwardly tries to >> > >>> explain >> > >>>>> it: >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> # A DAG represents a workflow, a >> > >>>>> collection >> > >>>>>>> of >> > >>>>>>>>> tasks >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> It makes sense to not refer to DAGs >> > >>> in >> > >>>>>> these >> > >>>>>>>>>>>>>>>>>>> introductions >> > >>>>>>>>>>>>>>>>>>>>> to >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Airflow, because *Airflow doesn't >> > >>>>>> orchestrate >> > >>>>>>>>> DAGs; >> > >>>>>>>>>>> it >> > >>>>>>>>>>>>>>>>>>>>>>>> orchestrates >> > >>>>>>>>>>>>>>>>>>>>>>>>> workflows*. >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> The >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> DAG is the model that, for reasons >> > >>>>>> irrelevant >> > >>>>>>>> to >> > >>>>>>>>>>> almost >> > >>>>>>>>>>>>>>>>>>>>> every >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> user, workflows must adhere to. >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> So, I propose at least adding an >> > >>> alias >> > >>>>> for >> > >>>>>>> the >> > >>>>>>>>> term >> > >>>>>>>>>>>> "DAG" >> > >>>>>>>>>>>>>>>>>>>>> and >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> updating documentation to replace >> > >>> "DAG" >> > >>>>>> with >> > >>>>>>>>>>>> "workflow". >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> For example, instead of... >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> @dag( >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> schedule="@daily", >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ... >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> dagrun_timeout=timedelta(hours=1) >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ) >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Users could do... >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> @workflow( >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> schedule="@daily", >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ... >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> run_timeout=timedelta(hours=1) >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ) >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> And with that... I will start running >> > >>>>> away. >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> > >>>>>>>>>>>>>>>>>>>>>>> >> > >>>>>>>>>>>>>>> >> > >>>>>>>>> >> > >>> ------------------------------------------------------------------ >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> --- To unsubscribe, e-mail: >> > >>>>>>>>>>>>>>>>>>>>> dev-unsubscr...@airflow.apache.org <mailto: >> > >>>>>>>>>>> dev-unsubscr...@airflow.apache.org> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> For additional commands, e-mail: >> > >>>>>>>>>>>>>>>>>>>>> dev-h...@airflow.apache.org <mailto: >> > >>>>>>>>>>> dev-h...@airflow.apache.org> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> >> > >>>>>>>>>>>>>>>>>>>>> >> > > >> > > >> > > --------------------------------------------------------------------- >> > > To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org >> > > For additional commands, e-mail: dev-h...@airflow.apache.org >> > > >> > >> > >> > >> > --------------------------------------------------------------------- >> > To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org >> > For additional commands, e-mail: dev-h...@airflow.apache.org >> > >> > >> >
