> -0.5 > Even though the new UI prefers "Dag", I don't think that means "DAG" is incorrect.
I think we have a bunch of related things discussed here - I am not exactly sure what the -0.5 in this context means Brent. Would it be possible to clarify ? On Tue, Feb 18, 2025 at 6:01 PM Ambika Garg <ambikagarg1...@gmail.com> wrote: > +1 would vote for > Airflow's *D*ynamic *A*ction *G*raph (DAG) > > *Dynamic*: Reflects the capability of Airflow to handle workflows that > aren’t strictly acyclic, but can adapt to changes, iterations and loops > within broader context of workflow. > > *Action*: Emphasizes that airflow is fundamentally about orchestrating > actions or tasks. Whether its data processing, model training or any other > form of computation. It also conveys operational aspect of what Airflow > does. > > *Graph*: Maintains the core concept that workflows in Airflow are > visualized and managed as graphs. This term keeps the idea of dependencies > and relationships between tasks intact. > > Ambika. > > On Tue, Feb 18, 2025 at 11:46 AM Brent Bovenzi <br...@astronomer.io.invalid > > > wrote: > > > -0.5 > > > > Even though the new UI prefers "Dag", I don't think that means "DAG" is > > incorrect. > > > > On Tue, Feb 18, 2025 at 11:36 AM Jarek Potiuk <ja...@potiuk.com> wrote: > > > > > I love "Dynamic Airflow Graph" +10 to that Constance. > > > > > > And yes. I fully agree with your "somewhat" acyclic statement. I saw > > people > > > mentioning "event-driven" workflows to better describe the AI/ML, but > > that > > > for me is a synonym of "completely unmanageable workflows" (similar to > > > micro-services) - what I was alluding to was exactly this - something > > like > > > curren DAG with parts of it being potentially cyclic - so we are > > completely > > > aligned here :). And that's what make it "not DAG - in mathematical > > sense" > > > - even more reason to depart from it, > > > > > > J. > > > > > > On Tue, Feb 18, 2025 at 5:23 PM Constance Martineau > > > <consta...@astronomer.io.invalid> wrote: > > > > > > > Hey all, > > > > > > > > I appreciate the discussion about how we position the concept of a > DAG > > in > > > > Airflow. While I agree with updating documentation to de-emphasize > the > > > > strict mathematical definition, I do not think we should change the > > > public > > > > interface to use "dag" instead of "DAG" everywhere. It adds > > > > unnecessary churn and breaks consistency across the ecosystem. > > > > > > > > Regarding the broader discussion on whether DAGs in Airflow should > > remain > > > > acyclic: I acknowledge that some workflows, particularly in ML and > AI, > > > > require cyclical behaviour (example: iterative refinement until a > > > condition > > > > is met). However, I think it's important to recognize that even in > > those > > > > cases, the workflow itself still has an overall start and end. It's > not > > > an > > > > unbounded loop but rather a controlled execution process that happens > > to > > > > have cycles *within* it. > > > > > > > > That being said, I do see value in reinterpreting "DAG" to better > align > > > > with how Airflow is actually used, and love shifting away from > > "Directed > > > > Acyclic Graph" in favour of something more intuitive. However "Data > > > Airflow > > > > Graph" makes it sound like we're defining Airflow itself rather than > > > > describing the structure of workflows within it. > > > > > > > > If we want a new meaning for DAG that resonates with our base while > > > keeping > > > > Airflow's identity clear, I am proposing three alternatives: > > > > > > > > > > > > 1. *Dynamic Airflow Graph*: Ties directly to Airflow while > > emphasizing > > > > flexibility > > > > 2. *Distributed Automation Graph*: Highlights execution across > > systems > > > > and automations without tying it strictly to data > > > > 3. *Directed Activity Graph*: Retains the useful meaning of > > "Directed" > > > > while shifting away from strict acyclic constraints. (My personal > > > > favourite) > > > > > > > > I think any shift in terminology should be driven by what best > > > communicates > > > > Airflow's role workflow automation, rather than just distancing > > ourselves > > > > from the mathematical definition. > > > > > > > > Curious to hear more thoughts! > > > > > > > > On Tue, Feb 18, 2025 at 10:40 AM Jarek Potiuk <ja...@potiuk.com> > > wrote: > > > > > > > > > Sorry for the few typos ... My slight dyslexia did not help and > > > > autocorrect > > > > > did not help either this time : dept -> debt, routed -> rooted and > a > > > few > > > > > other small typos :) . > > > > > > > > > > On Tue, Feb 18, 2025 at 4:28 PM Jarek Potiuk <ja...@potiuk.com> > > wrote: > > > > > > > > > > > First of all - yes, agree with Lee and Ash, I think we do not > need > > to > > > > > > change the public interface, I'd also be -0 on that, but docs, > > > internal > > > > > > strings, yes we could. > > > > > > > > > > > > But .... After thinking a bit and looking at the discussion here, > > > where > > > > > > people have concerns, I have an alternative take. And I know this > > > might > > > > > be > > > > > > controversial, and some people might find it confusing, but why > > don't > > > > we > > > > > > come up with a new acronym for DAG, one that will be more airflow > > and > > > > > less > > > > > > "mathematics/algorithm". And I have, I think a good reason for > it, > > > and > > > > I > > > > > > even have a proposal. Bear with me, > > > > > > > > > > > > DAG => *Data Airflow Graph* > > > > > > > > > > > > As many of us mentioned, DAG and Airflow are almost 100% 1-1 now > > > (with > > > > > > exception of some niche products - you know which one I speak > > about - > > > > > which > > > > > > are unfortunate enough to use the "Dag" in their name - I will > > > explain > > > > > why > > > > > > this is unfortunate, IMHO). > > > > > > > > > > > > We are lucky to have Airflow for "A" in the acronym and we can > make > > > it > > > > > > even more "synonymous" -> Airflow <> DAG with (A) being Airflow. > > > > > > > > > > > > Now, why I think "Directed Acyclic Graph" is unfortunate. > > > > > > > > > > > > Not only because "Directed" and "Acyclic" are mostly not in the > > > > > vocabulary > > > > > > of most of our users - as Ryan, starting this thread explained > > yes, I > > > > > agree > > > > > > with it, but also I think there is a more important reason - I > > > simply > > > > > > believe in the future our DAGs might not (and likely will not) be > > > > > "acyclic". > > > > > > > > > > > > I know for some it might be herezy and we have not discussed any > of > > > > that > > > > > > yet, but more and more I keep on hearing that Machine Learning > > > > Workflows > > > > > of > > > > > > the future are often somewhat cyclic. In our case there is one > type > > > of > > > > > > repeat-in-cycle behaviour - we have "task retry on failure" - but > > > this > > > > is > > > > > > only on failure and it basically mens "idempotent" retry - we > want > > to > > > > > redo > > > > > > something and get single result out of that and basically discard > > the > > > > > > previous result. But machine learning/AI workflows are > different. I > > > > had a > > > > > > lot of conversations recently (I even co-organized "AI low-level > > > > > > engineering and hacking" devroom to learn more about those) - > and I > > > > keep > > > > > on > > > > > > hearing that "Repeat a sequence of things several times and > refine" > > > is > > > > > > precisely what AI / ML workflows will need - especially with > > Agentic > > > > > > workflows - often we have cooperating agents which **could** be > > part > > > of > > > > > the > > > > > > same DAG, but currently in Airflow where our DAGs are "Acyclic" > we > > > can > > > > > > only express "repeat several time and refine" as a single task > that > > > > will > > > > > do > > > > > > such refining. If we would like to split such an "agent" doing > > their > > > > own > > > > > > work in "cycles" and communicating with other "agents" doing the > > > same - > > > > > > represented as a multi-task-entity, we have no way to express it > > > using > > > > > our > > > > > > current "acyclic" semantics. > > > > > > > > > > > > But - other than our implementation makes that assumption > > currently - > > > > > > there is nothing fundamental with Airflow being able to run such > > > > > "partially > > > > > > cyclic workflows". We have all the pieces, we would have to > change > > > > > > dependency calculation to allow for cycles, update UI to handle > > > cycles > > > > > > representation - and we could in theory quite seamlessly, support > > > > > "cyclic" > > > > > > workflows. > > > > > > > > > > > > I know, I know, that's the first time we hear it and that's huge > > > > change - > > > > > > of course, I do not want to start discussing any details of it, > not > > > > > before > > > > > > Airflow 3.0 is out - but I think we should be open that at some > > point > > > > of > > > > > > time the "Acyclic" property is going to go away, And my personal > > > > > intuition > > > > > > tells me that a year from now we will have acyclic workflows > > > (assuming > > > > > that > > > > > > all the foundational work we do on Airflow 3 will really help in > > > > > decreasing > > > > > > all the technical dept, and will open our minds to new ideas). > > > > > > > > > > > > That's why I think - the sooner we get rid of "Directed Acyclic" > - > > > the > > > > > > better. that will - even for us, maintainer free us from being > > routed > > > > in > > > > > > the acyclic nature of Airflow. > > > > > > > > > > > > How about we change to "Data Airflow Graph" (or any other good > > > acronym > > > > > > which has (A)irflow instead of (A)cyclic). The name is just a > > > proposal. > > > > > > > > > > > > Now.. I go hide somewhere as I believe that might cause a > > "flamewar" > > > > :), > > > > > > unless on request of Ryan you already sheathed your swords in the > > > > > > discussion (which seems everyone did). > > > > > > > > > > > > But honestly - I would really love what you think of "let's just > > > break > > > > > > with Acyclic NOW" and shape our future by using a different > > acronym. > > > > > > > > > > > > J. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Feb 18, 2025 at 10:54 AM Ephraim Anierobi < > > > > > > ephraimanier...@gmail.com> wrote: > > > > > > > > > > > >> When I started contributing to Airflow, I had to read up on DAG, > > and > > > > > now I > > > > > >> know what it means. I wonder if we are about to have users who > use > > > the > > > > > >> term > > > > > >> Dag without knowing that there's a DAG. > > > > > >> > > > > > >> Slightly concerned. > > > > > >> > > > > > >> -ephraim > > > > > >> > > > > > >> On Tue, 18 Feb 2025 at 10:26, Wei Lee <weilee...@gmail.com> > > wrote: > > > > > >> > > > > > >> > For user-facing things (e.g., `from airflow import DAG`), I’m > > more > > > > > like > > > > > >> > -0. But for documentation, docstring, internal things and > etc., > > we > > > > > >> probably > > > > > >> > could still change most of them? > > > > > >> > > > > > > >> > Best, > > > > > >> > Wei > > > > > >> > > > > > > >> > > On Feb 18, 2025, at 5:10 PM, Ash Berlin-Taylor < > > a...@apache.org> > > > > > >> wrote: > > > > > >> > > > > > > > >> > > I also don’t personally think it’s worth the pain (not to > > > mention > > > > > >> > backcompat workaround) to rename DAG to Dag, so I’d be -0.5 on > > > that. > > > > > >> > > > > > > > >> > > -ash > > > > > >> > > > > > > > >> > >> On 18 Feb 2025, at 04:40, Daniel Imberman < > > > > > daniel.imber...@gmail.com > > > > > >> > > > > > > >> > wrote: > > > > > >> > >> > > > > > >> > >> I think my biggest concern is a marketing one and not a > > > technical > > > > > >> one. > > > > > >> > >> > > > > > >> > >> As has been mentioned on the thread the terms airflow and > dag > > > are > > > > > >> kind > > > > > >> > of > > > > > >> > >> synonymous and I certainly don’t want to give the > impression > > > that > > > > > we > > > > > >> are > > > > > >> > >> breaking more than we are breaking. > > > > > >> > >> > > > > > >> > >> I wouldn’t die on this hill, but I’m slightly concerned. > > > > > >> > >> > > > > > >> > >> On Mon, Feb 17, 2025 at 6:00 PM Wei Lee <wei...@apache.org > > > > > > wrote: > > > > > >> > >> > > > > > >> > >>> I’m not sure about adding ruff rules here 🤔 I think ruff > > > rules > > > > > are > > > > > >> > best > > > > > >> > >>> suited for user-facing things but not the airflow code > base > > > > > itself. > > > > > >> If > > > > > >> > what > > > > > >> > >>> we mean is adding a rule to avoid users using "DAG" > *after* > > we > > > > > >> rename > > > > > >> > it, > > > > > >> > >>> it's definitely a +1000. > > > > > >> > >>> > > > > > >> > >>> I just created GitHub issues for this removing "DAG" > > > > > >> > >>> > > > > > >> > >>> https://github.com/apache/airflow/issues/46842 > > > > > >> > >>> https://github.com/apache/airflow/issues/46843 > > > > > >> > >>> > > > > > >> > >>> One thing I'm not sure about is whether we want to get rid > > of > > > > > `from > > > > > >> > >>> airflow import DAG` as well. 🤔 > > > > > >> > >>> > > > > > >> > >>> Best, > > > > > >> > >>> Wei > > > > > >> > >>> > > > > > >> > >>> On 2025/02/17 19:50:57 Jarek Potiuk wrote: > > > > > >> > >>>> Hard to say until it's looked at :) > > > > > >> > >>>> > > > > > >> > >>>> On Mon, Feb 17, 2025 at 8:45 PM Aritra Basu < > > > > > >> aritrabasu1...@gmail.com > > > > > >> > > > > > > > >> > >>>> wrote: > > > > > >> > >>>> > > > > > >> > >>>>> I can take it up, it's mostly just a doc update right? > Or > > > are > > > > we > > > > > >> > doing > > > > > >> > >>> code > > > > > >> > >>>>> files replacement too? > > > > > >> > >>>>> -- > > > > > >> > >>>>> Regards, > > > > > >> > >>>>> Aritra Basu > > > > > >> > >>>>> > > > > > >> > >>>>> On Tue, 18 Feb 2025, 12:55 am Jarek Potiuk, < > > > ja...@potiuk.com > > > > > > > > > > >> > wrote: > > > > > >> > >>>>> > > > > > >> > >>>>>> Sounds like another +1000 files big PR is coming :) ? > Any > > > > > >> volunteers > > > > > >> > >>> to > > > > > >> > >>>>>> make it ? It's fun. > > > > > >> > >>>>>> > > > > > >> > >>>>>> On Mon, Feb 17, 2025 at 6:52 PM Omkar P < > > > > > droiddev5...@gmail.com> > > > > > >> > >>> wrote: > > > > > >> > >>>>>> > > > > > >> > >>>>>>> +1 for ruff rules :) > > > > > >> > >>>>>>> > > > > > >> > >>>>>>> Also would be nice to introduce 'Dag' to replace 'DAG' > > in > > > > the > > > > > >> > >>> Airflow > > > > > >> > >>>>>>> docs, in line with the new UI changes and to make the > > > > renaming > > > > > >> > >>>>>>> consistent across user-facing pages. > > > > > >> > >>>>>>> > > > > > >> > >>>>>>> Regards, > > > > > >> > >>>>>>> Omkar > > > > > >> > >>>>>>> > > > > > >> > >>>>>>> On Mon, Feb 17, 2025 at 8:12 AM Jarek Potiuk < > > > > > ja...@potiuk.com> > > > > > >> > >>> wrote: > > > > > >> > >>>>>>> > > > > > >> > >>>>>>>> +1 :) . Maybe we could add a ruff rule for that :) > > > > > >> > >>>>>>>> > > > > > >> > >>>>>>>> On Mon, Feb 17, 2025 at 8:27 AM Wei Lee < > > > wei...@apache.org > > > > > > > > > > >> > >>> wrote: > > > > > >> > >>>>>>>> > > > > > >> > >>>>>>>>> It seems that our current conclusion is to use "dag" > > or > > > > > "Dag" > > > > > >> > >>>>> instead > > > > > >> > >>>>>>> of > > > > > >> > >>>>>>>>> "DAG" whenever possible. Should we replace all "DAG" > > in > > > > the > > > > > >> > >>>>> codebase > > > > > >> > >>>>>>> with > > > > > >> > >>>>>>>>> "dag" or "Dag"? If it's too late for that (which it > > > might > > > > be > > > > > >> > >>> 🤔), > > > > > >> > >>>>>>> should > > > > > >> > >>>>>>>> we > > > > > >> > >>>>>>>>> at least avoid introducing new "DAG" in the > following > > > PRs? > > > > > >> > >>>>>>>>> > > > > > >> > >>>>>>>>> Best, > > > > > >> > >>>>>>>>> Wei > > > > > >> > >>>>>>>>> > > > > > >> > >>>>>>>>> On 2024/10/23 17:16:55 Brent Bovenzi wrote: > > > > > >> > >>>>>>>>>> Here's a PR to use "dag" as a word in the new 3.0 > UI: > > > > > >> > >>>>>>>>>> https://github.com/apache/airflow/pull/43325 > > > > > >> > >>>>>>>>>> Let me know if that's the direction we want to go. > > > > > >> > >>>>>>>>>> > > > > > >> > >>>>>>>>>> On Wed, Oct 23, 2024 at 11:06 AM Bishundeo, > Rajeshwar > > > > > >> > >>>>>>>>>> <rbish...@amazon.com.invalid> wrote: > > > > > >> > >>>>>>>>>> > > > > > >> > >>>>>>>>>>> I think Brent & Daniel summarized it best, "dag" > is > > > > > >> > >>> synonymous > > > > > >> > >>>>>> with > > > > > >> > >>>>>>>>>>> workflows in Airflow through the way we talk and > > > explain > > > > > >> > >>> what > > > > > >> > >>>>>>> Airflow > > > > > >> > >>>>>>>>> is > > > > > >> > >>>>>>>>>>> all about. Although folks would ask, I don’t ever > > use > > > > the > > > > > >> > >>>>>>>> mathematical > > > > > >> > >>>>>>>>>>> definition of DAG. > > > > > >> > >>>>>>>>>>> It will be challenging and possibly confusing for > > many > > > > > >> > >>> users > > > > > >> > >>>>>> making > > > > > >> > >>>>>>>>> such a > > > > > >> > >>>>>>>>>>> change - I would rather direct that energy to > > > appending > > > > > the > > > > > >> > >>>>>> Oxford > > > > > >> > >>>>>>>>>>> definition of "dag" to include a reference to > > > workflows > > > > in > > > > > >> > >>>>>> Airflow. > > > > > >> > >>>>>>>> __ > > > > > >> > >>>>>>>>>>> > > > > > >> > >>>>>>>>>>> -- Rajesh > > > > > >> > >>>>>>>>>>> > > > > > >> > >>>>>>>>>>> > > > > > >> > >>>>>>>>>>> > > > > > >> > >>>>>>>>>>> > > > > > >> > >>>>>>>>>>> > > > > > >> > >>>>>>>>>>> > > > > > >> > >>>>>>>>>>> On 2024-10-23, 3:37 AM, "Jarek Potiuk" < > > > > ja...@potiuk.com > > > > > >> > >>>>>> <mailto: > > > > > >> > >>>>>>>>>>> ja...@potiuk.com>> wrote: > > > > > >> > >>>>>>>>>>> > > > > > >> > >>>>>>>>>>> > > > > > >> > >>>>>>>>>>> CAUTION: This email originated from outside of the > > > > > >> > >>>>> organization. > > > > > >> > >>>>>> Do > > > > > >> > >>>>>>>> not > > > > > >> > >>>>>>>>>>> click links or open attachments unless you can > > confirm > > > > the > > > > > >> > >>>>> sender > > > > > >> > >>>>>>> and > > > > > >> > >>>>>>>>> know > > > > > >> > >>>>>>>>>>> the content is safe. > > > > > >> > >>>>>>>>>>> > > > > > >> > >>>>>>>>>>> > > > > > >> > >>>>>>>>>>> > > > > > >> > >>>>>>>>>>> > > > > > >> > >>>>>>>>>>> > > > > > >> > >>>>>>>>>>> > > > > > >> > >>>>>>>>>>> AVERTISSEMENT: Ce courrier électronique provient > > d’un > > > > > >> > >>>>> expéditeur > > > > > >> > >>>>>>>>> externe. > > > > > >> > >>>>>>>>>>> Ne cliquez sur aucun lien et n’ouvrez aucune pièce > > > > jointe > > > > > >> > >>> si > > > > > >> > >>>>> vous > > > > > >> > >>>>>>> ne > > > > > >> > >>>>>>>>> pouvez > > > > > >> > >>>>>>>>>>> pas confirmer l’identité de l’expéditeur et si > vous > > > > n’êtes > > > > > >> > >>> pas > > > > > >> > >>>>>>>> certain > > > > > >> > >>>>>>>>> que > > > > > >> > >>>>>>>>>>> le contenu ne présente aucun risque. > > > > > >> > >>>>>>>>>>> > > > > > >> > >>>>>>>>>>> > > > > > >> > >>>>>>>>>>> > > > > > >> > >>>>>>>>>>> > > > > > >> > >>>>>>>>>>> > > > > > >> > >>>>>>>>>>> > > > > > >> > >>>>>>>>>>> From Guido's post: > > > > > >> > >>>>>>>>>>> > > > > > >> > >>>>>>>>>>> > > > > > >> > >>>>>>>>>>> "Around this time the renaming seems to have been > > > > > renamed". > > > > > >> > >>>>>>>>>>> > > > > > >> > >>>>>>>>>>> > > > > > >> > >>>>>>>>>>> Naming is hard. > > > > > >> > >>>>>>>>>>> > > > > > >> > >>>>>>>>>>> > > > > > >> > >>>>>>>>>>> J. > > > > > >> > >>>>>>>>>>> > > > > > >> > >>>>>>>>>>> > > > > > >> > >>>>>>>>>>> > > > > > >> > >>>>>>>>>>> > > > > > >> > >>>>>>>>>>> On Wed, Oct 23, 2024 at 7:11 AM Omkar P < > > > > > >> > >>>>> droiddev5...@gmail.com > > > > > >> > >>>>>>>>> <mailto: > > > > > >> > >>>>>>>>>>> droiddev5...@gmail.com>> wrote: > > > > > >> > >>>>>>>>>>> > > > > > >> > >>>>>>>>>>> > > > > > >> > >>>>>>>>>>>> Yes, completely agree with above comments, dag is > > an > > > > > >> > >>> Airflow > > > > > >> > >>>>>> term > > > > > >> > >>>>>>>> now > > > > > >> > >>>>>>>>>>>> rather > > > > > >> > >>>>>>>>>>>> than just a "directed acyclic graph". Believe it > or > > > > not, > > > > > >> > >>> I've > > > > > >> > >>>>>>>> worked > > > > > >> > >>>>>>>>> with > > > > > >> > >>>>>>>>>>>> work > > > > > >> > >>>>>>>>>>>> folks who've been using dags in Airflow for years > > now > > > > > >> > >>> and are > > > > > >> > >>>>>> pro > > > > > >> > >>>>>>>>> devs > > > > > >> > >>>>>>>>>>> but > > > > > >> > >>>>>>>>>>>> have trouble remembering the full form of a DAG! > > For > > > > > >> > >>> them, > > > > > >> > >>>>> dag > > > > > >> > >>>>>> = > > > > > >> > >>>>>>>>> Airflow. > > > > > >> > >>>>>>>>>>>> > > > > > >> > >>>>>>>>>>>> workflow, pipeline, flow are surely better terms > > but > > > > > >> > >>> will be > > > > > >> > >>>>> a > > > > > >> > >>>>>>>> major > > > > > >> > >>>>>>>>>>>> behavior > > > > > >> > >>>>>>>>>>>> change for the developer community, so we'll > need a > > > > solid > > > > > >> > >>>>> plan > > > > > >> > >>>>>> on > > > > > >> > >>>>>>>>> how to > > > > > >> > >>>>>>>>>>>> introduce it, when we do. > > > > > >> > >>>>>>>>>>>> > > > > > >> > >>>>>>>>>>>> While we discuss this, I'd like to share about > the > > > > Great > > > > > >> > >>>>>> Renaming > > > > > >> > >>>>>>>> in > > > > > >> > >>>>>>>>> core > > > > > >> > >>>>>>>>>>>> Python started by Guido back in 2009. We could > > > probably > > > > > >> > >>> get > > > > > >> > >>>>>> some > > > > > >> > >>>>>>>>>>> learnings > > > > > >> > >>>>>>>>>>>> from there? Who knows! > > > > > >> > >>>>>>>>>>>> > > > > > >> > >>>>>>>>>>>> Guido's blog (2009): > > > > > >> > >>>>>>>>>>>> > > > > > >> > >>>>>>>>> > > > > > >> > >>>>>>> > > > > > >> > >>>>> > > > > > >> > >>> > > > > > >> > > > > > > >> > > > > > > > > > https://python-history.blogspot.com/2009/03/great-or-grand-renaming.html > > > > > >> > >>>>>>>>>>> < > > > > > >> > >>>>>>>>> > > > > > >> > >>>>>>> > > > > > >> > >>>>> > > > > > >> > >>> > > > > > >> > > > > > > >> > > > > > > > > > https://python-history.blogspot.com/2009/03/great-or-grand-renaming.html > > > > > >> > >>>>>>>>> > > > > > >> > >>>>>>>>>>>> Follow-up discussion (2024): > > > > > >> > >>>>>>>>>>>> > > > > > >> > >>>>>> > > > > > https://discuss.python.org/t/finishing-the-great-renaming/54082 > > > > > >> > >>>>>>> < > > > > > >> > >>>>>>>>>>> > > > > > >> > >>>>> > > > > https://discuss.python.org/t/finishing-the-great-renaming/54082 > > > > > > > > > > > >> > >>>>>>>>>>>> > > > > > >> > >>>>>>>>>>>> Cheers, > > > > > >> > >>>>>>>>>>>> Omkar > > > > > >> > >>>>>>>>>>>> > > > > > >> > >>>>>>>>>>>> On Wed, Oct 23, 2024 at 7:10 AM Wei Lee < > > > > > >> > >>> weilee...@gmail.com > > > > > >> > >>>>>>>>> <mailto: > > > > > >> > >>>>>>>>>>> weilee...@gmail.com>> wrote: > > > > > >> > >>>>>>>>>>>> > > > > > >> > >>>>>>>>>>>>> I think we should probably just accept it as an > > > > airflow > > > > > >> > >>>>> term. > > > > > >> > >>>>>>> At > > > > > >> > >>>>>>>>> least, > > > > > >> > >>>>>>>>>>>>> that’s how I understood it when I first used > > > Airflow. > > > > I > > > > > >> > >>>>> feel > > > > > >> > >>>>>>>>> renaming > > > > > >> > >>>>>>>>>>> it > > > > > >> > >>>>>>>>>>>> at > > > > > >> > >>>>>>>>>>>>> this stage would require considerable effort > from > > > > > >> > >>>>> maintainers > > > > > >> > >>>>>>> and > > > > > >> > >>>>>>>>>>>> existing > > > > > >> > >>>>>>>>>>>>> users without providing equivalent benefits. > > > > > >> > >>>>>>>>>>>>> > > > > > >> > >>>>>>>>>>>>> Best, > > > > > >> > >>>>>>>>>>>>> Wei > > > > > >> > >>>>>>>>>>>>> > > > > > >> > >>>>>>>>>>>>>> On Oct 23, 2024, at 8:01 AM, Kaxil Naik < > > > > > >> > >>>>>> kaxiln...@gmail.com > > > > > >> > >>>>>>>>>>> <mailto:kaxiln...@gmail.com>> wrote: > > > > > >> > >>>>>>>>>>>>>> > > > > > >> > >>>>>>>>>>>>>> Same agreed with Brent & Daniel -- maybe we > > > re-kindle > > > > > >> > >>>>> this > > > > > >> > >>>>>>>>> discussion > > > > > >> > >>>>>>>>>>>> for > > > > > >> > >>>>>>>>>>>>>> Airflow 4 :) -- but right now it will cause too > > > much > > > > > >> > >>>>>>> disruption > > > > > >> > >>>>>>>>>>>>>> > > > > > >> > >>>>>>>>>>>>>> On Tue, 22 Oct 2024 at 21:27, Constance > Martineau > > > > > >> > >>>>>>>>>>>>>> <consta...@astronomer.io.inva <mailto: > > > > > >> > >>>>>>>>> consta...@astronomer.io.inva>lid> > > > > > >> > >>>>>>>>>>> wrote: > > > > > >> > >>>>>>>>>>>>>> > > > > > >> > >>>>>>>>>>>>>>> In my experience, when you ask those with > > Airflow > > > > > >> > >>>>>> experience > > > > > >> > >>>>>>>>> what a > > > > > >> > >>>>>>>>>>>> dag > > > > > >> > >>>>>>>>>>>>> is, > > > > > >> > >>>>>>>>>>>>>>> they'll start talking about workflow > attributes > > - > > > > > >> > >>> stuff > > > > > >> > >>>>>> like > > > > > >> > >>>>>>>>> dags > > > > > >> > >>>>>>>>>>>> being > > > > > >> > >>>>>>>>>>>>> a > > > > > >> > >>>>>>>>>>>>>>> series of steps or tasks with owners. The > > > structure > > > > > >> > >>>>>> doesn't > > > > > >> > >>>>>>>>> come up. > > > > > >> > >>>>>>>>>>>>>>> > > > > > >> > >>>>>>>>>>>>>>> Echo-ing others, at this point, my vote is to > > > > > >> > >>> embrace > > > > > >> > >>>>> the > > > > > >> > >>>>>>> name > > > > > >> > >>>>>>>>> and > > > > > >> > >>>>>>>>>>>>>>> de-emphasize the mathematical structure > aspect. > > > > > >> > >>>>>>>>>>>>>>> > > > > > >> > >>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 3:47 PM Vikram Koka > > > > > >> > >>>>>>>>>>>>> <vik...@astronomer.io.inva <mailto: > > > > > >> > >>>>> vik...@astronomer.io.inva > > > > > >> > >>>>>>>>> lid> > > > > > >> > >>>>>>>>>>>>>>> wrote: > > > > > >> > >>>>>>>>>>>>>>> > > > > > >> > >>>>>>>>>>>>>>>> It's an interesting discussion and I remember > > > > > >> > >>>>> struggling > > > > > >> > >>>>>>> with > > > > > >> > >>>>>>>>> this > > > > > >> > >>>>>>>>>>>>> when I > > > > > >> > >>>>>>>>>>>>>>>> started working with Airflow. > > > > > >> > >>>>>>>>>>>>>>>> But, I also agree with the viewpoint of it > > being > > > an > > > > > >> > >>>>>>>> established > > > > > >> > >>>>>>>>>>>> concept > > > > > >> > >>>>>>>>>>>>>>> now > > > > > >> > >>>>>>>>>>>>>>>> regardless of the origin. > > > > > >> > >>>>>>>>>>>>>>>> > > > > > >> > >>>>>>>>>>>>>>>> I am personally leaning towards the > perspective > > > > > >> > >>> best > > > > > >> > >>>>>>>> expressed > > > > > >> > >>>>>>>>> by > > > > > >> > >>>>>>>>>>>>> Daniel > > > > > >> > >>>>>>>>>>>>>>>> Standish and Brent of using Dag as a word, > > rather > > > > > >> > >>> than > > > > > >> > >>>>>> the > > > > > >> > >>>>>>>>> computer > > > > > >> > >>>>>>>>>>>>>>> science > > > > > >> > >>>>>>>>>>>>>>>> concept. > > > > > >> > >>>>>>>>>>>>>>>> > > > > > >> > >>>>>>>>>>>>>>>> Best regards, > > > > > >> > >>>>>>>>>>>>>>>> Vikram > > > > > >> > >>>>>>>>>>>>>>>> > > > > > >> > >>>>>>>>>>>>>>>> > > > > > >> > >>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 9:46 AM Oliveira, > Niko > > > > > >> > >>>>>>>>>>>>>>> <oniko...@amazon.com.inva <mailto: > > > > > >> > >>>>>> oniko...@amazon.com.inva > > > > > >> > >>>>>>>>> lid > > > > > >> > >>>>>>>>>>>>>>>>> > > > > > >> > >>>>>>>>>>>>>>>> wrote: > > > > > >> > >>>>>>>>>>>>>>>> > > > > > >> > >>>>>>>>>>>>>>>>> I agree with the general sentiment of: > You're > > > > > >> > >>> right > > > > > >> > >>>>>> Ryan, > > > > > >> > >>>>>>>> DAG > > > > > >> > >>>>>>>>>>> isn't > > > > > >> > >>>>>>>>>>>>>>> great > > > > > >> > >>>>>>>>>>>>>>>>> and I'd rather workflow, but changing it > will > > > > > >> > >>> cause > > > > > >> > >>>>> much > > > > > >> > >>>>>>>> more > > > > > >> > >>>>>>>>>>>> wreckage > > > > > >> > >>>>>>>>>>>>>>>> than > > > > > >> > >>>>>>>>>>>>>>>>> it solves. > > > > > >> > >>>>>>>>>>>>>>>>> > > > > > >> > >>>>>>>>>>>>>>>>> Also agree with the idea to just move away > > from > > > > > >> > >>>>> defining > > > > > >> > >>>>>>>> DAG. > > > > > >> > >>>>>>>>> I > > > > > >> > >>>>>>>>>>>> think > > > > > >> > >>>>>>>>>>>>>>>>> we've been naturally doing that as a > community > > > > > >> > >>> for a > > > > > >> > >>>>>> while > > > > > >> > >>>>>>>> now > > > > > >> > >>>>>>>>>>>> anyway, > > > > > >> > >>>>>>>>>>>>>>> so > > > > > >> > >>>>>>>>>>>>>>>>> that feels like a natural step. > > > > > >> > >>>>>>>>>>>>>>>>> > > > > > >> > >>>>>>>>>>>>>>>>> Cheers, > > > > > >> > >>>>>>>>>>>>>>>>> Niko > > > > > >> > >>>>>>>>>>>>>>>>> > > > > > >> > >>>>>>>>>>>>>>>>> ________________________________ > > > > > >> > >>>>>>>>>>>>>>>>> From: Ash Berlin-Taylor <a...@apache.org > > > <mailto: > > > > > >> > >>>>>>>>> a...@apache.org>> > > > > > >> > >>>>>>>>>>>>>>>>> Sent: Tuesday, October 22, 2024 9:06:39 AM > > > > > >> > >>>>>>>>>>>>>>>>> To: dev@airflow.apache.org <mailto: > > > > > >> > >>>>>> dev@airflow.apache.org > > > > > >> > >>>>>>>> > > > > > >> > >>>>>>>>>>>>>>>>> Subject: RE: [EXT] Airflow should deprecate > > the > > > > > >> > >>> term > > > > > >> > >>>>>> "DAG" > > > > > >> > >>>>>>>>> for end > > > > > >> > >>>>>>>>>>>>>>> users > > > > > >> > >>>>>>>>>>>>>>>>> > > > > > >> > >>>>>>>>>>>>>>>>> CAUTION: This email originated from outside > of > > > the > > > > > >> > >>>>>>>>> organization. > > > > > >> > >>>>>>>>>>> Do > > > > > >> > >>>>>>>>>>>>> not > > > > > >> > >>>>>>>>>>>>>>>>> click links or open attachments unless you > can > > > > > >> > >>> confirm > > > > > >> > >>>>>> the > > > > > >> > >>>>>>>>> sender > > > > > >> > >>>>>>>>>>>> and > > > > > >> > >>>>>>>>>>>>>>>> know > > > > > >> > >>>>>>>>>>>>>>>>> the content is safe. > > > > > >> > >>>>>>>>>>>>>>>>> > > > > > >> > >>>>>>>>>>>>>>>>> > > > > > >> > >>>>>>>>>>>>>>>>> > > > > > >> > >>>>>>>>>>>>>>>>> AVERTISSEMENT: Ce courrier électronique > > provient > > > > > >> > >>> d’un > > > > > >> > >>>>>>>>> expéditeur > > > > > >> > >>>>>>>>>>>>>>> externe. > > > > > >> > >>>>>>>>>>>>>>>>> Ne cliquez sur aucun lien et n’ouvrez aucune > > > pièce > > > > > >> > >>>>>> jointe > > > > > >> > >>>>>>> si > > > > > >> > >>>>>>>>> vous > > > > > >> > >>>>>>>>>>> ne > > > > > >> > >>>>>>>>>>>>>>>> pouvez > > > > > >> > >>>>>>>>>>>>>>>>> pas confirmer l’identité de l’expéditeur et > si > > > > > >> > >>> vous > > > > > >> > >>>>>> n’êtes > > > > > >> > >>>>>>>> pas > > > > > >> > >>>>>>>>>>>> certain > > > > > >> > >>>>>>>>>>>>>>>> que > > > > > >> > >>>>>>>>>>>>>>>>> le contenu ne présente aucun risque. > > > > > >> > >>>>>>>>>>>>>>>>> > > > > > >> > >>>>>>>>>>>>>>>>> > > > > > >> > >>>>>>>>>>>>>>>>> > > > > > >> > >>>>>>>>>>>>>>>>> Best argument in favour of keeping “dags” > as a > > > > > >> > >>> term — > > > > > >> > >>>>>>>> getting > > > > > >> > >>>>>>>>> to > > > > > >> > >>>>>>>>>>>>> re-use > > > > > >> > >>>>>>>>>>>>>>>>> puns like https://i.imgflip.com/1xhtwh.jpg > < > > > > > >> > >>>>>>>>>>> https://i.imgflip.com/1xhtwh.jpg> > > > > > >> > >>>>>>>>>>>>>>>>> > > > > > >> > >>>>>>>>>>>>>>>>> In all seriousness: I don’t mind either way, > > > both > > > > > >> > >>>>> sides > > > > > >> > >>>>>>> have > > > > > >> > >>>>>>>>> good > > > > > >> > >>>>>>>>>>>>>>> reasons > > > > > >> > >>>>>>>>>>>>>>>>> presented. > > > > > >> > >>>>>>>>>>>>>>>>> > > > > > >> > >>>>>>>>>>>>>>>>> -a > > > > > >> > >>>>>>>>>>>>>>>>> > > > > > >> > >>>>>>>>>>>>>>>>>> On 22 Oct 2024, at 17:03, Daniel Standish > > > > > >> > >>>>>>>>>>>>>>>>> <daniel.stand...@astronomer.io.inva > <mailto: > > > > > >> > >>>>>>>>>>> daniel.stand...@astronomer.io.inva>LID> wrote: > > > > > >> > >>>>>>>>>>>>>>>>>> > > > > > >> > >>>>>>>>>>>>>>>>>> Yeah just say, when asked where the name > > comes > > > > > >> > >>> from, > > > > > >> > >>>>>>> "well, > > > > > >> > >>>>>>>>> no > > > > > >> > >>>>>>>>>>> one > > > > > >> > >>>>>>>>>>>>>>>>> actually > > > > > >> > >>>>>>>>>>>>>>>>>> knows but..." and then make something up. > > > > > >> > >>>>>>>>>>>>>>>>>> > > > > > >> > >>>>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 8:31 AM Jarek > Potiuk > > < > > > > > >> > >>>>>>>>> ja...@potiuk.com > > > > > >> > >>>>>>>>>>> <mailto:ja...@potiuk.com>> > > > > > >> > >>>>>>>>>>>>>>> wrote: > > > > > >> > >>>>>>>>>>>>>>>>>> > > > > > >> > >>>>>>>>>>>>>>>>>>> Just to clarify - "directed acyclic graph" > > is > > > > > >> > >>> the > > > > > >> > >>>>>>>>>>> tongue-twister, > > > > > >> > >>>>>>>>>>>>>>>>>>> > > > > > >> > >>>>>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 5:29 PM Jarek > > Potiuk < > > > > > >> > >>>>>>>>> ja...@potiuk.com > > > > > >> > >>>>>>>>>>> <mailto:ja...@potiuk.com>> > > > > > >> > >>>>>>>>>>>>>>>> wrote: > > > > > >> > >>>>>>>>>>>>>>>>>>> > > > > > >> > >>>>>>>>>>>>>>>>>>>> I like what both Daniel and Brent wrote. > I > > > > > >> > >>> would > > > > > >> > >>>>> very > > > > > >> > >>>>>>>> much > > > > > >> > >>>>>>>>> want > > > > > >> > >>>>>>>>>>>> to > > > > > >> > >>>>>>>>>>>>>>> be > > > > > >> > >>>>>>>>>>>>>>>>>>> able > > > > > >> > >>>>>>>>>>>>>>>>>>>> to say just "dag" without explaining it > > > > > >> > >>> further. > > > > > >> > >>>>>>>>>>>>>>>>>>>> > > > > > >> > >>>>>>>>>>>>>>>>>>>> For me every time I explain "DAG" at a > talk > > > > > >> > >>> it's a > > > > > >> > >>>>>>>>>>>> tongue-twister, > > > > > >> > >>>>>>>>>>>>>>>> and > > > > > >> > >>>>>>>>>>>>>>>>> I > > > > > >> > >>>>>>>>>>>>>>>>>>>> almost stutter on trying to recall how to > > > > > >> > >>> pronounce > > > > > >> > >>>>>> it > > > > > >> > >>>>>>>>>>> properly. > > > > > >> > >>>>>>>>>>>>>>>>>>>> > > > > > >> > >>>>>>>>>>>>>>>>>>>> J. > > > > > >> > >>>>>>>>>>>>>>>>>>>> > > > > > >> > >>>>>>>>>>>>>>>>>>>> > > > > > >> > >>>>>>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 5:27 PM Brent > > Bovenzi > > > > > >> > >>>>>>>>>>>>>>>>>>> <br...@astronomer.io.inva <mailto: > > > > > >> > >>>>>>>> br...@astronomer.io.inva > > > > > >> > >>>>>>>>>> lid> > > > > > >> > >>>>>>>>>>>>>>>>>>>> wrote: > > > > > >> > >>>>>>>>>>>>>>>>>>>> > > > > > >> > >>>>>>>>>>>>>>>>>>>>> I remember we explored renaming "DAG" > when > > > > > >> > >>>>> starting > > > > > >> > >>>>>> on > > > > > >> > >>>>>>>>> AIP-38 > > > > > >> > >>>>>>>>>>> to > > > > > >> > >>>>>>>>>>>>>>>>>>> modernize > > > > > >> > >>>>>>>>>>>>>>>>>>>>> the UI. Both "pipeline" or "workflow" > are > > > more > > > > > >> > >>>>>>>>> descriptive of > > > > > >> > >>>>>>>>>>>> what > > > > > >> > >>>>>>>>>>>>>>>> one > > > > > >> > >>>>>>>>>>>>>>>>>>> is > > > > > >> > >>>>>>>>>>>>>>>>>>>>> actually doing while Directed Acyclic > > Graph > > > > > >> > >>> is an > > > > > >> > >>>>>>>>>>> implementation > > > > > >> > >>>>>>>>>>>>>>>>> detail. > > > > > >> > >>>>>>>>>>>>>>>>>>>>> But I agree with Daniel Standish, at > this > > > > > >> > >>> point > > > > > >> > >>>>>> "DAG" > > > > > >> > >>>>>>>> has > > > > > >> > >>>>>>>>>>> become > > > > > >> > >>>>>>>>>>>>>>>> "dag" > > > > > >> > >>>>>>>>>>>>>>>>>>> , a > > > > > >> > >>>>>>>>>>>>>>>>>>>>> word in its own right. > > > > > >> > >>>>>>>>>>>>>>>>>>>>> > > > > > >> > >>>>>>>>>>>>>>>>>>>>> Examples for "dag" are abound in > community > > > > > >> > >>>>>> discussion, > > > > > >> > >>>>>>>>> Airflow > > > > > >> > >>>>>>>>>>>>>>>> Summit > > > > > >> > >>>>>>>>>>>>>>>>>>>>> talks, documentation and even in the UI. > > > Let's > > > > > >> > >>>>>> embrace > > > > > >> > >>>>>>>>> "dag". > > > > > >> > >>>>>>>>>>> A > > > > > >> > >>>>>>>>>>>>>>> user > > > > > >> > >>>>>>>>>>>>>>>>>>> just > > > > > >> > >>>>>>>>>>>>>>>>>>>>> needs to learn one new word vs the > > technical > > > > > >> > >>>>> concept > > > > > >> > >>>>>>>>> behind > > > > > >> > >>>>>>>>>>> that > > > > > >> > >>>>>>>>>>>>>>>>> word. I > > > > > >> > >>>>>>>>>>>>>>>>>>>>> think that is much less effort than > > > > > >> > >>> refactoring so > > > > > >> > >>>>>>> much > > > > > >> > >>>>>>>>> code, > > > > > >> > >>>>>>>>>>>>>>>>>>>>> documentation, blog posts, stack > overflow > > > > > >> > >>>>> questions, > > > > > >> > >>>>>>>> etc. > > > > > >> > >>>>>>>>>>>>>>>>>>>>> > > > > > >> > >>>>>>>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 10:51 AM Daniel > > > > > >> > >>> Standish > > > > > >> > >>>>>>>>>>>>>>>>>>>>> <daniel.stand...@astronomer.io.inva > > > <mailto: > > > > > >> > >>>>>>>>>>> daniel.stand...@astronomer.io.inva>lid> wrote: > > > > > >> > >>>>>>>>>>>>>>>>>>>>> > > > > > >> > >>>>>>>>>>>>>>>>>>>>>> I am skeptical. Seems like introducing > a > > > lot > > > > > >> > >>> of > > > > > >> > >>>>>> pain > > > > > >> > >>>>>>>> for > > > > > >> > >>>>>>>>>>>>>>>>> questionable > > > > > >> > >>>>>>>>>>>>>>>>>>>>>> benefit. But, I am def sympathetic to > the > > > > > >> > >>> idea. I > > > > > >> > >>>>>>> agree > > > > > >> > >>>>>>>>> the > > > > > >> > >>>>>>>>>>>>>>>>>>>>> association > > > > > >> > >>>>>>>>>>>>>>>>>>>>>> with "directed acyclic graph" is not > > > helpful. > > > > > >> > >>>>>>>>>>>>>>>>>>>>>> > > > > > >> > >>>>>>>>>>>>>>>>>>>>>> And along those lines, I offer here > some > > > less > > > > > >> > >>>>>>> invasive > > > > > >> > >>>>>>>>>>>>>>> mitigations. > > > > > >> > >>>>>>>>>>>>>>>>>>>>>> > > > > > >> > >>>>>>>>>>>>>>>>>>>>>> One thing we can do no matter what is > to > > > > > >> > >>>>>> de-emphasize > > > > > >> > >>>>>>>> the > > > > > >> > >>>>>>>>>>> math > > > > > >> > >>>>>>>>>>>>>>> nerd > > > > > >> > >>>>>>>>>>>>>>>>>>>>> origins > > > > > >> > >>>>>>>>>>>>>>>>>>>>>> of the name. That is to say, in docs / > > > > > >> > >>> website / > > > > > >> > >>>>>> etc, > > > > > >> > >>>>>>>>> *never > > > > > >> > >>>>>>>>>>>>>>>> define* > > > > > >> > >>>>>>>>>>>>>>>>>>>>>> airflow's "dag" concept as a directed > > > acyclic > > > > > >> > >>>>>> graph. > > > > > >> > >>>>>>>>> Always > > > > > >> > >>>>>>>>>>>>>>> define > > > > > >> > >>>>>>>>>>>>>>>>> it > > > > > >> > >>>>>>>>>>>>>>>>>>>>> as a > > > > > >> > >>>>>>>>>>>>>>>>>>>>>> pipeline, collection of tasks, workflow > > > etc. > > > > > >> > >>>>>>>>>>>>>>>>>>>>>> > > > > > >> > >>>>>>>>>>>>>>>>>>>>>> The "directed acyclic graph" part of it > > is > > > > > >> > >>> like a > > > > > >> > >>>>>>>>> historical > > > > > >> > >>>>>>>>>>>>>>>>> footnote, > > > > > >> > >>>>>>>>>>>>>>>>>>>>> and > > > > > >> > >>>>>>>>>>>>>>>>>>>>>> we could make one mention of it > somewhere > > > > > >> > >>> hidden. > > > > > >> > >>>>>>>>>>>>>>>>>>>>>> > > > > > >> > >>>>>>>>>>>>>>>>>>>>>> We could also start using lowercase in > > the > > > > > >> > >>> docs > > > > > >> > >>>>> in > > > > > >> > >>>>>>>>> general > > > > > >> > >>>>>>>>>>> e.g. > > > > > >> > >>>>>>>>>>>>>>>>>>> writing > > > > > >> > >>>>>>>>>>>>>>>>>>>>>> "dag" / "dags" instead of writing > "DAG" / > > > > > >> > >>> "DAGs" > > > > > >> > >>>>>> etc. > > > > > >> > >>>>>>>> The > > > > > >> > >>>>>>>>>>>> upper > > > > > >> > >>>>>>>>>>>>>>>> case > > > > > >> > >>>>>>>>>>>>>>>>>>>>> part > > > > > >> > >>>>>>>>>>>>>>>>>>>>>> of it makes it look like an acronym; > but > > > > > >> > >>> "dag" in > > > > > >> > >>>>>>>>> airlfow is > > > > > >> > >>>>>>>>>>>> just > > > > > >> > >>>>>>>>>>>>>>>> an > > > > > >> > >>>>>>>>>>>>>>>>>>>>>> airflow concept and the association > with > > > > > >> > >>> "DAGs" > > > > > >> > >>>>> is > > > > > >> > >>>>>>> not > > > > > >> > >>>>>>>>> really > > > > > >> > >>>>>>>>>>>>>>>>>>> unhelpful. > > > > > >> > >>>>>>>>>>>>>>>>>>>>>> > > > > > >> > >>>>>>>>>>>>>>>>>>>>>> In other words embrace that "dag" in > > > airflow > > > > > >> > >>> is > > > > > >> > >>>>> its > > > > > >> > >>>>>>> own > > > > > >> > >>>>>>>>>>> thing, > > > > > >> > >>>>>>>>>>>> is > > > > > >> > >>>>>>>>>>>>>>>>>>>>>> *not* strictly > > > > > >> > >>>>>>>>>>>>>>>>>>>>>> speaking a directed acyclic graph > (which > > > > > >> > >>> nobody > > > > > >> > >>>>>> knows > > > > > >> > >>>>>>>>> about > > > > > >> > >>>>>>>>>>>>>>>> anyway), > > > > > >> > >>>>>>>>>>>>>>>>>>> and > > > > > >> > >>>>>>>>>>>>>>>>>>>>>> tell them what it is in simple terms > that > > > > > >> > >>> normal > > > > > >> > >>>>>>> people > > > > > >> > >>>>>>>>>>>>>>> understand. > > > > > >> > >>>>>>>>>>>>>>>>>>>>>> > > > > > >> > >>>>>>>>>>>>>>>>>>>>>> > > > > > >> > >>>>>>>>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 7:27 AM Jarek > > > Potiuk > > > > > >> > >>> < > > > > > >> > >>>>>>>>>>> ja...@potiuk.com <mailto:ja...@potiuk.com> > > > > > >> > >>>>>>>>>>>>> > > > > > >> > >>>>>>>>>>>>>>>>>>> wrote: > > > > > >> > >>>>>>>>>>>>>>>>>>>>>> > > > > > >> > >>>>>>>>>>>>>>>>>>>>>>> DAG is so embedded into what we do > that > > it > > > > > >> > >>> will > > > > > >> > >>>>> be > > > > > >> > >>>>>>>>> extremely > > > > > >> > >>>>>>>>>>>>>>>>>>>>> difficult to > > > > > >> > >>>>>>>>>>>>>>>>>>>>>>> get rid of it completely. Also I think > > it > > > > > >> > >>> will > > > > > >> > >>>>>> make > > > > > >> > >>>>>>> a > > > > > >> > >>>>>>>>> lot of > > > > > >> > >>>>>>>>>>>>>>>>>>> "google" > > > > > >> > >>>>>>>>>>>>>>>>>>>>>>> searches and "stack overflow" searches > > not > > > > > >> > >>>>> finding > > > > > >> > >>>>>>> the > > > > > >> > >>>>>>>>> right > > > > > >> > >>>>>>>>>>>>>>>>>>> answers. > > > > > >> > >>>>>>>>>>>>>>>>>>>>>> This > > > > > >> > >>>>>>>>>>>>>>>>>>>>>>> is one of the strengths of Airflow - > > > > > >> > >>> besides the > > > > > >> > >>>>>>>>> community > > > > > >> > >>>>>>>>>>> and > > > > > >> > >>>>>>>>>>>>>>>> ideas > > > > > >> > >>>>>>>>>>>>>>>>>>>>> that > > > > > >> > >>>>>>>>>>>>>>>>>>>>>>> Bernd mentioned - is the vast number > of > > > > > >> > >>>>> examples, > > > > > >> > >>>>>>>>> problems > > > > > >> > >>>>>>>>>>> and > > > > > >> > >>>>>>>>>>>>>>>>>>>>> solutions > > > > > >> > >>>>>>>>>>>>>>>>>>>>>>> you can so easily find (and we have to > > > > > >> > >>> remember > > > > > >> > >>>>>> that > > > > > >> > >>>>>>>>> all the > > > > > >> > >>>>>>>>>>>> AI > > > > > >> > >>>>>>>>>>>>>>>>>>>>> trained > > > > > >> > >>>>>>>>>>>>>>>>>>>>>> on > > > > > >> > >>>>>>>>>>>>>>>>>>>>>>> past data will be also rather poorly > > > > > >> > >>> matching > > > > > >> > >>>>>>> queries > > > > > >> > >>>>>>>> of > > > > > >> > >>>>>>>>>>>> people. > > > > > >> > >>>>>>>>>>>>>>>>>>>>>>> > > > > > >> > >>>>>>>>>>>>>>>>>>>>>>> I am not too attached to DAG. I could > > > easily > > > > > >> > >>>>>> switch. > > > > > >> > >>>>>>>>> And if > > > > > >> > >>>>>>>>>>> we > > > > > >> > >>>>>>>>>>>>>>> do > > > > > >> > >>>>>>>>>>>>>>>> - > > > > > >> > >>>>>>>>>>>>>>>>>>> I > > > > > >> > >>>>>>>>>>>>>>>>>>>>>>> would be for using workflow or > pipeline > > > > > >> > >>> instead > > > > > >> > >>>>> of > > > > > >> > >>>>>>>>> `dag` if > > > > > >> > >>>>>>>>>>>> not > > > > > >> > >>>>>>>>>>>>>>>> the > > > > > >> > >>>>>>>>>>>>>>>>>>>>> above > > > > > >> > >>>>>>>>>>>>>>>>>>>>>>> reason, but I think I am here with > Igor > > > > > >> > >>> that it > > > > > >> > >>>>>>> might > > > > > >> > >>>>>>>>> cause > > > > > >> > >>>>>>>>>>>> more > > > > > >> > >>>>>>>>>>>>>>>>>>>>> problems > > > > > >> > >>>>>>>>>>>>>>>>>>>>>>> than it solves. > > > > > >> > >>>>>>>>>>>>>>>>>>>>>>> > > > > > >> > >>>>>>>>>>>>>>>>>>>>>>> But I am not 100% against - if others > > will > > > > > >> > >>> think > > > > > >> > >>>>>>> it's > > > > > >> > >>>>>>>> a > > > > > >> > >>>>>>>>> good > > > > > >> > >>>>>>>>>>>>>>>> idea, I > > > > > >> > >>>>>>>>>>>>>>>>>>>>> am > > > > > >> > >>>>>>>>>>>>>>>>>>>>>> ok > > > > > >> > >>>>>>>>>>>>>>>>>>>>>>> with it. > > > > > >> > >>>>>>>>>>>>>>>>>>>>>>> > > > > > >> > >>>>>>>>>>>>>>>>>>>>>>> J, > > > > > >> > >>>>>>>>>>>>>>>>>>>>>>> > > > > > >> > >>>>>>>>>>>>>>>>>>>>>>> > > > > > >> > >>>>>>>>>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 3:12 PM > Abhishek > > > > > >> > >>> Bhakat > > > > > >> > >>>>>>>>>>>>>>>>>>>>>>> <abhishek.bha...@astronomer.io.inva > > > > > >> > >>> <mailto: > > > > > >> > >>>>>>>>>>> abhishek.bha...@astronomer.io.inva>lid> wrote: > > > > > >> > >>>>>>>>>>>>>>>>>>>>>>> > > > > > >> > >>>>>>>>>>>>>>>>>>>>>>>> Agreed that the word DAG makes very > > less > > > > > >> > >>> sense > > > > > >> > >>>>> to > > > > > >> > >>>>>>>>> someone > > > > > >> > >>>>>>>>>>> new > > > > > >> > >>>>>>>>>>>>>>> to > > > > > >> > >>>>>>>>>>>>>>>>>>>>>> workflow > > > > > >> > >>>>>>>>>>>>>>>>>>>>>>>> orchestration. But it does also show > > the > > > > > >> > >>> nature > > > > > >> > >>>>>> of > > > > > >> > >>>>>>>>> being > > > > > >> > >>>>>>>>>>>>>>> acyclic. > > > > > >> > >>>>>>>>>>>>>>>>>>>>> Sure, > > > > > >> > >>>>>>>>>>>>>>>>>>>>>>> as > > > > > >> > >>>>>>>>>>>>>>>>>>>>>>>> Bas mentioned, there are ways to > > > > > >> > >>> workaround it. > > > > > >> > >>>>>>>> Still, > > > > > >> > >>>>>>>>> in > > > > > >> > >>>>>>>>>>> my > > > > > >> > >>>>>>>>>>>>>>>>>>>>> opinion, > > > > > >> > >>>>>>>>>>>>>>>>>>>>>>> there > > > > > >> > >>>>>>>>>>>>>>>>>>>>>>>> is generally no need for cyclic > > behavior > > > in > > > > > >> > >>>>>>> workflow > > > > > >> > >>>>>>>>>>>>>>>>>>> orchestration. > > > > > >> > >>>>>>>>>>>>>>>>>>>>>> Most > > > > > >> > >>>>>>>>>>>>>>>>>>>>>>>> (*if > > > > > >> > >>>>>>>>>>>>>>>>>>>>>>>> not all*) cases can be in some way > can > > be > > > > > >> > >>>>> covered > > > > > >> > >>>>>>>>> using an > > > > > >> > >>>>>>>>>>>>>>>> acyclic > > > > > >> > >>>>>>>>>>>>>>>>>>>>>> manner > > > > > >> > >>>>>>>>>>>>>>>>>>>>>>>> with multiple runs. Hence, the > > > > > >> > >>> idempotency. So > > > > > >> > >>>>> I > > > > > >> > >>>>>>>> would > > > > > >> > >>>>>>>>> want > > > > > >> > >>>>>>>>>>>> the > > > > > >> > >>>>>>>>>>>>>>>>>>>>>> "acyclic" > > > > > >> > >>>>>>>>>>>>>>>>>>>>>>>> word to stick. > > > > > >> > >>>>>>>>>>>>>>>>>>>>>>>> > > > > > >> > >>>>>>>>>>>>>>>>>>>>>>>> Regards, > > > > > >> > >>>>>>>>>>>>>>>>>>>>>>>> Avi > > > > > >> > >>>>>>>>>>>>>>>>>>>>>>>> > > > > > >> > >>>>>>>>>>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 12:41 PM < > > > > > >> > >>>>>>>>>>> bernd.stroe...@kosakya.de <mailto: > > > > > >> > >>> bernd.stroe...@kosakya.de>> > > > > > >> > >>>>>>>>>>>>>>>>>>> wrote: > > > > > >> > >>>>>>>>>>>>>>>>>>>>>>>> > > > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> Brilliant, I am on the way to become > > an > > > > > >> > >>>>> Airflow > > > > > >> > >>>>>>> Fan; > > > > > >> > >>>>>>>>> so > > > > > >> > >>>>>>>>>>> many > > > > > >> > >>>>>>>>>>>>>>> new > > > > > >> > >>>>>>>>>>>>>>>>>>>>>> ideas. > > > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> The Term DAG is misleading; it > should > > be > > > > > >> > >>>>>> replaced > > > > > >> > >>>>>>> by > > > > > >> > >>>>>>>>> the > > > > > >> > >>>>>>>>>>>> more > > > > > >> > >>>>>>>>>>>>>>>>>>>>> general > > > > > >> > >>>>>>>>>>>>>>>>>>>>>>>> Term > > > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> Airflow (Workflow) Graph (AFG) or > > > Airflow > > > > > >> > >>>>>> (Petri) > > > > > >> > >>>>>>>> Net > > > > > >> > >>>>>>>>>>> (AFN) > > > > > >> > >>>>>>>>>>>>>>>>>>> (maybe > > > > > >> > >>>>>>>>>>>>>>>>>>>>>>>> without > > > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> a direction); > > > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> and ... these Graphs should be > stored > > > in a > > > > > >> > >>>>> Graph > > > > > >> > >>>>>>>>> Database. > > > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> Every Node or Sup-Graph of an > Airflow > > > > > >> > >>> Graph > > > > > >> > >>>>>> (AFG) > > > > > >> > >>>>>>>>> might be > > > > > >> > >>>>>>>>>>>>>>>>>>>>> assigned > > > > > >> > >>>>>>>>>>>>>>>>>>>>>> to > > > > > >> > >>>>>>>>>>>>>>>>>>>>>>> an > > > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> executable (Python-, Rust-, ... ) > > member > > > > > >> > >>> of a > > > > > >> > >>>>>>>> library. > > > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> A running Graph might have a > different > > > > > >> > >>>>> structure > > > > > >> > >>>>>>>> than > > > > > >> > >>>>>>>>> a > > > > > >> > >>>>>>>>>>>>>>>>>>>>> configuration > > > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> Graph. > > > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> Forget that if you think it's > > bullshit. > > > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> Best Regards > > > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> Bernd Ströhle > > > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> M: +49 171 5357916 > > > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> E: bernd.stroe...@gmail.com > <mailto: > > > > > >> > >>>>>>>>>>> bernd.stroe...@gmail.com> > > > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> -----Original Message----- > > > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> From: Igor Kholopov > > > > > >> > >>> <ikholo...@google.com.inva > > > > > >> > >>>>>>>>> <mailto: > > > > > >> > >>>>>>>>>>> ikholo...@google.com.inva>LID> > > > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> Sent: Tuesday, October 22, 2024 > 12:02 > > PM > > > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> To: dev@airflow.apache.org <mailto: > > > > > >> > >>>>>>>>> dev@airflow.apache.org > > > > > >> > >>>>>>>>>>>> > > > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> Subject: Re: Airflow should > deprecate > > > the > > > > > >> > >>> term > > > > > >> > >>>>>>> "DAG" > > > > > >> > >>>>>>>>> for > > > > > >> > >>>>>>>>>>> end > > > > > >> > >>>>>>>>>>>>>>>>>>> users > > > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> Even though the term "DAG" is > clearly > > > > > >> > >>>>>> suboptimal, > > > > > >> > >>>>>>> it > > > > > >> > >>>>>>>>> is > > > > > >> > >>>>>>>>>>> part > > > > > >> > >>>>> >
