+1 would vote for Airflow's *D*ynamic *A*ction *G*raph (DAG) *Dynamic*: Reflects the capability of Airflow to handle workflows that aren’t strictly acyclic, but can adapt to changes, iterations and loops within broader context of workflow.
*Action*: Emphasizes that airflow is fundamentally about orchestrating actions or tasks. Whether its data processing, model training or any other form of computation. It also conveys operational aspect of what Airflow does. *Graph*: Maintains the core concept that workflows in Airflow are visualized and managed as graphs. This term keeps the idea of dependencies and relationships between tasks intact. Ambika. On Tue, Feb 18, 2025 at 11:46 AM Brent Bovenzi <br...@astronomer.io.invalid> wrote: > -0.5 > > Even though the new UI prefers "Dag", I don't think that means "DAG" is > incorrect. > > On Tue, Feb 18, 2025 at 11:36 AM Jarek Potiuk <ja...@potiuk.com> wrote: > > > I love "Dynamic Airflow Graph" +10 to that Constance. > > > > And yes. I fully agree with your "somewhat" acyclic statement. I saw > people > > mentioning "event-driven" workflows to better describe the AI/ML, but > that > > for me is a synonym of "completely unmanageable workflows" (similar to > > micro-services) - what I was alluding to was exactly this - something > like > > curren DAG with parts of it being potentially cyclic - so we are > completely > > aligned here :). And that's what make it "not DAG - in mathematical > sense" > > - even more reason to depart from it, > > > > J. > > > > On Tue, Feb 18, 2025 at 5:23 PM Constance Martineau > > <consta...@astronomer.io.invalid> wrote: > > > > > Hey all, > > > > > > I appreciate the discussion about how we position the concept of a DAG > in > > > Airflow. While I agree with updating documentation to de-emphasize the > > > strict mathematical definition, I do not think we should change the > > public > > > interface to use "dag" instead of "DAG" everywhere. It adds > > > unnecessary churn and breaks consistency across the ecosystem. > > > > > > Regarding the broader discussion on whether DAGs in Airflow should > remain > > > acyclic: I acknowledge that some workflows, particularly in ML and AI, > > > require cyclical behaviour (example: iterative refinement until a > > condition > > > is met). However, I think it's important to recognize that even in > those > > > cases, the workflow itself still has an overall start and end. It's not > > an > > > unbounded loop but rather a controlled execution process that happens > to > > > have cycles *within* it. > > > > > > That being said, I do see value in reinterpreting "DAG" to better align > > > with how Airflow is actually used, and love shifting away from > "Directed > > > Acyclic Graph" in favour of something more intuitive. However "Data > > Airflow > > > Graph" makes it sound like we're defining Airflow itself rather than > > > describing the structure of workflows within it. > > > > > > If we want a new meaning for DAG that resonates with our base while > > keeping > > > Airflow's identity clear, I am proposing three alternatives: > > > > > > > > > 1. *Dynamic Airflow Graph*: Ties directly to Airflow while > emphasizing > > > flexibility > > > 2. *Distributed Automation Graph*: Highlights execution across > systems > > > and automations without tying it strictly to data > > > 3. *Directed Activity Graph*: Retains the useful meaning of > "Directed" > > > while shifting away from strict acyclic constraints. (My personal > > > favourite) > > > > > > I think any shift in terminology should be driven by what best > > communicates > > > Airflow's role workflow automation, rather than just distancing > ourselves > > > from the mathematical definition. > > > > > > Curious to hear more thoughts! > > > > > > On Tue, Feb 18, 2025 at 10:40 AM Jarek Potiuk <ja...@potiuk.com> > wrote: > > > > > > > Sorry for the few typos ... My slight dyslexia did not help and > > > autocorrect > > > > did not help either this time : dept -> debt, routed -> rooted and a > > few > > > > other small typos :) . > > > > > > > > On Tue, Feb 18, 2025 at 4:28 PM Jarek Potiuk <ja...@potiuk.com> > wrote: > > > > > > > > > First of all - yes, agree with Lee and Ash, I think we do not need > to > > > > > change the public interface, I'd also be -0 on that, but docs, > > internal > > > > > strings, yes we could. > > > > > > > > > > But .... After thinking a bit and looking at the discussion here, > > where > > > > > people have concerns, I have an alternative take. And I know this > > might > > > > be > > > > > controversial, and some people might find it confusing, but why > don't > > > we > > > > > come up with a new acronym for DAG, one that will be more airflow > and > > > > less > > > > > "mathematics/algorithm". And I have, I think a good reason for it, > > and > > > I > > > > > even have a proposal. Bear with me, > > > > > > > > > > DAG => *Data Airflow Graph* > > > > > > > > > > As many of us mentioned, DAG and Airflow are almost 100% 1-1 now > > (with > > > > > exception of some niche products - you know which one I speak > about - > > > > which > > > > > are unfortunate enough to use the "Dag" in their name - I will > > explain > > > > why > > > > > this is unfortunate, IMHO). > > > > > > > > > > We are lucky to have Airflow for "A" in the acronym and we can make > > it > > > > > even more "synonymous" -> Airflow <> DAG with (A) being Airflow. > > > > > > > > > > Now, why I think "Directed Acyclic Graph" is unfortunate. > > > > > > > > > > Not only because "Directed" and "Acyclic" are mostly not in the > > > > vocabulary > > > > > of most of our users - as Ryan, starting this thread explained > yes, I > > > > agree > > > > > with it, but also I think there is a more important reason - I > > simply > > > > > believe in the future our DAGs might not (and likely will not) be > > > > "acyclic". > > > > > > > > > > I know for some it might be herezy and we have not discussed any of > > > that > > > > > yet, but more and more I keep on hearing that Machine Learning > > > Workflows > > > > of > > > > > the future are often somewhat cyclic. In our case there is one type > > of > > > > > repeat-in-cycle behaviour - we have "task retry on failure" - but > > this > > > is > > > > > only on failure and it basically mens "idempotent" retry - we want > to > > > > redo > > > > > something and get single result out of that and basically discard > the > > > > > previous result. But machine learning/AI workflows are different. I > > > had a > > > > > lot of conversations recently (I even co-organized "AI low-level > > > > > engineering and hacking" devroom to learn more about those) - and I > > > keep > > > > on > > > > > hearing that "Repeat a sequence of things several times and refine" > > is > > > > > precisely what AI / ML workflows will need - especially with > Agentic > > > > > workflows - often we have cooperating agents which **could** be > part > > of > > > > the > > > > > same DAG, but currently in Airflow where our DAGs are "Acyclic" we > > can > > > > > only express "repeat several time and refine" as a single task that > > > will > > > > do > > > > > such refining. If we would like to split such an "agent" doing > their > > > own > > > > > work in "cycles" and communicating with other "agents" doing the > > same - > > > > > represented as a multi-task-entity, we have no way to express it > > using > > > > our > > > > > current "acyclic" semantics. > > > > > > > > > > But - other than our implementation makes that assumption > currently - > > > > > there is nothing fundamental with Airflow being able to run such > > > > "partially > > > > > cyclic workflows". We have all the pieces, we would have to change > > > > > dependency calculation to allow for cycles, update UI to handle > > cycles > > > > > representation - and we could in theory quite seamlessly, support > > > > "cyclic" > > > > > workflows. > > > > > > > > > > I know, I know, that's the first time we hear it and that's huge > > > change - > > > > > of course, I do not want to start discussing any details of it, not > > > > before > > > > > Airflow 3.0 is out - but I think we should be open that at some > point > > > of > > > > > time the "Acyclic" property is going to go away, And my personal > > > > intuition > > > > > tells me that a year from now we will have acyclic workflows > > (assuming > > > > that > > > > > all the foundational work we do on Airflow 3 will really help in > > > > decreasing > > > > > all the technical dept, and will open our minds to new ideas). > > > > > > > > > > That's why I think - the sooner we get rid of "Directed Acyclic" - > > the > > > > > better. that will - even for us, maintainer free us from being > routed > > > in > > > > > the acyclic nature of Airflow. > > > > > > > > > > How about we change to "Data Airflow Graph" (or any other good > > acronym > > > > > which has (A)irflow instead of (A)cyclic). The name is just a > > proposal. > > > > > > > > > > Now.. I go hide somewhere as I believe that might cause a > "flamewar" > > > :), > > > > > unless on request of Ryan you already sheathed your swords in the > > > > > discussion (which seems everyone did). > > > > > > > > > > But honestly - I would really love what you think of "let's just > > break > > > > > with Acyclic NOW" and shape our future by using a different > acronym. > > > > > > > > > > J. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Feb 18, 2025 at 10:54 AM Ephraim Anierobi < > > > > > ephraimanier...@gmail.com> wrote: > > > > > > > > > >> When I started contributing to Airflow, I had to read up on DAG, > and > > > > now I > > > > >> know what it means. I wonder if we are about to have users who use > > the > > > > >> term > > > > >> Dag without knowing that there's a DAG. > > > > >> > > > > >> Slightly concerned. > > > > >> > > > > >> -ephraim > > > > >> > > > > >> On Tue, 18 Feb 2025 at 10:26, Wei Lee <weilee...@gmail.com> > wrote: > > > > >> > > > > >> > For user-facing things (e.g., `from airflow import DAG`), I’m > more > > > > like > > > > >> > -0. But for documentation, docstring, internal things and etc., > we > > > > >> probably > > > > >> > could still change most of them? > > > > >> > > > > > >> > Best, > > > > >> > Wei > > > > >> > > > > > >> > > On Feb 18, 2025, at 5:10 PM, Ash Berlin-Taylor < > a...@apache.org> > > > > >> wrote: > > > > >> > > > > > > >> > > I also don’t personally think it’s worth the pain (not to > > mention > > > > >> > backcompat workaround) to rename DAG to Dag, so I’d be -0.5 on > > that. > > > > >> > > > > > > >> > > -ash > > > > >> > > > > > > >> > >> On 18 Feb 2025, at 04:40, Daniel Imberman < > > > > daniel.imber...@gmail.com > > > > >> > > > > > >> > wrote: > > > > >> > >> > > > > >> > >> I think my biggest concern is a marketing one and not a > > technical > > > > >> one. > > > > >> > >> > > > > >> > >> As has been mentioned on the thread the terms airflow and dag > > are > > > > >> kind > > > > >> > of > > > > >> > >> synonymous and I certainly don’t want to give the impression > > that > > > > we > > > > >> are > > > > >> > >> breaking more than we are breaking. > > > > >> > >> > > > > >> > >> I wouldn’t die on this hill, but I’m slightly concerned. > > > > >> > >> > > > > >> > >> On Mon, Feb 17, 2025 at 6:00 PM Wei Lee <wei...@apache.org> > > > wrote: > > > > >> > >> > > > > >> > >>> I’m not sure about adding ruff rules here 🤔 I think ruff > > rules > > > > are > > > > >> > best > > > > >> > >>> suited for user-facing things but not the airflow code base > > > > itself. > > > > >> If > > > > >> > what > > > > >> > >>> we mean is adding a rule to avoid users using "DAG" *after* > we > > > > >> rename > > > > >> > it, > > > > >> > >>> it's definitely a +1000. > > > > >> > >>> > > > > >> > >>> I just created GitHub issues for this removing "DAG" > > > > >> > >>> > > > > >> > >>> https://github.com/apache/airflow/issues/46842 > > > > >> > >>> https://github.com/apache/airflow/issues/46843 > > > > >> > >>> > > > > >> > >>> One thing I'm not sure about is whether we want to get rid > of > > > > `from > > > > >> > >>> airflow import DAG` as well. 🤔 > > > > >> > >>> > > > > >> > >>> Best, > > > > >> > >>> Wei > > > > >> > >>> > > > > >> > >>> On 2025/02/17 19:50:57 Jarek Potiuk wrote: > > > > >> > >>>> Hard to say until it's looked at :) > > > > >> > >>>> > > > > >> > >>>> On Mon, Feb 17, 2025 at 8:45 PM Aritra Basu < > > > > >> aritrabasu1...@gmail.com > > > > >> > > > > > > >> > >>>> wrote: > > > > >> > >>>> > > > > >> > >>>>> I can take it up, it's mostly just a doc update right? Or > > are > > > we > > > > >> > doing > > > > >> > >>> code > > > > >> > >>>>> files replacement too? > > > > >> > >>>>> -- > > > > >> > >>>>> Regards, > > > > >> > >>>>> Aritra Basu > > > > >> > >>>>> > > > > >> > >>>>> On Tue, 18 Feb 2025, 12:55 am Jarek Potiuk, < > > ja...@potiuk.com > > > > > > > > >> > wrote: > > > > >> > >>>>> > > > > >> > >>>>>> Sounds like another +1000 files big PR is coming :) ? Any > > > > >> volunteers > > > > >> > >>> to > > > > >> > >>>>>> make it ? It's fun. > > > > >> > >>>>>> > > > > >> > >>>>>> On Mon, Feb 17, 2025 at 6:52 PM Omkar P < > > > > droiddev5...@gmail.com> > > > > >> > >>> wrote: > > > > >> > >>>>>> > > > > >> > >>>>>>> +1 for ruff rules :) > > > > >> > >>>>>>> > > > > >> > >>>>>>> Also would be nice to introduce 'Dag' to replace 'DAG' > in > > > the > > > > >> > >>> Airflow > > > > >> > >>>>>>> docs, in line with the new UI changes and to make the > > > renaming > > > > >> > >>>>>>> consistent across user-facing pages. > > > > >> > >>>>>>> > > > > >> > >>>>>>> Regards, > > > > >> > >>>>>>> Omkar > > > > >> > >>>>>>> > > > > >> > >>>>>>> On Mon, Feb 17, 2025 at 8:12 AM Jarek Potiuk < > > > > ja...@potiuk.com> > > > > >> > >>> wrote: > > > > >> > >>>>>>> > > > > >> > >>>>>>>> +1 :) . Maybe we could add a ruff rule for that :) > > > > >> > >>>>>>>> > > > > >> > >>>>>>>> On Mon, Feb 17, 2025 at 8:27 AM Wei Lee < > > wei...@apache.org > > > > > > > > >> > >>> wrote: > > > > >> > >>>>>>>> > > > > >> > >>>>>>>>> It seems that our current conclusion is to use "dag" > or > > > > "Dag" > > > > >> > >>>>> instead > > > > >> > >>>>>>> of > > > > >> > >>>>>>>>> "DAG" whenever possible. Should we replace all "DAG" > in > > > the > > > > >> > >>>>> codebase > > > > >> > >>>>>>> with > > > > >> > >>>>>>>>> "dag" or "Dag"? If it's too late for that (which it > > might > > > be > > > > >> > >>> 🤔), > > > > >> > >>>>>>> should > > > > >> > >>>>>>>> we > > > > >> > >>>>>>>>> at least avoid introducing new "DAG" in the following > > PRs? > > > > >> > >>>>>>>>> > > > > >> > >>>>>>>>> Best, > > > > >> > >>>>>>>>> Wei > > > > >> > >>>>>>>>> > > > > >> > >>>>>>>>> On 2024/10/23 17:16:55 Brent Bovenzi wrote: > > > > >> > >>>>>>>>>> Here's a PR to use "dag" as a word in the new 3.0 UI: > > > > >> > >>>>>>>>>> https://github.com/apache/airflow/pull/43325 > > > > >> > >>>>>>>>>> Let me know if that's the direction we want to go. > > > > >> > >>>>>>>>>> > > > > >> > >>>>>>>>>> On Wed, Oct 23, 2024 at 11:06 AM Bishundeo, Rajeshwar > > > > >> > >>>>>>>>>> <rbish...@amazon.com.invalid> wrote: > > > > >> > >>>>>>>>>> > > > > >> > >>>>>>>>>>> I think Brent & Daniel summarized it best, "dag" is > > > > >> > >>> synonymous > > > > >> > >>>>>> with > > > > >> > >>>>>>>>>>> workflows in Airflow through the way we talk and > > explain > > > > >> > >>> what > > > > >> > >>>>>>> Airflow > > > > >> > >>>>>>>>> is > > > > >> > >>>>>>>>>>> all about. Although folks would ask, I don’t ever > use > > > the > > > > >> > >>>>>>>> mathematical > > > > >> > >>>>>>>>>>> definition of DAG. > > > > >> > >>>>>>>>>>> It will be challenging and possibly confusing for > many > > > > >> > >>> users > > > > >> > >>>>>> making > > > > >> > >>>>>>>>> such a > > > > >> > >>>>>>>>>>> change - I would rather direct that energy to > > appending > > > > the > > > > >> > >>>>>> Oxford > > > > >> > >>>>>>>>>>> definition of "dag" to include a reference to > > workflows > > > in > > > > >> > >>>>>> Airflow. > > > > >> > >>>>>>>> __ > > > > >> > >>>>>>>>>>> > > > > >> > >>>>>>>>>>> -- Rajesh > > > > >> > >>>>>>>>>>> > > > > >> > >>>>>>>>>>> > > > > >> > >>>>>>>>>>> > > > > >> > >>>>>>>>>>> > > > > >> > >>>>>>>>>>> > > > > >> > >>>>>>>>>>> > > > > >> > >>>>>>>>>>> On 2024-10-23, 3:37 AM, "Jarek Potiuk" < > > > ja...@potiuk.com > > > > >> > >>>>>> <mailto: > > > > >> > >>>>>>>>>>> ja...@potiuk.com>> wrote: > > > > >> > >>>>>>>>>>> > > > > >> > >>>>>>>>>>> > > > > >> > >>>>>>>>>>> CAUTION: This email originated from outside of the > > > > >> > >>>>> organization. > > > > >> > >>>>>> Do > > > > >> > >>>>>>>> not > > > > >> > >>>>>>>>>>> click links or open attachments unless you can > confirm > > > the > > > > >> > >>>>> sender > > > > >> > >>>>>>> and > > > > >> > >>>>>>>>> know > > > > >> > >>>>>>>>>>> the content is safe. > > > > >> > >>>>>>>>>>> > > > > >> > >>>>>>>>>>> > > > > >> > >>>>>>>>>>> > > > > >> > >>>>>>>>>>> > > > > >> > >>>>>>>>>>> > > > > >> > >>>>>>>>>>> > > > > >> > >>>>>>>>>>> AVERTISSEMENT: Ce courrier électronique provient > d’un > > > > >> > >>>>> expéditeur > > > > >> > >>>>>>>>> externe. > > > > >> > >>>>>>>>>>> Ne cliquez sur aucun lien et n’ouvrez aucune pièce > > > jointe > > > > >> > >>> si > > > > >> > >>>>> vous > > > > >> > >>>>>>> ne > > > > >> > >>>>>>>>> pouvez > > > > >> > >>>>>>>>>>> pas confirmer l’identité de l’expéditeur et si vous > > > n’êtes > > > > >> > >>> pas > > > > >> > >>>>>>>> certain > > > > >> > >>>>>>>>> que > > > > >> > >>>>>>>>>>> le contenu ne présente aucun risque. > > > > >> > >>>>>>>>>>> > > > > >> > >>>>>>>>>>> > > > > >> > >>>>>>>>>>> > > > > >> > >>>>>>>>>>> > > > > >> > >>>>>>>>>>> > > > > >> > >>>>>>>>>>> > > > > >> > >>>>>>>>>>> From Guido's post: > > > > >> > >>>>>>>>>>> > > > > >> > >>>>>>>>>>> > > > > >> > >>>>>>>>>>> "Around this time the renaming seems to have been > > > > renamed". > > > > >> > >>>>>>>>>>> > > > > >> > >>>>>>>>>>> > > > > >> > >>>>>>>>>>> Naming is hard. > > > > >> > >>>>>>>>>>> > > > > >> > >>>>>>>>>>> > > > > >> > >>>>>>>>>>> J. > > > > >> > >>>>>>>>>>> > > > > >> > >>>>>>>>>>> > > > > >> > >>>>>>>>>>> > > > > >> > >>>>>>>>>>> > > > > >> > >>>>>>>>>>> On Wed, Oct 23, 2024 at 7:11 AM Omkar P < > > > > >> > >>>>> droiddev5...@gmail.com > > > > >> > >>>>>>>>> <mailto: > > > > >> > >>>>>>>>>>> droiddev5...@gmail.com>> wrote: > > > > >> > >>>>>>>>>>> > > > > >> > >>>>>>>>>>> > > > > >> > >>>>>>>>>>>> Yes, completely agree with above comments, dag is > an > > > > >> > >>> Airflow > > > > >> > >>>>>> term > > > > >> > >>>>>>>> now > > > > >> > >>>>>>>>>>>> rather > > > > >> > >>>>>>>>>>>> than just a "directed acyclic graph". Believe it or > > > not, > > > > >> > >>> I've > > > > >> > >>>>>>>> worked > > > > >> > >>>>>>>>> with > > > > >> > >>>>>>>>>>>> work > > > > >> > >>>>>>>>>>>> folks who've been using dags in Airflow for years > now > > > > >> > >>> and are > > > > >> > >>>>>> pro > > > > >> > >>>>>>>>> devs > > > > >> > >>>>>>>>>>> but > > > > >> > >>>>>>>>>>>> have trouble remembering the full form of a DAG! > For > > > > >> > >>> them, > > > > >> > >>>>> dag > > > > >> > >>>>>> = > > > > >> > >>>>>>>>> Airflow. > > > > >> > >>>>>>>>>>>> > > > > >> > >>>>>>>>>>>> workflow, pipeline, flow are surely better terms > but > > > > >> > >>> will be > > > > >> > >>>>> a > > > > >> > >>>>>>>> major > > > > >> > >>>>>>>>>>>> behavior > > > > >> > >>>>>>>>>>>> change for the developer community, so we'll need a > > > solid > > > > >> > >>>>> plan > > > > >> > >>>>>> on > > > > >> > >>>>>>>>> how to > > > > >> > >>>>>>>>>>>> introduce it, when we do. > > > > >> > >>>>>>>>>>>> > > > > >> > >>>>>>>>>>>> While we discuss this, I'd like to share about the > > > Great > > > > >> > >>>>>> Renaming > > > > >> > >>>>>>>> in > > > > >> > >>>>>>>>> core > > > > >> > >>>>>>>>>>>> Python started by Guido back in 2009. We could > > probably > > > > >> > >>> get > > > > >> > >>>>>> some > > > > >> > >>>>>>>>>>> learnings > > > > >> > >>>>>>>>>>>> from there? Who knows! > > > > >> > >>>>>>>>>>>> > > > > >> > >>>>>>>>>>>> Guido's blog (2009): > > > > >> > >>>>>>>>>>>> > > > > >> > >>>>>>>>> > > > > >> > >>>>>>> > > > > >> > >>>>> > > > > >> > >>> > > > > >> > > > > > >> > > > > > > https://python-history.blogspot.com/2009/03/great-or-grand-renaming.html > > > > >> > >>>>>>>>>>> < > > > > >> > >>>>>>>>> > > > > >> > >>>>>>> > > > > >> > >>>>> > > > > >> > >>> > > > > >> > > > > > >> > > > > > > https://python-history.blogspot.com/2009/03/great-or-grand-renaming.html > > > > >> > >>>>>>>>> > > > > >> > >>>>>>>>>>>> Follow-up discussion (2024): > > > > >> > >>>>>>>>>>>> > > > > >> > >>>>>> > > > > https://discuss.python.org/t/finishing-the-great-renaming/54082 > > > > >> > >>>>>>> < > > > > >> > >>>>>>>>>>> > > > > >> > >>>>> > > > https://discuss.python.org/t/finishing-the-great-renaming/54082 > > > > > > > > > >> > >>>>>>>>>>>> > > > > >> > >>>>>>>>>>>> Cheers, > > > > >> > >>>>>>>>>>>> Omkar > > > > >> > >>>>>>>>>>>> > > > > >> > >>>>>>>>>>>> On Wed, Oct 23, 2024 at 7:10 AM Wei Lee < > > > > >> > >>> weilee...@gmail.com > > > > >> > >>>>>>>>> <mailto: > > > > >> > >>>>>>>>>>> weilee...@gmail.com>> wrote: > > > > >> > >>>>>>>>>>>> > > > > >> > >>>>>>>>>>>>> I think we should probably just accept it as an > > > airflow > > > > >> > >>>>> term. > > > > >> > >>>>>>> At > > > > >> > >>>>>>>>> least, > > > > >> > >>>>>>>>>>>>> that’s how I understood it when I first used > > Airflow. > > > I > > > > >> > >>>>> feel > > > > >> > >>>>>>>>> renaming > > > > >> > >>>>>>>>>>> it > > > > >> > >>>>>>>>>>>> at > > > > >> > >>>>>>>>>>>>> this stage would require considerable effort from > > > > >> > >>>>> maintainers > > > > >> > >>>>>>> and > > > > >> > >>>>>>>>>>>> existing > > > > >> > >>>>>>>>>>>>> users without providing equivalent benefits. > > > > >> > >>>>>>>>>>>>> > > > > >> > >>>>>>>>>>>>> Best, > > > > >> > >>>>>>>>>>>>> Wei > > > > >> > >>>>>>>>>>>>> > > > > >> > >>>>>>>>>>>>>> On Oct 23, 2024, at 8:01 AM, Kaxil Naik < > > > > >> > >>>>>> kaxiln...@gmail.com > > > > >> > >>>>>>>>>>> <mailto:kaxiln...@gmail.com>> wrote: > > > > >> > >>>>>>>>>>>>>> > > > > >> > >>>>>>>>>>>>>> Same agreed with Brent & Daniel -- maybe we > > re-kindle > > > > >> > >>>>> this > > > > >> > >>>>>>>>> discussion > > > > >> > >>>>>>>>>>>> for > > > > >> > >>>>>>>>>>>>>> Airflow 4 :) -- but right now it will cause too > > much > > > > >> > >>>>>>> disruption > > > > >> > >>>>>>>>>>>>>> > > > > >> > >>>>>>>>>>>>>> On Tue, 22 Oct 2024 at 21:27, Constance Martineau > > > > >> > >>>>>>>>>>>>>> <consta...@astronomer.io.inva <mailto: > > > > >> > >>>>>>>>> consta...@astronomer.io.inva>lid> > > > > >> > >>>>>>>>>>> wrote: > > > > >> > >>>>>>>>>>>>>> > > > > >> > >>>>>>>>>>>>>>> In my experience, when you ask those with > Airflow > > > > >> > >>>>>> experience > > > > >> > >>>>>>>>> what a > > > > >> > >>>>>>>>>>>> dag > > > > >> > >>>>>>>>>>>>> is, > > > > >> > >>>>>>>>>>>>>>> they'll start talking about workflow attributes > - > > > > >> > >>> stuff > > > > >> > >>>>>> like > > > > >> > >>>>>>>>> dags > > > > >> > >>>>>>>>>>>> being > > > > >> > >>>>>>>>>>>>> a > > > > >> > >>>>>>>>>>>>>>> series of steps or tasks with owners. The > > structure > > > > >> > >>>>>> doesn't > > > > >> > >>>>>>>>> come up. > > > > >> > >>>>>>>>>>>>>>> > > > > >> > >>>>>>>>>>>>>>> Echo-ing others, at this point, my vote is to > > > > >> > >>> embrace > > > > >> > >>>>> the > > > > >> > >>>>>>> name > > > > >> > >>>>>>>>> and > > > > >> > >>>>>>>>>>>>>>> de-emphasize the mathematical structure aspect. > > > > >> > >>>>>>>>>>>>>>> > > > > >> > >>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 3:47 PM Vikram Koka > > > > >> > >>>>>>>>>>>>> <vik...@astronomer.io.inva <mailto: > > > > >> > >>>>> vik...@astronomer.io.inva > > > > >> > >>>>>>>>> lid> > > > > >> > >>>>>>>>>>>>>>> wrote: > > > > >> > >>>>>>>>>>>>>>> > > > > >> > >>>>>>>>>>>>>>>> It's an interesting discussion and I remember > > > > >> > >>>>> struggling > > > > >> > >>>>>>> with > > > > >> > >>>>>>>>> this > > > > >> > >>>>>>>>>>>>> when I > > > > >> > >>>>>>>>>>>>>>>> started working with Airflow. > > > > >> > >>>>>>>>>>>>>>>> But, I also agree with the viewpoint of it > being > > an > > > > >> > >>>>>>>> established > > > > >> > >>>>>>>>>>>> concept > > > > >> > >>>>>>>>>>>>>>> now > > > > >> > >>>>>>>>>>>>>>>> regardless of the origin. > > > > >> > >>>>>>>>>>>>>>>> > > > > >> > >>>>>>>>>>>>>>>> I am personally leaning towards the perspective > > > > >> > >>> best > > > > >> > >>>>>>>> expressed > > > > >> > >>>>>>>>> by > > > > >> > >>>>>>>>>>>>> Daniel > > > > >> > >>>>>>>>>>>>>>>> Standish and Brent of using Dag as a word, > rather > > > > >> > >>> than > > > > >> > >>>>>> the > > > > >> > >>>>>>>>> computer > > > > >> > >>>>>>>>>>>>>>> science > > > > >> > >>>>>>>>>>>>>>>> concept. > > > > >> > >>>>>>>>>>>>>>>> > > > > >> > >>>>>>>>>>>>>>>> Best regards, > > > > >> > >>>>>>>>>>>>>>>> Vikram > > > > >> > >>>>>>>>>>>>>>>> > > > > >> > >>>>>>>>>>>>>>>> > > > > >> > >>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 9:46 AM Oliveira, Niko > > > > >> > >>>>>>>>>>>>>>> <oniko...@amazon.com.inva <mailto: > > > > >> > >>>>>> oniko...@amazon.com.inva > > > > >> > >>>>>>>>> lid > > > > >> > >>>>>>>>>>>>>>>>> > > > > >> > >>>>>>>>>>>>>>>> wrote: > > > > >> > >>>>>>>>>>>>>>>> > > > > >> > >>>>>>>>>>>>>>>>> I agree with the general sentiment of: You're > > > > >> > >>> right > > > > >> > >>>>>> Ryan, > > > > >> > >>>>>>>> DAG > > > > >> > >>>>>>>>>>> isn't > > > > >> > >>>>>>>>>>>>>>> great > > > > >> > >>>>>>>>>>>>>>>>> and I'd rather workflow, but changing it will > > > > >> > >>> cause > > > > >> > >>>>> much > > > > >> > >>>>>>>> more > > > > >> > >>>>>>>>>>>> wreckage > > > > >> > >>>>>>>>>>>>>>>> than > > > > >> > >>>>>>>>>>>>>>>>> it solves. > > > > >> > >>>>>>>>>>>>>>>>> > > > > >> > >>>>>>>>>>>>>>>>> Also agree with the idea to just move away > from > > > > >> > >>>>> defining > > > > >> > >>>>>>>> DAG. > > > > >> > >>>>>>>>> I > > > > >> > >>>>>>>>>>>> think > > > > >> > >>>>>>>>>>>>>>>>> we've been naturally doing that as a community > > > > >> > >>> for a > > > > >> > >>>>>> while > > > > >> > >>>>>>>> now > > > > >> > >>>>>>>>>>>> anyway, > > > > >> > >>>>>>>>>>>>>>> so > > > > >> > >>>>>>>>>>>>>>>>> that feels like a natural step. > > > > >> > >>>>>>>>>>>>>>>>> > > > > >> > >>>>>>>>>>>>>>>>> Cheers, > > > > >> > >>>>>>>>>>>>>>>>> Niko > > > > >> > >>>>>>>>>>>>>>>>> > > > > >> > >>>>>>>>>>>>>>>>> ________________________________ > > > > >> > >>>>>>>>>>>>>>>>> From: Ash Berlin-Taylor <a...@apache.org > > <mailto: > > > > >> > >>>>>>>>> a...@apache.org>> > > > > >> > >>>>>>>>>>>>>>>>> Sent: Tuesday, October 22, 2024 9:06:39 AM > > > > >> > >>>>>>>>>>>>>>>>> To: dev@airflow.apache.org <mailto: > > > > >> > >>>>>> dev@airflow.apache.org > > > > >> > >>>>>>>> > > > > >> > >>>>>>>>>>>>>>>>> Subject: RE: [EXT] Airflow should deprecate > the > > > > >> > >>> term > > > > >> > >>>>>> "DAG" > > > > >> > >>>>>>>>> for end > > > > >> > >>>>>>>>>>>>>>> users > > > > >> > >>>>>>>>>>>>>>>>> > > > > >> > >>>>>>>>>>>>>>>>> CAUTION: This email originated from outside of > > the > > > > >> > >>>>>>>>> organization. > > > > >> > >>>>>>>>>>> Do > > > > >> > >>>>>>>>>>>>> not > > > > >> > >>>>>>>>>>>>>>>>> click links or open attachments unless you can > > > > >> > >>> confirm > > > > >> > >>>>>> the > > > > >> > >>>>>>>>> sender > > > > >> > >>>>>>>>>>>> and > > > > >> > >>>>>>>>>>>>>>>> know > > > > >> > >>>>>>>>>>>>>>>>> the content is safe. > > > > >> > >>>>>>>>>>>>>>>>> > > > > >> > >>>>>>>>>>>>>>>>> > > > > >> > >>>>>>>>>>>>>>>>> > > > > >> > >>>>>>>>>>>>>>>>> AVERTISSEMENT: Ce courrier électronique > provient > > > > >> > >>> d’un > > > > >> > >>>>>>>>> expéditeur > > > > >> > >>>>>>>>>>>>>>> externe. > > > > >> > >>>>>>>>>>>>>>>>> Ne cliquez sur aucun lien et n’ouvrez aucune > > pièce > > > > >> > >>>>>> jointe > > > > >> > >>>>>>> si > > > > >> > >>>>>>>>> vous > > > > >> > >>>>>>>>>>> ne > > > > >> > >>>>>>>>>>>>>>>> pouvez > > > > >> > >>>>>>>>>>>>>>>>> pas confirmer l’identité de l’expéditeur et si > > > > >> > >>> vous > > > > >> > >>>>>> n’êtes > > > > >> > >>>>>>>> pas > > > > >> > >>>>>>>>>>>> certain > > > > >> > >>>>>>>>>>>>>>>> que > > > > >> > >>>>>>>>>>>>>>>>> le contenu ne présente aucun risque. > > > > >> > >>>>>>>>>>>>>>>>> > > > > >> > >>>>>>>>>>>>>>>>> > > > > >> > >>>>>>>>>>>>>>>>> > > > > >> > >>>>>>>>>>>>>>>>> Best argument in favour of keeping “dags” as a > > > > >> > >>> term — > > > > >> > >>>>>>>> getting > > > > >> > >>>>>>>>> to > > > > >> > >>>>>>>>>>>>> re-use > > > > >> > >>>>>>>>>>>>>>>>> puns like https://i.imgflip.com/1xhtwh.jpg < > > > > >> > >>>>>>>>>>> https://i.imgflip.com/1xhtwh.jpg> > > > > >> > >>>>>>>>>>>>>>>>> > > > > >> > >>>>>>>>>>>>>>>>> In all seriousness: I don’t mind either way, > > both > > > > >> > >>>>> sides > > > > >> > >>>>>>> have > > > > >> > >>>>>>>>> good > > > > >> > >>>>>>>>>>>>>>> reasons > > > > >> > >>>>>>>>>>>>>>>>> presented. > > > > >> > >>>>>>>>>>>>>>>>> > > > > >> > >>>>>>>>>>>>>>>>> -a > > > > >> > >>>>>>>>>>>>>>>>> > > > > >> > >>>>>>>>>>>>>>>>>> On 22 Oct 2024, at 17:03, Daniel Standish > > > > >> > >>>>>>>>>>>>>>>>> <daniel.stand...@astronomer.io.inva <mailto: > > > > >> > >>>>>>>>>>> daniel.stand...@astronomer.io.inva>LID> wrote: > > > > >> > >>>>>>>>>>>>>>>>>> > > > > >> > >>>>>>>>>>>>>>>>>> Yeah just say, when asked where the name > comes > > > > >> > >>> from, > > > > >> > >>>>>>> "well, > > > > >> > >>>>>>>>> no > > > > >> > >>>>>>>>>>> one > > > > >> > >>>>>>>>>>>>>>>>> actually > > > > >> > >>>>>>>>>>>>>>>>>> knows but..." and then make something up. > > > > >> > >>>>>>>>>>>>>>>>>> > > > > >> > >>>>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 8:31 AM Jarek Potiuk > < > > > > >> > >>>>>>>>> ja...@potiuk.com > > > > >> > >>>>>>>>>>> <mailto:ja...@potiuk.com>> > > > > >> > >>>>>>>>>>>>>>> wrote: > > > > >> > >>>>>>>>>>>>>>>>>> > > > > >> > >>>>>>>>>>>>>>>>>>> Just to clarify - "directed acyclic graph" > is > > > > >> > >>> the > > > > >> > >>>>>>>>>>> tongue-twister, > > > > >> > >>>>>>>>>>>>>>>>>>> > > > > >> > >>>>>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 5:29 PM Jarek > Potiuk < > > > > >> > >>>>>>>>> ja...@potiuk.com > > > > >> > >>>>>>>>>>> <mailto:ja...@potiuk.com>> > > > > >> > >>>>>>>>>>>>>>>> wrote: > > > > >> > >>>>>>>>>>>>>>>>>>> > > > > >> > >>>>>>>>>>>>>>>>>>>> I like what both Daniel and Brent wrote. I > > > > >> > >>> would > > > > >> > >>>>> very > > > > >> > >>>>>>>> much > > > > >> > >>>>>>>>> want > > > > >> > >>>>>>>>>>>> to > > > > >> > >>>>>>>>>>>>>>> be > > > > >> > >>>>>>>>>>>>>>>>>>> able > > > > >> > >>>>>>>>>>>>>>>>>>>> to say just "dag" without explaining it > > > > >> > >>> further. > > > > >> > >>>>>>>>>>>>>>>>>>>> > > > > >> > >>>>>>>>>>>>>>>>>>>> For me every time I explain "DAG" at a talk > > > > >> > >>> it's a > > > > >> > >>>>>>>>>>>> tongue-twister, > > > > >> > >>>>>>>>>>>>>>>> and > > > > >> > >>>>>>>>>>>>>>>>> I > > > > >> > >>>>>>>>>>>>>>>>>>>> almost stutter on trying to recall how to > > > > >> > >>> pronounce > > > > >> > >>>>>> it > > > > >> > >>>>>>>>>>> properly. > > > > >> > >>>>>>>>>>>>>>>>>>>> > > > > >> > >>>>>>>>>>>>>>>>>>>> J. > > > > >> > >>>>>>>>>>>>>>>>>>>> > > > > >> > >>>>>>>>>>>>>>>>>>>> > > > > >> > >>>>>>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 5:27 PM Brent > Bovenzi > > > > >> > >>>>>>>>>>>>>>>>>>> <br...@astronomer.io.inva <mailto: > > > > >> > >>>>>>>> br...@astronomer.io.inva > > > > >> > >>>>>>>>>> lid> > > > > >> > >>>>>>>>>>>>>>>>>>>> wrote: > > > > >> > >>>>>>>>>>>>>>>>>>>> > > > > >> > >>>>>>>>>>>>>>>>>>>>> I remember we explored renaming "DAG" when > > > > >> > >>>>> starting > > > > >> > >>>>>> on > > > > >> > >>>>>>>>> AIP-38 > > > > >> > >>>>>>>>>>> to > > > > >> > >>>>>>>>>>>>>>>>>>> modernize > > > > >> > >>>>>>>>>>>>>>>>>>>>> the UI. Both "pipeline" or "workflow" are > > more > > > > >> > >>>>>>>>> descriptive of > > > > >> > >>>>>>>>>>>> what > > > > >> > >>>>>>>>>>>>>>>> one > > > > >> > >>>>>>>>>>>>>>>>>>> is > > > > >> > >>>>>>>>>>>>>>>>>>>>> actually doing while Directed Acyclic > Graph > > > > >> > >>> is an > > > > >> > >>>>>>>>>>> implementation > > > > >> > >>>>>>>>>>>>>>>>> detail. > > > > >> > >>>>>>>>>>>>>>>>>>>>> But I agree with Daniel Standish, at this > > > > >> > >>> point > > > > >> > >>>>>> "DAG" > > > > >> > >>>>>>>> has > > > > >> > >>>>>>>>>>> become > > > > >> > >>>>>>>>>>>>>>>> "dag" > > > > >> > >>>>>>>>>>>>>>>>>>> , a > > > > >> > >>>>>>>>>>>>>>>>>>>>> word in its own right. > > > > >> > >>>>>>>>>>>>>>>>>>>>> > > > > >> > >>>>>>>>>>>>>>>>>>>>> Examples for "dag" are abound in community > > > > >> > >>>>>> discussion, > > > > >> > >>>>>>>>> Airflow > > > > >> > >>>>>>>>>>>>>>>> Summit > > > > >> > >>>>>>>>>>>>>>>>>>>>> talks, documentation and even in the UI. > > Let's > > > > >> > >>>>>> embrace > > > > >> > >>>>>>>>> "dag". > > > > >> > >>>>>>>>>>> A > > > > >> > >>>>>>>>>>>>>>> user > > > > >> > >>>>>>>>>>>>>>>>>>> just > > > > >> > >>>>>>>>>>>>>>>>>>>>> needs to learn one new word vs the > technical > > > > >> > >>>>> concept > > > > >> > >>>>>>>>> behind > > > > >> > >>>>>>>>>>> that > > > > >> > >>>>>>>>>>>>>>>>> word. I > > > > >> > >>>>>>>>>>>>>>>>>>>>> think that is much less effort than > > > > >> > >>> refactoring so > > > > >> > >>>>>>> much > > > > >> > >>>>>>>>> code, > > > > >> > >>>>>>>>>>>>>>>>>>>>> documentation, blog posts, stack overflow > > > > >> > >>>>> questions, > > > > >> > >>>>>>>> etc. > > > > >> > >>>>>>>>>>>>>>>>>>>>> > > > > >> > >>>>>>>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 10:51 AM Daniel > > > > >> > >>> Standish > > > > >> > >>>>>>>>>>>>>>>>>>>>> <daniel.stand...@astronomer.io.inva > > <mailto: > > > > >> > >>>>>>>>>>> daniel.stand...@astronomer.io.inva>lid> wrote: > > > > >> > >>>>>>>>>>>>>>>>>>>>> > > > > >> > >>>>>>>>>>>>>>>>>>>>>> I am skeptical. Seems like introducing a > > lot > > > > >> > >>> of > > > > >> > >>>>>> pain > > > > >> > >>>>>>>> for > > > > >> > >>>>>>>>>>>>>>>>> questionable > > > > >> > >>>>>>>>>>>>>>>>>>>>>> benefit. But, I am def sympathetic to the > > > > >> > >>> idea. I > > > > >> > >>>>>>> agree > > > > >> > >>>>>>>>> the > > > > >> > >>>>>>>>>>>>>>>>>>>>> association > > > > >> > >>>>>>>>>>>>>>>>>>>>>> with "directed acyclic graph" is not > > helpful. > > > > >> > >>>>>>>>>>>>>>>>>>>>>> > > > > >> > >>>>>>>>>>>>>>>>>>>>>> And along those lines, I offer here some > > less > > > > >> > >>>>>>> invasive > > > > >> > >>>>>>>>>>>>>>> mitigations. > > > > >> > >>>>>>>>>>>>>>>>>>>>>> > > > > >> > >>>>>>>>>>>>>>>>>>>>>> One thing we can do no matter what is to > > > > >> > >>>>>> de-emphasize > > > > >> > >>>>>>>> the > > > > >> > >>>>>>>>>>> math > > > > >> > >>>>>>>>>>>>>>> nerd > > > > >> > >>>>>>>>>>>>>>>>>>>>> origins > > > > >> > >>>>>>>>>>>>>>>>>>>>>> of the name. That is to say, in docs / > > > > >> > >>> website / > > > > >> > >>>>>> etc, > > > > >> > >>>>>>>>> *never > > > > >> > >>>>>>>>>>>>>>>> define* > > > > >> > >>>>>>>>>>>>>>>>>>>>>> airflow's "dag" concept as a directed > > acyclic > > > > >> > >>>>>> graph. > > > > >> > >>>>>>>>> Always > > > > >> > >>>>>>>>>>>>>>> define > > > > >> > >>>>>>>>>>>>>>>>> it > > > > >> > >>>>>>>>>>>>>>>>>>>>> as a > > > > >> > >>>>>>>>>>>>>>>>>>>>>> pipeline, collection of tasks, workflow > > etc. > > > > >> > >>>>>>>>>>>>>>>>>>>>>> > > > > >> > >>>>>>>>>>>>>>>>>>>>>> The "directed acyclic graph" part of it > is > > > > >> > >>> like a > > > > >> > >>>>>>>>> historical > > > > >> > >>>>>>>>>>>>>>>>> footnote, > > > > >> > >>>>>>>>>>>>>>>>>>>>> and > > > > >> > >>>>>>>>>>>>>>>>>>>>>> we could make one mention of it somewhere > > > > >> > >>> hidden. > > > > >> > >>>>>>>>>>>>>>>>>>>>>> > > > > >> > >>>>>>>>>>>>>>>>>>>>>> We could also start using lowercase in > the > > > > >> > >>> docs > > > > >> > >>>>> in > > > > >> > >>>>>>>>> general > > > > >> > >>>>>>>>>>> e.g. > > > > >> > >>>>>>>>>>>>>>>>>>> writing > > > > >> > >>>>>>>>>>>>>>>>>>>>>> "dag" / "dags" instead of writing "DAG" / > > > > >> > >>> "DAGs" > > > > >> > >>>>>> etc. > > > > >> > >>>>>>>> The > > > > >> > >>>>>>>>>>>> upper > > > > >> > >>>>>>>>>>>>>>>> case > > > > >> > >>>>>>>>>>>>>>>>>>>>> part > > > > >> > >>>>>>>>>>>>>>>>>>>>>> of it makes it look like an acronym; but > > > > >> > >>> "dag" in > > > > >> > >>>>>>>>> airlfow is > > > > >> > >>>>>>>>>>>> just > > > > >> > >>>>>>>>>>>>>>>> an > > > > >> > >>>>>>>>>>>>>>>>>>>>>> airflow concept and the association with > > > > >> > >>> "DAGs" > > > > >> > >>>>> is > > > > >> > >>>>>>> not > > > > >> > >>>>>>>>> really > > > > >> > >>>>>>>>>>>>>>>>>>> unhelpful. > > > > >> > >>>>>>>>>>>>>>>>>>>>>> > > > > >> > >>>>>>>>>>>>>>>>>>>>>> In other words embrace that "dag" in > > airflow > > > > >> > >>> is > > > > >> > >>>>> its > > > > >> > >>>>>>> own > > > > >> > >>>>>>>>>>> thing, > > > > >> > >>>>>>>>>>>> is > > > > >> > >>>>>>>>>>>>>>>>>>>>>> *not* strictly > > > > >> > >>>>>>>>>>>>>>>>>>>>>> speaking a directed acyclic graph (which > > > > >> > >>> nobody > > > > >> > >>>>>> knows > > > > >> > >>>>>>>>> about > > > > >> > >>>>>>>>>>>>>>>> anyway), > > > > >> > >>>>>>>>>>>>>>>>>>> and > > > > >> > >>>>>>>>>>>>>>>>>>>>>> tell them what it is in simple terms that > > > > >> > >>> normal > > > > >> > >>>>>>> people > > > > >> > >>>>>>>>>>>>>>> understand. > > > > >> > >>>>>>>>>>>>>>>>>>>>>> > > > > >> > >>>>>>>>>>>>>>>>>>>>>> > > > > >> > >>>>>>>>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 7:27 AM Jarek > > Potiuk > > > > >> > >>> < > > > > >> > >>>>>>>>>>> ja...@potiuk.com <mailto:ja...@potiuk.com> > > > > >> > >>>>>>>>>>>>> > > > > >> > >>>>>>>>>>>>>>>>>>> wrote: > > > > >> > >>>>>>>>>>>>>>>>>>>>>> > > > > >> > >>>>>>>>>>>>>>>>>>>>>>> DAG is so embedded into what we do that > it > > > > >> > >>> will > > > > >> > >>>>> be > > > > >> > >>>>>>>>> extremely > > > > >> > >>>>>>>>>>>>>>>>>>>>> difficult to > > > > >> > >>>>>>>>>>>>>>>>>>>>>>> get rid of it completely. Also I think > it > > > > >> > >>> will > > > > >> > >>>>>> make > > > > >> > >>>>>>> a > > > > >> > >>>>>>>>> lot of > > > > >> > >>>>>>>>>>>>>>>>>>> "google" > > > > >> > >>>>>>>>>>>>>>>>>>>>>>> searches and "stack overflow" searches > not > > > > >> > >>>>> finding > > > > >> > >>>>>>> the > > > > >> > >>>>>>>>> right > > > > >> > >>>>>>>>>>>>>>>>>>> answers. > > > > >> > >>>>>>>>>>>>>>>>>>>>>> This > > > > >> > >>>>>>>>>>>>>>>>>>>>>>> is one of the strengths of Airflow - > > > > >> > >>> besides the > > > > >> > >>>>>>>>> community > > > > >> > >>>>>>>>>>> and > > > > >> > >>>>>>>>>>>>>>>> ideas > > > > >> > >>>>>>>>>>>>>>>>>>>>> that > > > > >> > >>>>>>>>>>>>>>>>>>>>>>> Bernd mentioned - is the vast number of > > > > >> > >>>>> examples, > > > > >> > >>>>>>>>> problems > > > > >> > >>>>>>>>>>> and > > > > >> > >>>>>>>>>>>>>>>>>>>>> solutions > > > > >> > >>>>>>>>>>>>>>>>>>>>>>> you can so easily find (and we have to > > > > >> > >>> remember > > > > >> > >>>>>> that > > > > >> > >>>>>>>>> all the > > > > >> > >>>>>>>>>>>> AI > > > > >> > >>>>>>>>>>>>>>>>>>>>> trained > > > > >> > >>>>>>>>>>>>>>>>>>>>>> on > > > > >> > >>>>>>>>>>>>>>>>>>>>>>> past data will be also rather poorly > > > > >> > >>> matching > > > > >> > >>>>>>> queries > > > > >> > >>>>>>>> of > > > > >> > >>>>>>>>>>>> people. > > > > >> > >>>>>>>>>>>>>>>>>>>>>>> > > > > >> > >>>>>>>>>>>>>>>>>>>>>>> I am not too attached to DAG. I could > > easily > > > > >> > >>>>>> switch. > > > > >> > >>>>>>>>> And if > > > > >> > >>>>>>>>>>> we > > > > >> > >>>>>>>>>>>>>>> do > > > > >> > >>>>>>>>>>>>>>>> - > > > > >> > >>>>>>>>>>>>>>>>>>> I > > > > >> > >>>>>>>>>>>>>>>>>>>>>>> would be for using workflow or pipeline > > > > >> > >>> instead > > > > >> > >>>>> of > > > > >> > >>>>>>>>> `dag` if > > > > >> > >>>>>>>>>>>> not > > > > >> > >>>>>>>>>>>>>>>> the > > > > >> > >>>>>>>>>>>>>>>>>>>>> above > > > > >> > >>>>>>>>>>>>>>>>>>>>>>> reason, but I think I am here with Igor > > > > >> > >>> that it > > > > >> > >>>>>>> might > > > > >> > >>>>>>>>> cause > > > > >> > >>>>>>>>>>>> more > > > > >> > >>>>>>>>>>>>>>>>>>>>> problems > > > > >> > >>>>>>>>>>>>>>>>>>>>>>> than it solves. > > > > >> > >>>>>>>>>>>>>>>>>>>>>>> > > > > >> > >>>>>>>>>>>>>>>>>>>>>>> But I am not 100% against - if others > will > > > > >> > >>> think > > > > >> > >>>>>>> it's > > > > >> > >>>>>>>> a > > > > >> > >>>>>>>>> good > > > > >> > >>>>>>>>>>>>>>>> idea, I > > > > >> > >>>>>>>>>>>>>>>>>>>>> am > > > > >> > >>>>>>>>>>>>>>>>>>>>>> ok > > > > >> > >>>>>>>>>>>>>>>>>>>>>>> with it. > > > > >> > >>>>>>>>>>>>>>>>>>>>>>> > > > > >> > >>>>>>>>>>>>>>>>>>>>>>> J, > > > > >> > >>>>>>>>>>>>>>>>>>>>>>> > > > > >> > >>>>>>>>>>>>>>>>>>>>>>> > > > > >> > >>>>>>>>>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 3:12 PM Abhishek > > > > >> > >>> Bhakat > > > > >> > >>>>>>>>>>>>>>>>>>>>>>> <abhishek.bha...@astronomer.io.inva > > > > >> > >>> <mailto: > > > > >> > >>>>>>>>>>> abhishek.bha...@astronomer.io.inva>lid> wrote: > > > > >> > >>>>>>>>>>>>>>>>>>>>>>> > > > > >> > >>>>>>>>>>>>>>>>>>>>>>>> Agreed that the word DAG makes very > less > > > > >> > >>> sense > > > > >> > >>>>> to > > > > >> > >>>>>>>>> someone > > > > >> > >>>>>>>>>>> new > > > > >> > >>>>>>>>>>>>>>> to > > > > >> > >>>>>>>>>>>>>>>>>>>>>> workflow > > > > >> > >>>>>>>>>>>>>>>>>>>>>>>> orchestration. But it does also show > the > > > > >> > >>> nature > > > > >> > >>>>>> of > > > > >> > >>>>>>>>> being > > > > >> > >>>>>>>>>>>>>>> acyclic. > > > > >> > >>>>>>>>>>>>>>>>>>>>> Sure, > > > > >> > >>>>>>>>>>>>>>>>>>>>>>> as > > > > >> > >>>>>>>>>>>>>>>>>>>>>>>> Bas mentioned, there are ways to > > > > >> > >>> workaround it. > > > > >> > >>>>>>>> Still, > > > > >> > >>>>>>>>> in > > > > >> > >>>>>>>>>>> my > > > > >> > >>>>>>>>>>>>>>>>>>>>> opinion, > > > > >> > >>>>>>>>>>>>>>>>>>>>>>> there > > > > >> > >>>>>>>>>>>>>>>>>>>>>>>> is generally no need for cyclic > behavior > > in > > > > >> > >>>>>>> workflow > > > > >> > >>>>>>>>>>>>>>>>>>> orchestration. > > > > >> > >>>>>>>>>>>>>>>>>>>>>> Most > > > > >> > >>>>>>>>>>>>>>>>>>>>>>>> (*if > > > > >> > >>>>>>>>>>>>>>>>>>>>>>>> not all*) cases can be in some way can > be > > > > >> > >>>>> covered > > > > >> > >>>>>>>>> using an > > > > >> > >>>>>>>>>>>>>>>> acyclic > > > > >> > >>>>>>>>>>>>>>>>>>>>>> manner > > > > >> > >>>>>>>>>>>>>>>>>>>>>>>> with multiple runs. Hence, the > > > > >> > >>> idempotency. So > > > > >> > >>>>> I > > > > >> > >>>>>>>> would > > > > >> > >>>>>>>>> want > > > > >> > >>>>>>>>>>>> the > > > > >> > >>>>>>>>>>>>>>>>>>>>>> "acyclic" > > > > >> > >>>>>>>>>>>>>>>>>>>>>>>> word to stick. > > > > >> > >>>>>>>>>>>>>>>>>>>>>>>> > > > > >> > >>>>>>>>>>>>>>>>>>>>>>>> Regards, > > > > >> > >>>>>>>>>>>>>>>>>>>>>>>> Avi > > > > >> > >>>>>>>>>>>>>>>>>>>>>>>> > > > > >> > >>>>>>>>>>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 12:41 PM < > > > > >> > >>>>>>>>>>> bernd.stroe...@kosakya.de <mailto: > > > > >> > >>> bernd.stroe...@kosakya.de>> > > > > >> > >>>>>>>>>>>>>>>>>>> wrote: > > > > >> > >>>>>>>>>>>>>>>>>>>>>>>> > > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> Brilliant, I am on the way to become > an > > > > >> > >>>>> Airflow > > > > >> > >>>>>>> Fan; > > > > >> > >>>>>>>>> so > > > > >> > >>>>>>>>>>> many > > > > >> > >>>>>>>>>>>>>>> new > > > > >> > >>>>>>>>>>>>>>>>>>>>>> ideas. > > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> > > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> The Term DAG is misleading; it should > be > > > > >> > >>>>>> replaced > > > > >> > >>>>>>> by > > > > >> > >>>>>>>>> the > > > > >> > >>>>>>>>>>>> more > > > > >> > >>>>>>>>>>>>>>>>>>>>> general > > > > >> > >>>>>>>>>>>>>>>>>>>>>>>> Term > > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> Airflow (Workflow) Graph (AFG) or > > Airflow > > > > >> > >>>>>> (Petri) > > > > >> > >>>>>>>> Net > > > > >> > >>>>>>>>>>> (AFN) > > > > >> > >>>>>>>>>>>>>>>>>>> (maybe > > > > >> > >>>>>>>>>>>>>>>>>>>>>>>> without > > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> a direction); > > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> and ... these Graphs should be stored > > in a > > > > >> > >>>>> Graph > > > > >> > >>>>>>>>> Database. > > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> > > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> Every Node or Sup-Graph of an Airflow > > > > >> > >>> Graph > > > > >> > >>>>>> (AFG) > > > > >> > >>>>>>>>> might be > > > > >> > >>>>>>>>>>>>>>>>>>>>> assigned > > > > >> > >>>>>>>>>>>>>>>>>>>>>> to > > > > >> > >>>>>>>>>>>>>>>>>>>>>>> an > > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> executable (Python-, Rust-, ... ) > member > > > > >> > >>> of a > > > > >> > >>>>>>>> library. > > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> > > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> A running Graph might have a different > > > > >> > >>>>> structure > > > > >> > >>>>>>>> than > > > > >> > >>>>>>>>> a > > > > >> > >>>>>>>>>>>>>>>>>>>>> configuration > > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> Graph. > > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> > > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> Forget that if you think it's > bullshit. > > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> > > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> Best Regards > > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> > > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> Bernd Ströhle > > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> M: +49 171 5357916 > > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> E: bernd.stroe...@gmail.com <mailto: > > > > >> > >>>>>>>>>>> bernd.stroe...@gmail.com> > > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> > > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> > > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> -----Original Message----- > > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> From: Igor Kholopov > > > > >> > >>> <ikholo...@google.com.inva > > > > >> > >>>>>>>>> <mailto: > > > > >> > >>>>>>>>>>> ikholo...@google.com.inva>LID> > > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> Sent: Tuesday, October 22, 2024 12:02 > PM > > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> To: dev@airflow.apache.org <mailto: > > > > >> > >>>>>>>>> dev@airflow.apache.org > > > > >> > >>>>>>>>>>>> > > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> Subject: Re: Airflow should deprecate > > the > > > > >> > >>> term > > > > >> > >>>>>>> "DAG" > > > > >> > >>>>>>>>> for > > > > >> > >>>>>>>>>>> end > > > > >> > >>>>>>>>>>>>>>>>>>> users > > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> > > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> Even though the term "DAG" is clearly > > > > >> > >>>>>> suboptimal, > > > > >> > >>>>>>> it > > > > >> > >>>>>>>>> is > > > > >> > >>>>>>>>>>> part > > > > >> > >>>>>
