-0.5 Even though the new UI prefers "Dag", I don't think that means "DAG" is incorrect.
On Tue, Feb 18, 2025 at 11:36 AM Jarek Potiuk <ja...@potiuk.com> wrote: > I love "Dynamic Airflow Graph" +10 to that Constance. > > And yes. I fully agree with your "somewhat" acyclic statement. I saw people > mentioning "event-driven" workflows to better describe the AI/ML, but that > for me is a synonym of "completely unmanageable workflows" (similar to > micro-services) - what I was alluding to was exactly this - something like > curren DAG with parts of it being potentially cyclic - so we are completely > aligned here :). And that's what make it "not DAG - in mathematical sense" > - even more reason to depart from it, > > J. > > On Tue, Feb 18, 2025 at 5:23 PM Constance Martineau > <consta...@astronomer.io.invalid> wrote: > > > Hey all, > > > > I appreciate the discussion about how we position the concept of a DAG in > > Airflow. While I agree with updating documentation to de-emphasize the > > strict mathematical definition, I do not think we should change the > public > > interface to use "dag" instead of "DAG" everywhere. It adds > > unnecessary churn and breaks consistency across the ecosystem. > > > > Regarding the broader discussion on whether DAGs in Airflow should remain > > acyclic: I acknowledge that some workflows, particularly in ML and AI, > > require cyclical behaviour (example: iterative refinement until a > condition > > is met). However, I think it's important to recognize that even in those > > cases, the workflow itself still has an overall start and end. It's not > an > > unbounded loop but rather a controlled execution process that happens to > > have cycles *within* it. > > > > That being said, I do see value in reinterpreting "DAG" to better align > > with how Airflow is actually used, and love shifting away from "Directed > > Acyclic Graph" in favour of something more intuitive. However "Data > Airflow > > Graph" makes it sound like we're defining Airflow itself rather than > > describing the structure of workflows within it. > > > > If we want a new meaning for DAG that resonates with our base while > keeping > > Airflow's identity clear, I am proposing three alternatives: > > > > > > 1. *Dynamic Airflow Graph*: Ties directly to Airflow while emphasizing > > flexibility > > 2. *Distributed Automation Graph*: Highlights execution across systems > > and automations without tying it strictly to data > > 3. *Directed Activity Graph*: Retains the useful meaning of "Directed" > > while shifting away from strict acyclic constraints. (My personal > > favourite) > > > > I think any shift in terminology should be driven by what best > communicates > > Airflow's role workflow automation, rather than just distancing ourselves > > from the mathematical definition. > > > > Curious to hear more thoughts! > > > > On Tue, Feb 18, 2025 at 10:40 AM Jarek Potiuk <ja...@potiuk.com> wrote: > > > > > Sorry for the few typos ... My slight dyslexia did not help and > > autocorrect > > > did not help either this time : dept -> debt, routed -> rooted and a > few > > > other small typos :) . > > > > > > On Tue, Feb 18, 2025 at 4:28 PM Jarek Potiuk <ja...@potiuk.com> wrote: > > > > > > > First of all - yes, agree with Lee and Ash, I think we do not need to > > > > change the public interface, I'd also be -0 on that, but docs, > internal > > > > strings, yes we could. > > > > > > > > But .... After thinking a bit and looking at the discussion here, > where > > > > people have concerns, I have an alternative take. And I know this > might > > > be > > > > controversial, and some people might find it confusing, but why don't > > we > > > > come up with a new acronym for DAG, one that will be more airflow and > > > less > > > > "mathematics/algorithm". And I have, I think a good reason for it, > and > > I > > > > even have a proposal. Bear with me, > > > > > > > > DAG => *Data Airflow Graph* > > > > > > > > As many of us mentioned, DAG and Airflow are almost 100% 1-1 now > (with > > > > exception of some niche products - you know which one I speak about - > > > which > > > > are unfortunate enough to use the "Dag" in their name - I will > explain > > > why > > > > this is unfortunate, IMHO). > > > > > > > > We are lucky to have Airflow for "A" in the acronym and we can make > it > > > > even more "synonymous" -> Airflow <> DAG with (A) being Airflow. > > > > > > > > Now, why I think "Directed Acyclic Graph" is unfortunate. > > > > > > > > Not only because "Directed" and "Acyclic" are mostly not in the > > > vocabulary > > > > of most of our users - as Ryan, starting this thread explained yes, I > > > agree > > > > with it, but also I think there is a more important reason - I > simply > > > > believe in the future our DAGs might not (and likely will not) be > > > "acyclic". > > > > > > > > I know for some it might be herezy and we have not discussed any of > > that > > > > yet, but more and more I keep on hearing that Machine Learning > > Workflows > > > of > > > > the future are often somewhat cyclic. In our case there is one type > of > > > > repeat-in-cycle behaviour - we have "task retry on failure" - but > this > > is > > > > only on failure and it basically mens "idempotent" retry - we want to > > > redo > > > > something and get single result out of that and basically discard the > > > > previous result. But machine learning/AI workflows are different. I > > had a > > > > lot of conversations recently (I even co-organized "AI low-level > > > > engineering and hacking" devroom to learn more about those) - and I > > keep > > > on > > > > hearing that "Repeat a sequence of things several times and refine" > is > > > > precisely what AI / ML workflows will need - especially with Agentic > > > > workflows - often we have cooperating agents which **could** be part > of > > > the > > > > same DAG, but currently in Airflow where our DAGs are "Acyclic" we > can > > > > only express "repeat several time and refine" as a single task that > > will > > > do > > > > such refining. If we would like to split such an "agent" doing their > > own > > > > work in "cycles" and communicating with other "agents" doing the > same - > > > > represented as a multi-task-entity, we have no way to express it > using > > > our > > > > current "acyclic" semantics. > > > > > > > > But - other than our implementation makes that assumption currently - > > > > there is nothing fundamental with Airflow being able to run such > > > "partially > > > > cyclic workflows". We have all the pieces, we would have to change > > > > dependency calculation to allow for cycles, update UI to handle > cycles > > > > representation - and we could in theory quite seamlessly, support > > > "cyclic" > > > > workflows. > > > > > > > > I know, I know, that's the first time we hear it and that's huge > > change - > > > > of course, I do not want to start discussing any details of it, not > > > before > > > > Airflow 3.0 is out - but I think we should be open that at some point > > of > > > > time the "Acyclic" property is going to go away, And my personal > > > intuition > > > > tells me that a year from now we will have acyclic workflows > (assuming > > > that > > > > all the foundational work we do on Airflow 3 will really help in > > > decreasing > > > > all the technical dept, and will open our minds to new ideas). > > > > > > > > That's why I think - the sooner we get rid of "Directed Acyclic" - > the > > > > better. that will - even for us, maintainer free us from being routed > > in > > > > the acyclic nature of Airflow. > > > > > > > > How about we change to "Data Airflow Graph" (or any other good > acronym > > > > which has (A)irflow instead of (A)cyclic). The name is just a > proposal. > > > > > > > > Now.. I go hide somewhere as I believe that might cause a "flamewar" > > :), > > > > unless on request of Ryan you already sheathed your swords in the > > > > discussion (which seems everyone did). > > > > > > > > But honestly - I would really love what you think of "let's just > break > > > > with Acyclic NOW" and shape our future by using a different acronym. > > > > > > > > J. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Feb 18, 2025 at 10:54 AM Ephraim Anierobi < > > > > ephraimanier...@gmail.com> wrote: > > > > > > > >> When I started contributing to Airflow, I had to read up on DAG, and > > > now I > > > >> know what it means. I wonder if we are about to have users who use > the > > > >> term > > > >> Dag without knowing that there's a DAG. > > > >> > > > >> Slightly concerned. > > > >> > > > >> -ephraim > > > >> > > > >> On Tue, 18 Feb 2025 at 10:26, Wei Lee <weilee...@gmail.com> wrote: > > > >> > > > >> > For user-facing things (e.g., `from airflow import DAG`), I’m more > > > like > > > >> > -0. But for documentation, docstring, internal things and etc., we > > > >> probably > > > >> > could still change most of them? > > > >> > > > > >> > Best, > > > >> > Wei > > > >> > > > > >> > > On Feb 18, 2025, at 5:10 PM, Ash Berlin-Taylor <a...@apache.org> > > > >> wrote: > > > >> > > > > > >> > > I also don’t personally think it’s worth the pain (not to > mention > > > >> > backcompat workaround) to rename DAG to Dag, so I’d be -0.5 on > that. > > > >> > > > > > >> > > -ash > > > >> > > > > > >> > >> On 18 Feb 2025, at 04:40, Daniel Imberman < > > > daniel.imber...@gmail.com > > > >> > > > > >> > wrote: > > > >> > >> > > > >> > >> I think my biggest concern is a marketing one and not a > technical > > > >> one. > > > >> > >> > > > >> > >> As has been mentioned on the thread the terms airflow and dag > are > > > >> kind > > > >> > of > > > >> > >> synonymous and I certainly don’t want to give the impression > that > > > we > > > >> are > > > >> > >> breaking more than we are breaking. > > > >> > >> > > > >> > >> I wouldn’t die on this hill, but I’m slightly concerned. > > > >> > >> > > > >> > >> On Mon, Feb 17, 2025 at 6:00 PM Wei Lee <wei...@apache.org> > > wrote: > > > >> > >> > > > >> > >>> I’m not sure about adding ruff rules here 🤔 I think ruff > rules > > > are > > > >> > best > > > >> > >>> suited for user-facing things but not the airflow code base > > > itself. > > > >> If > > > >> > what > > > >> > >>> we mean is adding a rule to avoid users using "DAG" *after* we > > > >> rename > > > >> > it, > > > >> > >>> it's definitely a +1000. > > > >> > >>> > > > >> > >>> I just created GitHub issues for this removing "DAG" > > > >> > >>> > > > >> > >>> https://github.com/apache/airflow/issues/46842 > > > >> > >>> https://github.com/apache/airflow/issues/46843 > > > >> > >>> > > > >> > >>> One thing I'm not sure about is whether we want to get rid of > > > `from > > > >> > >>> airflow import DAG` as well. 🤔 > > > >> > >>> > > > >> > >>> Best, > > > >> > >>> Wei > > > >> > >>> > > > >> > >>> On 2025/02/17 19:50:57 Jarek Potiuk wrote: > > > >> > >>>> Hard to say until it's looked at :) > > > >> > >>>> > > > >> > >>>> On Mon, Feb 17, 2025 at 8:45 PM Aritra Basu < > > > >> aritrabasu1...@gmail.com > > > >> > > > > > >> > >>>> wrote: > > > >> > >>>> > > > >> > >>>>> I can take it up, it's mostly just a doc update right? Or > are > > we > > > >> > doing > > > >> > >>> code > > > >> > >>>>> files replacement too? > > > >> > >>>>> -- > > > >> > >>>>> Regards, > > > >> > >>>>> Aritra Basu > > > >> > >>>>> > > > >> > >>>>> On Tue, 18 Feb 2025, 12:55 am Jarek Potiuk, < > ja...@potiuk.com > > > > > > >> > wrote: > > > >> > >>>>> > > > >> > >>>>>> Sounds like another +1000 files big PR is coming :) ? Any > > > >> volunteers > > > >> > >>> to > > > >> > >>>>>> make it ? It's fun. > > > >> > >>>>>> > > > >> > >>>>>> On Mon, Feb 17, 2025 at 6:52 PM Omkar P < > > > droiddev5...@gmail.com> > > > >> > >>> wrote: > > > >> > >>>>>> > > > >> > >>>>>>> +1 for ruff rules :) > > > >> > >>>>>>> > > > >> > >>>>>>> Also would be nice to introduce 'Dag' to replace 'DAG' in > > the > > > >> > >>> Airflow > > > >> > >>>>>>> docs, in line with the new UI changes and to make the > > renaming > > > >> > >>>>>>> consistent across user-facing pages. > > > >> > >>>>>>> > > > >> > >>>>>>> Regards, > > > >> > >>>>>>> Omkar > > > >> > >>>>>>> > > > >> > >>>>>>> On Mon, Feb 17, 2025 at 8:12 AM Jarek Potiuk < > > > ja...@potiuk.com> > > > >> > >>> wrote: > > > >> > >>>>>>> > > > >> > >>>>>>>> +1 :) . Maybe we could add a ruff rule for that :) > > > >> > >>>>>>>> > > > >> > >>>>>>>> On Mon, Feb 17, 2025 at 8:27 AM Wei Lee < > wei...@apache.org > > > > > > >> > >>> wrote: > > > >> > >>>>>>>> > > > >> > >>>>>>>>> It seems that our current conclusion is to use "dag" or > > > "Dag" > > > >> > >>>>> instead > > > >> > >>>>>>> of > > > >> > >>>>>>>>> "DAG" whenever possible. Should we replace all "DAG" in > > the > > > >> > >>>>> codebase > > > >> > >>>>>>> with > > > >> > >>>>>>>>> "dag" or "Dag"? If it's too late for that (which it > might > > be > > > >> > >>> 🤔), > > > >> > >>>>>>> should > > > >> > >>>>>>>> we > > > >> > >>>>>>>>> at least avoid introducing new "DAG" in the following > PRs? > > > >> > >>>>>>>>> > > > >> > >>>>>>>>> Best, > > > >> > >>>>>>>>> Wei > > > >> > >>>>>>>>> > > > >> > >>>>>>>>> On 2024/10/23 17:16:55 Brent Bovenzi wrote: > > > >> > >>>>>>>>>> Here's a PR to use "dag" as a word in the new 3.0 UI: > > > >> > >>>>>>>>>> https://github.com/apache/airflow/pull/43325 > > > >> > >>>>>>>>>> Let me know if that's the direction we want to go. > > > >> > >>>>>>>>>> > > > >> > >>>>>>>>>> On Wed, Oct 23, 2024 at 11:06 AM Bishundeo, Rajeshwar > > > >> > >>>>>>>>>> <rbish...@amazon.com.invalid> wrote: > > > >> > >>>>>>>>>> > > > >> > >>>>>>>>>>> I think Brent & Daniel summarized it best, "dag" is > > > >> > >>> synonymous > > > >> > >>>>>> with > > > >> > >>>>>>>>>>> workflows in Airflow through the way we talk and > explain > > > >> > >>> what > > > >> > >>>>>>> Airflow > > > >> > >>>>>>>>> is > > > >> > >>>>>>>>>>> all about. Although folks would ask, I don’t ever use > > the > > > >> > >>>>>>>> mathematical > > > >> > >>>>>>>>>>> definition of DAG. > > > >> > >>>>>>>>>>> It will be challenging and possibly confusing for many > > > >> > >>> users > > > >> > >>>>>> making > > > >> > >>>>>>>>> such a > > > >> > >>>>>>>>>>> change - I would rather direct that energy to > appending > > > the > > > >> > >>>>>> Oxford > > > >> > >>>>>>>>>>> definition of "dag" to include a reference to > workflows > > in > > > >> > >>>>>> Airflow. > > > >> > >>>>>>>> __ > > > >> > >>>>>>>>>>> > > > >> > >>>>>>>>>>> -- Rajesh > > > >> > >>>>>>>>>>> > > > >> > >>>>>>>>>>> > > > >> > >>>>>>>>>>> > > > >> > >>>>>>>>>>> > > > >> > >>>>>>>>>>> > > > >> > >>>>>>>>>>> > > > >> > >>>>>>>>>>> On 2024-10-23, 3:37 AM, "Jarek Potiuk" < > > ja...@potiuk.com > > > >> > >>>>>> <mailto: > > > >> > >>>>>>>>>>> ja...@potiuk.com>> wrote: > > > >> > >>>>>>>>>>> > > > >> > >>>>>>>>>>> > > > >> > >>>>>>>>>>> CAUTION: This email originated from outside of the > > > >> > >>>>> organization. > > > >> > >>>>>> Do > > > >> > >>>>>>>> not > > > >> > >>>>>>>>>>> click links or open attachments unless you can confirm > > the > > > >> > >>>>> sender > > > >> > >>>>>>> and > > > >> > >>>>>>>>> know > > > >> > >>>>>>>>>>> the content is safe. > > > >> > >>>>>>>>>>> > > > >> > >>>>>>>>>>> > > > >> > >>>>>>>>>>> > > > >> > >>>>>>>>>>> > > > >> > >>>>>>>>>>> > > > >> > >>>>>>>>>>> > > > >> > >>>>>>>>>>> AVERTISSEMENT: Ce courrier électronique provient d’un > > > >> > >>>>> expéditeur > > > >> > >>>>>>>>> externe. > > > >> > >>>>>>>>>>> Ne cliquez sur aucun lien et n’ouvrez aucune pièce > > jointe > > > >> > >>> si > > > >> > >>>>> vous > > > >> > >>>>>>> ne > > > >> > >>>>>>>>> pouvez > > > >> > >>>>>>>>>>> pas confirmer l’identité de l’expéditeur et si vous > > n’êtes > > > >> > >>> pas > > > >> > >>>>>>>> certain > > > >> > >>>>>>>>> que > > > >> > >>>>>>>>>>> le contenu ne présente aucun risque. > > > >> > >>>>>>>>>>> > > > >> > >>>>>>>>>>> > > > >> > >>>>>>>>>>> > > > >> > >>>>>>>>>>> > > > >> > >>>>>>>>>>> > > > >> > >>>>>>>>>>> > > > >> > >>>>>>>>>>> From Guido's post: > > > >> > >>>>>>>>>>> > > > >> > >>>>>>>>>>> > > > >> > >>>>>>>>>>> "Around this time the renaming seems to have been > > > renamed". > > > >> > >>>>>>>>>>> > > > >> > >>>>>>>>>>> > > > >> > >>>>>>>>>>> Naming is hard. > > > >> > >>>>>>>>>>> > > > >> > >>>>>>>>>>> > > > >> > >>>>>>>>>>> J. > > > >> > >>>>>>>>>>> > > > >> > >>>>>>>>>>> > > > >> > >>>>>>>>>>> > > > >> > >>>>>>>>>>> > > > >> > >>>>>>>>>>> On Wed, Oct 23, 2024 at 7:11 AM Omkar P < > > > >> > >>>>> droiddev5...@gmail.com > > > >> > >>>>>>>>> <mailto: > > > >> > >>>>>>>>>>> droiddev5...@gmail.com>> wrote: > > > >> > >>>>>>>>>>> > > > >> > >>>>>>>>>>> > > > >> > >>>>>>>>>>>> Yes, completely agree with above comments, dag is an > > > >> > >>> Airflow > > > >> > >>>>>> term > > > >> > >>>>>>>> now > > > >> > >>>>>>>>>>>> rather > > > >> > >>>>>>>>>>>> than just a "directed acyclic graph". Believe it or > > not, > > > >> > >>> I've > > > >> > >>>>>>>> worked > > > >> > >>>>>>>>> with > > > >> > >>>>>>>>>>>> work > > > >> > >>>>>>>>>>>> folks who've been using dags in Airflow for years now > > > >> > >>> and are > > > >> > >>>>>> pro > > > >> > >>>>>>>>> devs > > > >> > >>>>>>>>>>> but > > > >> > >>>>>>>>>>>> have trouble remembering the full form of a DAG! For > > > >> > >>> them, > > > >> > >>>>> dag > > > >> > >>>>>> = > > > >> > >>>>>>>>> Airflow. > > > >> > >>>>>>>>>>>> > > > >> > >>>>>>>>>>>> workflow, pipeline, flow are surely better terms but > > > >> > >>> will be > > > >> > >>>>> a > > > >> > >>>>>>>> major > > > >> > >>>>>>>>>>>> behavior > > > >> > >>>>>>>>>>>> change for the developer community, so we'll need a > > solid > > > >> > >>>>> plan > > > >> > >>>>>> on > > > >> > >>>>>>>>> how to > > > >> > >>>>>>>>>>>> introduce it, when we do. > > > >> > >>>>>>>>>>>> > > > >> > >>>>>>>>>>>> While we discuss this, I'd like to share about the > > Great > > > >> > >>>>>> Renaming > > > >> > >>>>>>>> in > > > >> > >>>>>>>>> core > > > >> > >>>>>>>>>>>> Python started by Guido back in 2009. We could > probably > > > >> > >>> get > > > >> > >>>>>> some > > > >> > >>>>>>>>>>> learnings > > > >> > >>>>>>>>>>>> from there? Who knows! > > > >> > >>>>>>>>>>>> > > > >> > >>>>>>>>>>>> Guido's blog (2009): > > > >> > >>>>>>>>>>>> > > > >> > >>>>>>>>> > > > >> > >>>>>>> > > > >> > >>>>> > > > >> > >>> > > > >> > > > > >> > > > > https://python-history.blogspot.com/2009/03/great-or-grand-renaming.html > > > >> > >>>>>>>>>>> < > > > >> > >>>>>>>>> > > > >> > >>>>>>> > > > >> > >>>>> > > > >> > >>> > > > >> > > > > >> > > > > https://python-history.blogspot.com/2009/03/great-or-grand-renaming.html > > > >> > >>>>>>>>> > > > >> > >>>>>>>>>>>> Follow-up discussion (2024): > > > >> > >>>>>>>>>>>> > > > >> > >>>>>> > > > https://discuss.python.org/t/finishing-the-great-renaming/54082 > > > >> > >>>>>>> < > > > >> > >>>>>>>>>>> > > > >> > >>>>> > > https://discuss.python.org/t/finishing-the-great-renaming/54082 > > > > > > > >> > >>>>>>>>>>>> > > > >> > >>>>>>>>>>>> Cheers, > > > >> > >>>>>>>>>>>> Omkar > > > >> > >>>>>>>>>>>> > > > >> > >>>>>>>>>>>> On Wed, Oct 23, 2024 at 7:10 AM Wei Lee < > > > >> > >>> weilee...@gmail.com > > > >> > >>>>>>>>> <mailto: > > > >> > >>>>>>>>>>> weilee...@gmail.com>> wrote: > > > >> > >>>>>>>>>>>> > > > >> > >>>>>>>>>>>>> I think we should probably just accept it as an > > airflow > > > >> > >>>>> term. > > > >> > >>>>>>> At > > > >> > >>>>>>>>> least, > > > >> > >>>>>>>>>>>>> that’s how I understood it when I first used > Airflow. > > I > > > >> > >>>>> feel > > > >> > >>>>>>>>> renaming > > > >> > >>>>>>>>>>> it > > > >> > >>>>>>>>>>>> at > > > >> > >>>>>>>>>>>>> this stage would require considerable effort from > > > >> > >>>>> maintainers > > > >> > >>>>>>> and > > > >> > >>>>>>>>>>>> existing > > > >> > >>>>>>>>>>>>> users without providing equivalent benefits. > > > >> > >>>>>>>>>>>>> > > > >> > >>>>>>>>>>>>> Best, > > > >> > >>>>>>>>>>>>> Wei > > > >> > >>>>>>>>>>>>> > > > >> > >>>>>>>>>>>>>> On Oct 23, 2024, at 8:01 AM, Kaxil Naik < > > > >> > >>>>>> kaxiln...@gmail.com > > > >> > >>>>>>>>>>> <mailto:kaxiln...@gmail.com>> wrote: > > > >> > >>>>>>>>>>>>>> > > > >> > >>>>>>>>>>>>>> Same agreed with Brent & Daniel -- maybe we > re-kindle > > > >> > >>>>> this > > > >> > >>>>>>>>> discussion > > > >> > >>>>>>>>>>>> for > > > >> > >>>>>>>>>>>>>> Airflow 4 :) -- but right now it will cause too > much > > > >> > >>>>>>> disruption > > > >> > >>>>>>>>>>>>>> > > > >> > >>>>>>>>>>>>>> On Tue, 22 Oct 2024 at 21:27, Constance Martineau > > > >> > >>>>>>>>>>>>>> <consta...@astronomer.io.inva <mailto: > > > >> > >>>>>>>>> consta...@astronomer.io.inva>lid> > > > >> > >>>>>>>>>>> wrote: > > > >> > >>>>>>>>>>>>>> > > > >> > >>>>>>>>>>>>>>> In my experience, when you ask those with Airflow > > > >> > >>>>>> experience > > > >> > >>>>>>>>> what a > > > >> > >>>>>>>>>>>> dag > > > >> > >>>>>>>>>>>>> is, > > > >> > >>>>>>>>>>>>>>> they'll start talking about workflow attributes - > > > >> > >>> stuff > > > >> > >>>>>> like > > > >> > >>>>>>>>> dags > > > >> > >>>>>>>>>>>> being > > > >> > >>>>>>>>>>>>> a > > > >> > >>>>>>>>>>>>>>> series of steps or tasks with owners. The > structure > > > >> > >>>>>> doesn't > > > >> > >>>>>>>>> come up. > > > >> > >>>>>>>>>>>>>>> > > > >> > >>>>>>>>>>>>>>> Echo-ing others, at this point, my vote is to > > > >> > >>> embrace > > > >> > >>>>> the > > > >> > >>>>>>> name > > > >> > >>>>>>>>> and > > > >> > >>>>>>>>>>>>>>> de-emphasize the mathematical structure aspect. > > > >> > >>>>>>>>>>>>>>> > > > >> > >>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 3:47 PM Vikram Koka > > > >> > >>>>>>>>>>>>> <vik...@astronomer.io.inva <mailto: > > > >> > >>>>> vik...@astronomer.io.inva > > > >> > >>>>>>>>> lid> > > > >> > >>>>>>>>>>>>>>> wrote: > > > >> > >>>>>>>>>>>>>>> > > > >> > >>>>>>>>>>>>>>>> It's an interesting discussion and I remember > > > >> > >>>>> struggling > > > >> > >>>>>>> with > > > >> > >>>>>>>>> this > > > >> > >>>>>>>>>>>>> when I > > > >> > >>>>>>>>>>>>>>>> started working with Airflow. > > > >> > >>>>>>>>>>>>>>>> But, I also agree with the viewpoint of it being > an > > > >> > >>>>>>>> established > > > >> > >>>>>>>>>>>> concept > > > >> > >>>>>>>>>>>>>>> now > > > >> > >>>>>>>>>>>>>>>> regardless of the origin. > > > >> > >>>>>>>>>>>>>>>> > > > >> > >>>>>>>>>>>>>>>> I am personally leaning towards the perspective > > > >> > >>> best > > > >> > >>>>>>>> expressed > > > >> > >>>>>>>>> by > > > >> > >>>>>>>>>>>>> Daniel > > > >> > >>>>>>>>>>>>>>>> Standish and Brent of using Dag as a word, rather > > > >> > >>> than > > > >> > >>>>>> the > > > >> > >>>>>>>>> computer > > > >> > >>>>>>>>>>>>>>> science > > > >> > >>>>>>>>>>>>>>>> concept. > > > >> > >>>>>>>>>>>>>>>> > > > >> > >>>>>>>>>>>>>>>> Best regards, > > > >> > >>>>>>>>>>>>>>>> Vikram > > > >> > >>>>>>>>>>>>>>>> > > > >> > >>>>>>>>>>>>>>>> > > > >> > >>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 9:46 AM Oliveira, Niko > > > >> > >>>>>>>>>>>>>>> <oniko...@amazon.com.inva <mailto: > > > >> > >>>>>> oniko...@amazon.com.inva > > > >> > >>>>>>>>> lid > > > >> > >>>>>>>>>>>>>>>>> > > > >> > >>>>>>>>>>>>>>>> wrote: > > > >> > >>>>>>>>>>>>>>>> > > > >> > >>>>>>>>>>>>>>>>> I agree with the general sentiment of: You're > > > >> > >>> right > > > >> > >>>>>> Ryan, > > > >> > >>>>>>>> DAG > > > >> > >>>>>>>>>>> isn't > > > >> > >>>>>>>>>>>>>>> great > > > >> > >>>>>>>>>>>>>>>>> and I'd rather workflow, but changing it will > > > >> > >>> cause > > > >> > >>>>> much > > > >> > >>>>>>>> more > > > >> > >>>>>>>>>>>> wreckage > > > >> > >>>>>>>>>>>>>>>> than > > > >> > >>>>>>>>>>>>>>>>> it solves. > > > >> > >>>>>>>>>>>>>>>>> > > > >> > >>>>>>>>>>>>>>>>> Also agree with the idea to just move away from > > > >> > >>>>> defining > > > >> > >>>>>>>> DAG. > > > >> > >>>>>>>>> I > > > >> > >>>>>>>>>>>> think > > > >> > >>>>>>>>>>>>>>>>> we've been naturally doing that as a community > > > >> > >>> for a > > > >> > >>>>>> while > > > >> > >>>>>>>> now > > > >> > >>>>>>>>>>>> anyway, > > > >> > >>>>>>>>>>>>>>> so > > > >> > >>>>>>>>>>>>>>>>> that feels like a natural step. > > > >> > >>>>>>>>>>>>>>>>> > > > >> > >>>>>>>>>>>>>>>>> Cheers, > > > >> > >>>>>>>>>>>>>>>>> Niko > > > >> > >>>>>>>>>>>>>>>>> > > > >> > >>>>>>>>>>>>>>>>> ________________________________ > > > >> > >>>>>>>>>>>>>>>>> From: Ash Berlin-Taylor <a...@apache.org > <mailto: > > > >> > >>>>>>>>> a...@apache.org>> > > > >> > >>>>>>>>>>>>>>>>> Sent: Tuesday, October 22, 2024 9:06:39 AM > > > >> > >>>>>>>>>>>>>>>>> To: dev@airflow.apache.org <mailto: > > > >> > >>>>>> dev@airflow.apache.org > > > >> > >>>>>>>> > > > >> > >>>>>>>>>>>>>>>>> Subject: RE: [EXT] Airflow should deprecate the > > > >> > >>> term > > > >> > >>>>>> "DAG" > > > >> > >>>>>>>>> for end > > > >> > >>>>>>>>>>>>>>> users > > > >> > >>>>>>>>>>>>>>>>> > > > >> > >>>>>>>>>>>>>>>>> CAUTION: This email originated from outside of > the > > > >> > >>>>>>>>> organization. > > > >> > >>>>>>>>>>> Do > > > >> > >>>>>>>>>>>>> not > > > >> > >>>>>>>>>>>>>>>>> click links or open attachments unless you can > > > >> > >>> confirm > > > >> > >>>>>> the > > > >> > >>>>>>>>> sender > > > >> > >>>>>>>>>>>> and > > > >> > >>>>>>>>>>>>>>>> know > > > >> > >>>>>>>>>>>>>>>>> the content is safe. > > > >> > >>>>>>>>>>>>>>>>> > > > >> > >>>>>>>>>>>>>>>>> > > > >> > >>>>>>>>>>>>>>>>> > > > >> > >>>>>>>>>>>>>>>>> AVERTISSEMENT: Ce courrier électronique provient > > > >> > >>> d’un > > > >> > >>>>>>>>> expéditeur > > > >> > >>>>>>>>>>>>>>> externe. > > > >> > >>>>>>>>>>>>>>>>> Ne cliquez sur aucun lien et n’ouvrez aucune > pièce > > > >> > >>>>>> jointe > > > >> > >>>>>>> si > > > >> > >>>>>>>>> vous > > > >> > >>>>>>>>>>> ne > > > >> > >>>>>>>>>>>>>>>> pouvez > > > >> > >>>>>>>>>>>>>>>>> pas confirmer l’identité de l’expéditeur et si > > > >> > >>> vous > > > >> > >>>>>> n’êtes > > > >> > >>>>>>>> pas > > > >> > >>>>>>>>>>>> certain > > > >> > >>>>>>>>>>>>>>>> que > > > >> > >>>>>>>>>>>>>>>>> le contenu ne présente aucun risque. > > > >> > >>>>>>>>>>>>>>>>> > > > >> > >>>>>>>>>>>>>>>>> > > > >> > >>>>>>>>>>>>>>>>> > > > >> > >>>>>>>>>>>>>>>>> Best argument in favour of keeping “dags” as a > > > >> > >>> term — > > > >> > >>>>>>>> getting > > > >> > >>>>>>>>> to > > > >> > >>>>>>>>>>>>> re-use > > > >> > >>>>>>>>>>>>>>>>> puns like https://i.imgflip.com/1xhtwh.jpg < > > > >> > >>>>>>>>>>> https://i.imgflip.com/1xhtwh.jpg> > > > >> > >>>>>>>>>>>>>>>>> > > > >> > >>>>>>>>>>>>>>>>> In all seriousness: I don’t mind either way, > both > > > >> > >>>>> sides > > > >> > >>>>>>> have > > > >> > >>>>>>>>> good > > > >> > >>>>>>>>>>>>>>> reasons > > > >> > >>>>>>>>>>>>>>>>> presented. > > > >> > >>>>>>>>>>>>>>>>> > > > >> > >>>>>>>>>>>>>>>>> -a > > > >> > >>>>>>>>>>>>>>>>> > > > >> > >>>>>>>>>>>>>>>>>> On 22 Oct 2024, at 17:03, Daniel Standish > > > >> > >>>>>>>>>>>>>>>>> <daniel.stand...@astronomer.io.inva <mailto: > > > >> > >>>>>>>>>>> daniel.stand...@astronomer.io.inva>LID> wrote: > > > >> > >>>>>>>>>>>>>>>>>> > > > >> > >>>>>>>>>>>>>>>>>> Yeah just say, when asked where the name comes > > > >> > >>> from, > > > >> > >>>>>>> "well, > > > >> > >>>>>>>>> no > > > >> > >>>>>>>>>>> one > > > >> > >>>>>>>>>>>>>>>>> actually > > > >> > >>>>>>>>>>>>>>>>>> knows but..." and then make something up. > > > >> > >>>>>>>>>>>>>>>>>> > > > >> > >>>>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 8:31 AM Jarek Potiuk < > > > >> > >>>>>>>>> ja...@potiuk.com > > > >> > >>>>>>>>>>> <mailto:ja...@potiuk.com>> > > > >> > >>>>>>>>>>>>>>> wrote: > > > >> > >>>>>>>>>>>>>>>>>> > > > >> > >>>>>>>>>>>>>>>>>>> Just to clarify - "directed acyclic graph" is > > > >> > >>> the > > > >> > >>>>>>>>>>> tongue-twister, > > > >> > >>>>>>>>>>>>>>>>>>> > > > >> > >>>>>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 5:29 PM Jarek Potiuk < > > > >> > >>>>>>>>> ja...@potiuk.com > > > >> > >>>>>>>>>>> <mailto:ja...@potiuk.com>> > > > >> > >>>>>>>>>>>>>>>> wrote: > > > >> > >>>>>>>>>>>>>>>>>>> > > > >> > >>>>>>>>>>>>>>>>>>>> I like what both Daniel and Brent wrote. I > > > >> > >>> would > > > >> > >>>>> very > > > >> > >>>>>>>> much > > > >> > >>>>>>>>> want > > > >> > >>>>>>>>>>>> to > > > >> > >>>>>>>>>>>>>>> be > > > >> > >>>>>>>>>>>>>>>>>>> able > > > >> > >>>>>>>>>>>>>>>>>>>> to say just "dag" without explaining it > > > >> > >>> further. > > > >> > >>>>>>>>>>>>>>>>>>>> > > > >> > >>>>>>>>>>>>>>>>>>>> For me every time I explain "DAG" at a talk > > > >> > >>> it's a > > > >> > >>>>>>>>>>>> tongue-twister, > > > >> > >>>>>>>>>>>>>>>> and > > > >> > >>>>>>>>>>>>>>>>> I > > > >> > >>>>>>>>>>>>>>>>>>>> almost stutter on trying to recall how to > > > >> > >>> pronounce > > > >> > >>>>>> it > > > >> > >>>>>>>>>>> properly. > > > >> > >>>>>>>>>>>>>>>>>>>> > > > >> > >>>>>>>>>>>>>>>>>>>> J. > > > >> > >>>>>>>>>>>>>>>>>>>> > > > >> > >>>>>>>>>>>>>>>>>>>> > > > >> > >>>>>>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 5:27 PM Brent Bovenzi > > > >> > >>>>>>>>>>>>>>>>>>> <br...@astronomer.io.inva <mailto: > > > >> > >>>>>>>> br...@astronomer.io.inva > > > >> > >>>>>>>>>> lid> > > > >> > >>>>>>>>>>>>>>>>>>>> wrote: > > > >> > >>>>>>>>>>>>>>>>>>>> > > > >> > >>>>>>>>>>>>>>>>>>>>> I remember we explored renaming "DAG" when > > > >> > >>>>> starting > > > >> > >>>>>> on > > > >> > >>>>>>>>> AIP-38 > > > >> > >>>>>>>>>>> to > > > >> > >>>>>>>>>>>>>>>>>>> modernize > > > >> > >>>>>>>>>>>>>>>>>>>>> the UI. Both "pipeline" or "workflow" are > more > > > >> > >>>>>>>>> descriptive of > > > >> > >>>>>>>>>>>> what > > > >> > >>>>>>>>>>>>>>>> one > > > >> > >>>>>>>>>>>>>>>>>>> is > > > >> > >>>>>>>>>>>>>>>>>>>>> actually doing while Directed Acyclic Graph > > > >> > >>> is an > > > >> > >>>>>>>>>>> implementation > > > >> > >>>>>>>>>>>>>>>>> detail. > > > >> > >>>>>>>>>>>>>>>>>>>>> But I agree with Daniel Standish, at this > > > >> > >>> point > > > >> > >>>>>> "DAG" > > > >> > >>>>>>>> has > > > >> > >>>>>>>>>>> become > > > >> > >>>>>>>>>>>>>>>> "dag" > > > >> > >>>>>>>>>>>>>>>>>>> , a > > > >> > >>>>>>>>>>>>>>>>>>>>> word in its own right. > > > >> > >>>>>>>>>>>>>>>>>>>>> > > > >> > >>>>>>>>>>>>>>>>>>>>> Examples for "dag" are abound in community > > > >> > >>>>>> discussion, > > > >> > >>>>>>>>> Airflow > > > >> > >>>>>>>>>>>>>>>> Summit > > > >> > >>>>>>>>>>>>>>>>>>>>> talks, documentation and even in the UI. > Let's > > > >> > >>>>>> embrace > > > >> > >>>>>>>>> "dag". > > > >> > >>>>>>>>>>> A > > > >> > >>>>>>>>>>>>>>> user > > > >> > >>>>>>>>>>>>>>>>>>> just > > > >> > >>>>>>>>>>>>>>>>>>>>> needs to learn one new word vs the technical > > > >> > >>>>> concept > > > >> > >>>>>>>>> behind > > > >> > >>>>>>>>>>> that > > > >> > >>>>>>>>>>>>>>>>> word. I > > > >> > >>>>>>>>>>>>>>>>>>>>> think that is much less effort than > > > >> > >>> refactoring so > > > >> > >>>>>>> much > > > >> > >>>>>>>>> code, > > > >> > >>>>>>>>>>>>>>>>>>>>> documentation, blog posts, stack overflow > > > >> > >>>>> questions, > > > >> > >>>>>>>> etc. > > > >> > >>>>>>>>>>>>>>>>>>>>> > > > >> > >>>>>>>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 10:51 AM Daniel > > > >> > >>> Standish > > > >> > >>>>>>>>>>>>>>>>>>>>> <daniel.stand...@astronomer.io.inva > <mailto: > > > >> > >>>>>>>>>>> daniel.stand...@astronomer.io.inva>lid> wrote: > > > >> > >>>>>>>>>>>>>>>>>>>>> > > > >> > >>>>>>>>>>>>>>>>>>>>>> I am skeptical. Seems like introducing a > lot > > > >> > >>> of > > > >> > >>>>>> pain > > > >> > >>>>>>>> for > > > >> > >>>>>>>>>>>>>>>>> questionable > > > >> > >>>>>>>>>>>>>>>>>>>>>> benefit. But, I am def sympathetic to the > > > >> > >>> idea. I > > > >> > >>>>>>> agree > > > >> > >>>>>>>>> the > > > >> > >>>>>>>>>>>>>>>>>>>>> association > > > >> > >>>>>>>>>>>>>>>>>>>>>> with "directed acyclic graph" is not > helpful. > > > >> > >>>>>>>>>>>>>>>>>>>>>> > > > >> > >>>>>>>>>>>>>>>>>>>>>> And along those lines, I offer here some > less > > > >> > >>>>>>> invasive > > > >> > >>>>>>>>>>>>>>> mitigations. > > > >> > >>>>>>>>>>>>>>>>>>>>>> > > > >> > >>>>>>>>>>>>>>>>>>>>>> One thing we can do no matter what is to > > > >> > >>>>>> de-emphasize > > > >> > >>>>>>>> the > > > >> > >>>>>>>>>>> math > > > >> > >>>>>>>>>>>>>>> nerd > > > >> > >>>>>>>>>>>>>>>>>>>>> origins > > > >> > >>>>>>>>>>>>>>>>>>>>>> of the name. That is to say, in docs / > > > >> > >>> website / > > > >> > >>>>>> etc, > > > >> > >>>>>>>>> *never > > > >> > >>>>>>>>>>>>>>>> define* > > > >> > >>>>>>>>>>>>>>>>>>>>>> airflow's "dag" concept as a directed > acyclic > > > >> > >>>>>> graph. > > > >> > >>>>>>>>> Always > > > >> > >>>>>>>>>>>>>>> define > > > >> > >>>>>>>>>>>>>>>>> it > > > >> > >>>>>>>>>>>>>>>>>>>>> as a > > > >> > >>>>>>>>>>>>>>>>>>>>>> pipeline, collection of tasks, workflow > etc. > > > >> > >>>>>>>>>>>>>>>>>>>>>> > > > >> > >>>>>>>>>>>>>>>>>>>>>> The "directed acyclic graph" part of it is > > > >> > >>> like a > > > >> > >>>>>>>>> historical > > > >> > >>>>>>>>>>>>>>>>> footnote, > > > >> > >>>>>>>>>>>>>>>>>>>>> and > > > >> > >>>>>>>>>>>>>>>>>>>>>> we could make one mention of it somewhere > > > >> > >>> hidden. > > > >> > >>>>>>>>>>>>>>>>>>>>>> > > > >> > >>>>>>>>>>>>>>>>>>>>>> We could also start using lowercase in the > > > >> > >>> docs > > > >> > >>>>> in > > > >> > >>>>>>>>> general > > > >> > >>>>>>>>>>> e.g. > > > >> > >>>>>>>>>>>>>>>>>>> writing > > > >> > >>>>>>>>>>>>>>>>>>>>>> "dag" / "dags" instead of writing "DAG" / > > > >> > >>> "DAGs" > > > >> > >>>>>> etc. > > > >> > >>>>>>>> The > > > >> > >>>>>>>>>>>> upper > > > >> > >>>>>>>>>>>>>>>> case > > > >> > >>>>>>>>>>>>>>>>>>>>> part > > > >> > >>>>>>>>>>>>>>>>>>>>>> of it makes it look like an acronym; but > > > >> > >>> "dag" in > > > >> > >>>>>>>>> airlfow is > > > >> > >>>>>>>>>>>> just > > > >> > >>>>>>>>>>>>>>>> an > > > >> > >>>>>>>>>>>>>>>>>>>>>> airflow concept and the association with > > > >> > >>> "DAGs" > > > >> > >>>>> is > > > >> > >>>>>>> not > > > >> > >>>>>>>>> really > > > >> > >>>>>>>>>>>>>>>>>>> unhelpful. > > > >> > >>>>>>>>>>>>>>>>>>>>>> > > > >> > >>>>>>>>>>>>>>>>>>>>>> In other words embrace that "dag" in > airflow > > > >> > >>> is > > > >> > >>>>> its > > > >> > >>>>>>> own > > > >> > >>>>>>>>>>> thing, > > > >> > >>>>>>>>>>>> is > > > >> > >>>>>>>>>>>>>>>>>>>>>> *not* strictly > > > >> > >>>>>>>>>>>>>>>>>>>>>> speaking a directed acyclic graph (which > > > >> > >>> nobody > > > >> > >>>>>> knows > > > >> > >>>>>>>>> about > > > >> > >>>>>>>>>>>>>>>> anyway), > > > >> > >>>>>>>>>>>>>>>>>>> and > > > >> > >>>>>>>>>>>>>>>>>>>>>> tell them what it is in simple terms that > > > >> > >>> normal > > > >> > >>>>>>> people > > > >> > >>>>>>>>>>>>>>> understand. > > > >> > >>>>>>>>>>>>>>>>>>>>>> > > > >> > >>>>>>>>>>>>>>>>>>>>>> > > > >> > >>>>>>>>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 7:27 AM Jarek > Potiuk > > > >> > >>> < > > > >> > >>>>>>>>>>> ja...@potiuk.com <mailto:ja...@potiuk.com> > > > >> > >>>>>>>>>>>>> > > > >> > >>>>>>>>>>>>>>>>>>> wrote: > > > >> > >>>>>>>>>>>>>>>>>>>>>> > > > >> > >>>>>>>>>>>>>>>>>>>>>>> DAG is so embedded into what we do that it > > > >> > >>> will > > > >> > >>>>> be > > > >> > >>>>>>>>> extremely > > > >> > >>>>>>>>>>>>>>>>>>>>> difficult to > > > >> > >>>>>>>>>>>>>>>>>>>>>>> get rid of it completely. Also I think it > > > >> > >>> will > > > >> > >>>>>> make > > > >> > >>>>>>> a > > > >> > >>>>>>>>> lot of > > > >> > >>>>>>>>>>>>>>>>>>> "google" > > > >> > >>>>>>>>>>>>>>>>>>>>>>> searches and "stack overflow" searches not > > > >> > >>>>> finding > > > >> > >>>>>>> the > > > >> > >>>>>>>>> right > > > >> > >>>>>>>>>>>>>>>>>>> answers. > > > >> > >>>>>>>>>>>>>>>>>>>>>> This > > > >> > >>>>>>>>>>>>>>>>>>>>>>> is one of the strengths of Airflow - > > > >> > >>> besides the > > > >> > >>>>>>>>> community > > > >> > >>>>>>>>>>> and > > > >> > >>>>>>>>>>>>>>>> ideas > > > >> > >>>>>>>>>>>>>>>>>>>>> that > > > >> > >>>>>>>>>>>>>>>>>>>>>>> Bernd mentioned - is the vast number of > > > >> > >>>>> examples, > > > >> > >>>>>>>>> problems > > > >> > >>>>>>>>>>> and > > > >> > >>>>>>>>>>>>>>>>>>>>> solutions > > > >> > >>>>>>>>>>>>>>>>>>>>>>> you can so easily find (and we have to > > > >> > >>> remember > > > >> > >>>>>> that > > > >> > >>>>>>>>> all the > > > >> > >>>>>>>>>>>> AI > > > >> > >>>>>>>>>>>>>>>>>>>>> trained > > > >> > >>>>>>>>>>>>>>>>>>>>>> on > > > >> > >>>>>>>>>>>>>>>>>>>>>>> past data will be also rather poorly > > > >> > >>> matching > > > >> > >>>>>>> queries > > > >> > >>>>>>>> of > > > >> > >>>>>>>>>>>> people. > > > >> > >>>>>>>>>>>>>>>>>>>>>>> > > > >> > >>>>>>>>>>>>>>>>>>>>>>> I am not too attached to DAG. I could > easily > > > >> > >>>>>> switch. > > > >> > >>>>>>>>> And if > > > >> > >>>>>>>>>>> we > > > >> > >>>>>>>>>>>>>>> do > > > >> > >>>>>>>>>>>>>>>> - > > > >> > >>>>>>>>>>>>>>>>>>> I > > > >> > >>>>>>>>>>>>>>>>>>>>>>> would be for using workflow or pipeline > > > >> > >>> instead > > > >> > >>>>> of > > > >> > >>>>>>>>> `dag` if > > > >> > >>>>>>>>>>>> not > > > >> > >>>>>>>>>>>>>>>> the > > > >> > >>>>>>>>>>>>>>>>>>>>> above > > > >> > >>>>>>>>>>>>>>>>>>>>>>> reason, but I think I am here with Igor > > > >> > >>> that it > > > >> > >>>>>>> might > > > >> > >>>>>>>>> cause > > > >> > >>>>>>>>>>>> more > > > >> > >>>>>>>>>>>>>>>>>>>>> problems > > > >> > >>>>>>>>>>>>>>>>>>>>>>> than it solves. > > > >> > >>>>>>>>>>>>>>>>>>>>>>> > > > >> > >>>>>>>>>>>>>>>>>>>>>>> But I am not 100% against - if others will > > > >> > >>> think > > > >> > >>>>>>> it's > > > >> > >>>>>>>> a > > > >> > >>>>>>>>> good > > > >> > >>>>>>>>>>>>>>>> idea, I > > > >> > >>>>>>>>>>>>>>>>>>>>> am > > > >> > >>>>>>>>>>>>>>>>>>>>>> ok > > > >> > >>>>>>>>>>>>>>>>>>>>>>> with it. > > > >> > >>>>>>>>>>>>>>>>>>>>>>> > > > >> > >>>>>>>>>>>>>>>>>>>>>>> J, > > > >> > >>>>>>>>>>>>>>>>>>>>>>> > > > >> > >>>>>>>>>>>>>>>>>>>>>>> > > > >> > >>>>>>>>>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 3:12 PM Abhishek > > > >> > >>> Bhakat > > > >> > >>>>>>>>>>>>>>>>>>>>>>> <abhishek.bha...@astronomer.io.inva > > > >> > >>> <mailto: > > > >> > >>>>>>>>>>> abhishek.bha...@astronomer.io.inva>lid> wrote: > > > >> > >>>>>>>>>>>>>>>>>>>>>>> > > > >> > >>>>>>>>>>>>>>>>>>>>>>>> Agreed that the word DAG makes very less > > > >> > >>> sense > > > >> > >>>>> to > > > >> > >>>>>>>>> someone > > > >> > >>>>>>>>>>> new > > > >> > >>>>>>>>>>>>>>> to > > > >> > >>>>>>>>>>>>>>>>>>>>>> workflow > > > >> > >>>>>>>>>>>>>>>>>>>>>>>> orchestration. But it does also show the > > > >> > >>> nature > > > >> > >>>>>> of > > > >> > >>>>>>>>> being > > > >> > >>>>>>>>>>>>>>> acyclic. > > > >> > >>>>>>>>>>>>>>>>>>>>> Sure, > > > >> > >>>>>>>>>>>>>>>>>>>>>>> as > > > >> > >>>>>>>>>>>>>>>>>>>>>>>> Bas mentioned, there are ways to > > > >> > >>> workaround it. > > > >> > >>>>>>>> Still, > > > >> > >>>>>>>>> in > > > >> > >>>>>>>>>>> my > > > >> > >>>>>>>>>>>>>>>>>>>>> opinion, > > > >> > >>>>>>>>>>>>>>>>>>>>>>> there > > > >> > >>>>>>>>>>>>>>>>>>>>>>>> is generally no need for cyclic behavior > in > > > >> > >>>>>>> workflow > > > >> > >>>>>>>>>>>>>>>>>>> orchestration. > > > >> > >>>>>>>>>>>>>>>>>>>>>> Most > > > >> > >>>>>>>>>>>>>>>>>>>>>>>> (*if > > > >> > >>>>>>>>>>>>>>>>>>>>>>>> not all*) cases can be in some way can be > > > >> > >>>>> covered > > > >> > >>>>>>>>> using an > > > >> > >>>>>>>>>>>>>>>> acyclic > > > >> > >>>>>>>>>>>>>>>>>>>>>> manner > > > >> > >>>>>>>>>>>>>>>>>>>>>>>> with multiple runs. Hence, the > > > >> > >>> idempotency. So > > > >> > >>>>> I > > > >> > >>>>>>>> would > > > >> > >>>>>>>>> want > > > >> > >>>>>>>>>>>> the > > > >> > >>>>>>>>>>>>>>>>>>>>>> "acyclic" > > > >> > >>>>>>>>>>>>>>>>>>>>>>>> word to stick. > > > >> > >>>>>>>>>>>>>>>>>>>>>>>> > > > >> > >>>>>>>>>>>>>>>>>>>>>>>> Regards, > > > >> > >>>>>>>>>>>>>>>>>>>>>>>> Avi > > > >> > >>>>>>>>>>>>>>>>>>>>>>>> > > > >> > >>>>>>>>>>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 12:41 PM < > > > >> > >>>>>>>>>>> bernd.stroe...@kosakya.de <mailto: > > > >> > >>> bernd.stroe...@kosakya.de>> > > > >> > >>>>>>>>>>>>>>>>>>> wrote: > > > >> > >>>>>>>>>>>>>>>>>>>>>>>> > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> Brilliant, I am on the way to become an > > > >> > >>>>> Airflow > > > >> > >>>>>>> Fan; > > > >> > >>>>>>>>> so > > > >> > >>>>>>>>>>> many > > > >> > >>>>>>>>>>>>>>> new > > > >> > >>>>>>>>>>>>>>>>>>>>>> ideas. > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> The Term DAG is misleading; it should be > > > >> > >>>>>> replaced > > > >> > >>>>>>> by > > > >> > >>>>>>>>> the > > > >> > >>>>>>>>>>>> more > > > >> > >>>>>>>>>>>>>>>>>>>>> general > > > >> > >>>>>>>>>>>>>>>>>>>>>>>> Term > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> Airflow (Workflow) Graph (AFG) or > Airflow > > > >> > >>>>>> (Petri) > > > >> > >>>>>>>> Net > > > >> > >>>>>>>>>>> (AFN) > > > >> > >>>>>>>>>>>>>>>>>>> (maybe > > > >> > >>>>>>>>>>>>>>>>>>>>>>>> without > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> a direction); > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> and ... these Graphs should be stored > in a > > > >> > >>>>> Graph > > > >> > >>>>>>>>> Database. > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> Every Node or Sup-Graph of an Airflow > > > >> > >>> Graph > > > >> > >>>>>> (AFG) > > > >> > >>>>>>>>> might be > > > >> > >>>>>>>>>>>>>>>>>>>>> assigned > > > >> > >>>>>>>>>>>>>>>>>>>>>> to > > > >> > >>>>>>>>>>>>>>>>>>>>>>> an > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> executable (Python-, Rust-, ... ) member > > > >> > >>> of a > > > >> > >>>>>>>> library. > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> A running Graph might have a different > > > >> > >>>>> structure > > > >> > >>>>>>>> than > > > >> > >>>>>>>>> a > > > >> > >>>>>>>>>>>>>>>>>>>>> configuration > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> Graph. > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> Forget that if you think it's bullshit. > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> Best Regards > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> Bernd Ströhle > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> M: +49 171 5357916 > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> E: bernd.stroe...@gmail.com <mailto: > > > >> > >>>>>>>>>>> bernd.stroe...@gmail.com> > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> -----Original Message----- > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> From: Igor Kholopov > > > >> > >>> <ikholo...@google.com.inva > > > >> > >>>>>>>>> <mailto: > > > >> > >>>>>>>>>>> ikholo...@google.com.inva>LID> > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> Sent: Tuesday, October 22, 2024 12:02 PM > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> To: dev@airflow.apache.org <mailto: > > > >> > >>>>>>>>> dev@airflow.apache.org > > > >> > >>>>>>>>>>>> > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> Subject: Re: Airflow should deprecate > the > > > >> > >>> term > > > >> > >>>>>>> "DAG" > > > >> > >>>>>>>>> for > > > >> > >>>>>>>>>>> end > > > >> > >>>>>>>>>>>>>>>>>>> users > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> Even though the term "DAG" is clearly > > > >> > >>>>>> suboptimal, > > > >> > >>>>>>> it > > > >> > >>>>>>>>> is > > > >> > >>>>>>>>>>> part > > > >> > >>>>>>>>>>>>>>> of > > > >> > >>>>>>>>>>>>>>>>>>>>>> Airflow > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> DAG definition interface at so many > > > >> > >>> levels, > > > >> > >>>>> that > > > >> > >>>>>>> any > > > >> > >>>>>>>>>>> attempt > > > >> > >>>>>>>>>>>>>>> to > > > >> > >>>>>>>>>>>>>>>>>>>>>> change > > > >> > >>>>>>>>>>>>>>>>>>>>>>> it > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> will only introduce more chaos, not > > > >> > >>> reduce it. > > > >> > >>>>>> The > > > >> > >>>>>>>>> only > > > >> > >>>>>>>>>>>> thing > > > >> > >>>>>>>>>>>>>>>>>>>>> that is > > > >> > >>>>>>>>>>>>>>>>>>>>>>>> worse > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> than a poorly chosen name in the code is > > > >> > >>> when > > > >> > >>>>>>> there > > > >> > >>>>>>>>> are > > > >> > >>>>>>>>>>> two > > > >> > >>>>>>>>>>>>>>> ways > > > >> > >>>>>>>>>>>>>>>>>>>>> to > > > >> > >>>>>>>>>>>>>>>>>>>>>>>> define > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> the same thing. Countless articles and > > > >> > >>>>> tutorials > > > >> > >>>>>>>> will > > > >> > >>>>>>>>>>>> suddenly > > > >> > >>>>>>>>>>>>>>>>>>>>> become > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> confusing as they all refer to workflows > > > >> > >>> as > > > >> > >>>>>>> "DAG"s. > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> We are already at risk of scaring the > > > >> > >>> users > > > >> > >>>>> away > > > >> > >>>>>>>> with > > > >> > >>>>>>>>> a > > > >> > >>>>>>>>>>>> number > > > >> > >>>>>>>>>>>>>>>>>>> of > > > >> > >>>>>>>>>>>>>>>>>>>>>>>> breaking > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> changes in Airflow 3, promising even > more > > > >> > >>>>>> breaking > > > >> > >>>>>>>>> changes > > > >> > >>>>>>>>>>>> for > > > >> > >>>>>>>>>>>>>>>>>>> the > > > >> > >>>>>>>>>>>>>>>>>>>>>> most > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> basic things is not something that > people > > > >> > >>> are > > > >> > >>>>>>>> looking > > > >> > >>>>>>>>> for. > > > >> > >>>>>>>>>>>>>>>>>>>>> Attempting > > > >> > >>>>>>>>>>>>>>>>>>>>>>> to > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> change the fundamental terms will be > > > >> > >>>>> interpreted > > > >> > >>>>>>> as > > > >> > >>>>>>>> an > > > >> > >>>>>>>>>>> even > > > >> > >>>>>>>>>>>>>>>>>>>>> stronger > > > >> > >>>>>>>>>>>>>>>>>>>>>>>> signal > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> of project immaturity. > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> Given that, I oppose the idea of > changing > > > >> > >>> the > > > >> > >>>>>> term > > > >> > >>>>>>>> in > > > >> > >>>>>>>>> the > > > >> > >>>>>>>>>>>> long > > > >> > >>>>>>>>>>>>>>>>>>>>> run. I > > > >> > >>>>>>>>>>>>>>>>>>>>>>>> even > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> stricter oppose the idea of deprecating > > > >> > >>> it in > > > >> > >>>>>> the > > > >> > >>>>>>>> DAG > > > >> > >>>>>>>>>>>>>>> definition > > > >> > >>>>>>>>>>>>>>>>>>>>>>>> interface. > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> We better put our time and efforts in > > > >> > >>> other > > > >> > >>>>>> places > > > >> > >>>>>>>> in > > > >> > >>>>>>>>>>>> Airflow, > > > >> > >>>>>>>>>>>>>>>>>>> of > > > >> > >>>>>>>>>>>>>>>>>>>>>> which > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> there are plenty. > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> Kind regards, > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> Igor > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 10:36 AM Bas > > > >> > >>> Harenslak > > > >> > >>>>>>>>>>>>>>>>>>>>>>> <b...@astronomer.io.inva <mailto: > > > >> > >>>>>>>> b...@astronomer.io.inva > > > >> > >>>>>>>>>> lid > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> wrote: > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>> Couple of thoughts: > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>> > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>> 1. The boundaries/properties of “DAG” > > > >> > >>> have > > > >> > >>>>>>> already > > > >> > >>>>>>>>> faded > > > >> > >>>>>>>>>>>> over > > > >> > >>>>>>>>>>>>>>>>>>>>> time, > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>> for example there are now several ways > to > > > >> > >>>>>> create > > > >> > >>>>>>>>> cyclic > > > >> > >>>>>>>>>>>>>>>>>>> graphs, > > > >> > >>>>>>>>>>>>>>>>>>>>>> e.g. > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>> using the @continuous schedule. I > imagine > > > >> > >>>>> these > > > >> > >>>>>>>>>>> properties > > > >> > >>>>>>>>>>>>>>>>>>>>>> vanishing > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>> even more in the future, so from that > > > >> > >>>>>>> perspective I > > > >> > >>>>>>>>>>> support > > > >> > >>>>>>>>>>>>>>>>>>>>>> changing > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>> “DAG" to a more generic name. > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>> > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>> 2. How other orchestration frameworks > do > > > >> > >>>>>> naming: > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>> Dagster: pipeline > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>> Prefect: flow > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>> Flyte: workflow > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>> Temporal: workflow > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>> Kestra: flow > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>> > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>> I think “workflow” is the most fitting > > > >> > >>> name. > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>> > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>> 3. Given the large impact of this > > > >> > >>> change, I > > > >> > >>>>>>> suggest > > > >> > >>>>>>>>>>>> defining > > > >> > >>>>>>>>>>>>>>> a > > > >> > >>>>>>>>>>>>>>>>>>>>>> clear > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>> path forward. Would we first introduce > > > >> > >>> the > > > >> > >>>>>>>>> deprecation in > > > >> > >>>>>>>>>>>>>>>>>>>>> Airflow > > > >> > >>>>>>>>>>>>>>>>>>>>>> 3, > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>> and remove “DAG” in Airflow 4? > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>> > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>> Bas > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>> > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>> On 22 Oct 2024, at 09:22, Neil < > > > >> > >>>>>>>> neil4r...@gmail.com > > > >> > >>>>>>>>>>> <mailto:neil4r...@gmail.com>> > > > >> > >>>>>>>>>>>> wrote: > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>> > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>> I don't see a problem with the term > DAG, > > > >> > >>>>>>>> especially > > > >> > >>>>>>>>> when > > > >> > >>>>>>>>>>>>>>>>>>> most > > > >> > >>>>>>>>>>>>>>>>>>>>>> other > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>> platforms embrace the term > > > >> > >>> wholeheartedly. > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>> I don't see anything intimidating or > > > >> > >>>>> confusing > > > >> > >>>>>>>>> about it > > > >> > >>>>>>>>>>> at > > > >> > >>>>>>>>>>>>>>>>>>>>> all, > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>> changing the term though would be > fairly > > > >> > >>>>>>> confusing > > > >> > >>>>>>>>> to > > > >> > >>>>>>>>>>> most > > > >> > >>>>>>>>>>>>>>>>>>>>> users > > > >> > >>>>>>>>>>>>>>>>>>>>>>> who > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>> have been > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>> using > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>> the term for years. > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>> > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 1:18 AM > Tzu-ping > > > >> > >>>>> Chung > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>> <t...@astronomer.io.inva <mailto: > > > >> > >>>>>>>>> t...@astronomer.io.inva > > > >> > >>>>>>>>>>>> lid > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>> > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>> wrote: > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>> > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> I totally agree with doing away with > > > >> > >>> the > > > >> > >>>>> term > > > >> > >>>>>>>> DAG. > > > >> > >>>>>>>>> The > > > >> > >>>>>>>>>>>> only > > > >> > >>>>>>>>>>>>>>>>>>>>>>> problem > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>> (aside > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> from actually telling > people—including > > > >> > >>>>>>> myself—to > > > >> > >>>>>>>>> stop > > > >> > >>>>>>>>>>>> using > > > >> > >>>>>>>>>>>>>>>>>>>>> the > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> term) > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>> is to > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> come up with a reasonable > alternative. > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> I can’t recall who, but someone > > > >> > >>> mentioned > > > >> > >>>>>>>>> “workflow” is > > > >> > >>>>>>>>>>>> not > > > >> > >>>>>>>>>>>>>>>>>>>>> very > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>> accurate > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> for Airflow. The term “definition” > was > > > >> > >>>>>>> proposed, > > > >> > >>>>>>>>> but > > > >> > >>>>>>>>>>>> it’s a > > > >> > >>>>>>>>>>>>>>>>>>>>> bit > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> broad; I tried to use it in a few > > > >> > >>> places > > > >> > >>>>> and > > > >> > >>>>>>> kept > > > >> > >>>>>>>>>>> finding > > > >> > >>>>>>>>>>>>>>>>>>>>> myself > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> doubting “what definition?” and > > > >> > >>> wanting to > > > >> > >>>>>>>> clarify > > > >> > >>>>>>>>> “DAG > > > >> > >>>>>>>>>>>>>>>>>>>>>>> definition” > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> (defeating the purpose). > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> TP > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 22 Oct 2024, at 13:07, Jens > > > >> > >>> Scheffler > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> <j_scheff...@gmx.de.inva <mailto: > > > >> > >>>>>>>>>>> j_scheff...@gmx.de.inva>LID> > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote: > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Ryan, > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for posting. I share the > > > >> > >>> exactly > > > >> > >>>>> same > > > >> > >>>>>>>>>>>> observation, > > > >> > >>>>>>>>>>>>>>>>>>>>> had a > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> short > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> laight because the DAG question is > > > >> > >>> always > > > >> > >>>>> an > > > >> > >>>>>>>>>>> introduction > > > >> > >>>>>>>>>>>>>>>>>>> if > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> someone > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>> joins > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> the party. I think a global renaming > > > >> > >>> makes > > > >> > >>>>>>> sense. > > > >> > >>>>>>>>>>>>>>>>>>> Especially > > > >> > >>>>>>>>>>>>>>>>>>>>>> when > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> we > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>> also > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> rename Dataset to Asset this is also > a > > > >> > >>>>>>> reasonable > > > >> > >>>>>>>>> step. > > > >> > >>>>>>>>>>>>>>>>>>>>> Concepts > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> still > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>> can > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> stay the same. > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> So I hope I don‘t need to join > hiding > > > >> > >>>>> below > > > >> > >>>>>>> the > > > >> > >>>>>>>>> desk > > > >> > >>>>>>>>>>>> with > > > >> > >>>>>>>>>>>>>>>>>>>>> you > > > >> > >>>>>>>>>>>>>>>>>>>>>> and > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> +1 > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>> for > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> raising the discussion. > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Technically we can still think if we > > > >> > >>> keep > > > >> > >>>>>>>> details > > > >> > >>>>>>>>> of > > > >> > >>>>>>>>>>>>>>>>>>> python > > > >> > >>>>>>>>>>>>>>>>>>>>>> names > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> the > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> same because the execution is still a > > > >> > >>> DAG… > > > >> > >>>>>> but > > > >> > >>>>>>>> user > > > >> > >>>>>>>>>>>> facing > > > >> > >>>>>>>>>>>>>>>>>>> it > > > >> > >>>>>>>>>>>>>>>>>>>>>> is a > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>> workflow. > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jens > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Sent from my Smartphone > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 21. Oct 2024, at 23:56, Ryan > > > >> > >>> Hatter < > > > >> > >>>>>>>>>>>>>>>>>>>>>>> ryan.hat...@astronomer.io <mailto: > > > >> > >>>>>>>>> ryan.hat...@astronomer.io > > > >> > >>>>>>>>>>>> > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>> .invalid> > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote: > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Everyone please sheathe your > > > >> > >>> swords... at > > > >> > >>>>>>> least > > > >> > >>>>>>>>> for > > > >> > >>>>>>>>>>>> now. > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The term "DAG" has very little > > > >> > >>> meaning to > > > >> > >>>>>>>> Airflow > > > >> > >>>>>>>>>>>> users. > > > >> > >>>>>>>>>>>>>>>>>>>>>> Indeed, > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> it > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>> has > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> little meaning outside of some > > > >> > >>>>>> mathematicians > > > >> > >>>>>>>> and > > > >> > >>>>>>>>>>>>>>>>>>> software > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> engineers > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>> for > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> whom the properties of a DAG > actually > > > >> > >>>>>> matter. > > > >> > >>>>>>>> For > > > >> > >>>>>>>>>>>> someone > > > >> > >>>>>>>>>>>>>>>>>>>>> new > > > >> > >>>>>>>>>>>>>>>>>>>>>> to > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> data engineering or workflow > > > >> > >>>>> orchestration, > > > >> > >>>>>>> one > > > >> > >>>>>>>>> of > > > >> > >>>>>>>>>>> the > > > >> > >>>>>>>>>>>>>>>>>>>>> first > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> questions they > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> will > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> likely have is, "what on earth is a > > > >> > >>> DAG?" > > > >> > >>>>>> The > > > >> > >>>>>>>>> answer > > > >> > >>>>>>>>>>> is > > > >> > >>>>>>>>>>>>>>>>>>>>> almost > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> always, "It's a directed acyclic > > > >> > >>> graph. > > > >> > >>>>> You > > > >> > >>>>>>>> don't > > > >> > >>>>>>>>>>> need > > > >> > >>>>>>>>>>>> to > > > >> > >>>>>>>>>>>>>>>>>>>>>> worry > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> about what > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>> that > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> means; it's just a term for your > > > >> > >>>>> workflow." > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The term "DAG" is problematic for > at > > > >> > >>>>> least > > > >> > >>>>>> a > > > >> > >>>>>>>>> couple > > > >> > >>>>>>>>>>>>>>>>>>>>> important > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> reasons: > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> *Complexity for New Users*: As > > > >> > >>> mentioned > > > >> > >>>>>>> above, > > > >> > >>>>>>>>> "DAG" > > > >> > >>>>>>>>>>>> is > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> unnecessarily intimidating and > > > >> > >>> confusing. > > > >> > >>>>>> We > > > >> > >>>>>>>> want > > > >> > >>>>>>>>>>>> Airflow > > > >> > >>>>>>>>>>>>>>>>>>>>> to > > > >> > >>>>>>>>>>>>>>>>>>>>>> be > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> approachable, and > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> using > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> technical jargon like "DAG" right > > > >> > >>> off the > > > >> > >>>>>> bat > > > >> > >>>>>>>>> creates > > > >> > >>>>>>>>>>>> an > > > >> > >>>>>>>>>>>>>>>>>>>>>> initial > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> barrier to > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> understanding. > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> *Disconnect Between DAG and > Workflow > > > >> > >>>>>>> Concepts*: > > > >> > >>>>>>>>> The > > > >> > >>>>>>>>>>> DAG > > > >> > >>>>>>>>>>>>>>>>>>> is > > > >> > >>>>>>>>>>>>>>>>>>>>>> just > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> one component of an Airflow > > > >> > >>> workflow. The > > > >> > >>>>>>>>> workflow > > > >> > >>>>>>>>>>>>>>>>>>> includes > > > >> > >>>>>>>>>>>>>>>>>>>>>> its > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> schedule, retries, timeouts, a > dozen > > > >> > >>>>> other > > > >> > >>>>>>>>>>> parameters, > > > >> > >>>>>>>>>>>>>>>>>>> and > > > >> > >>>>>>>>>>>>>>>>>>>>>> other > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> metadata that > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>> the > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> DAG component doesn’t account for. > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Consider the following from the > > > >> > >>> Airflow > > > >> > >>>>>>>> homepage > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <https://airflow.apache.org/> < > > > >> > >>>>>>>>>>> https://airflow.apache.org/>>. > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Apache Airflow® is a platform > > > >> > >>> created by > > > >> > >>>>>> the > > > >> > >>>>>>>>>>> community > > > >> > >>>>>>>>>>>> to > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> programmatically > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> author, schedule and monitor > > > >> > >>> workflows. > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Then, if we look at the "What is > > > >> > >>>>> Airflow?" > > > >> > >>>>>>> docs > > > >> > >>>>>>>>> page > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> < > > > >> > >>>>>>>>>>>>>>>>>>>>>>> > > > >> > >>>>>>>>>>>>>>> > > > >> > >>>>>>>>> > > > >> > >>> > > https://airflow.apache.org/docs/apache-airflow/stable/index.html > > > < > > > >> > >>>>>>>>>>> > > > >> > >>>>> > > > https://airflow.apache.org/docs/apache-airflow/stable/index.html > > > >> > >>>>>>> > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> , > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>> we > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> can > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> see that the docs explain what > > > >> > >>> Airflow is > > > >> > >>>>>>>> without > > > >> > >>>>>>>>>>> using > > > >> > >>>>>>>>>>>>>>>>>>>>> "DAG." > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> It's > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> only in > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the *workflow* Python code that the > > > >> > >>> term > > > >> > >>>>> is > > > >> > >>>>>>>>>>> introduced > > > >> > >>>>>>>>>>>>>>>>>>> out > > > >> > >>>>>>>>>>>>>>>>>>>>> of > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nowhere > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> as a > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> comment that awkwardly tries to > > > >> > >>> explain > > > >> > >>>>> it: > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> # A DAG represents a workflow, a > > > >> > >>>>> collection > > > >> > >>>>>>> of > > > >> > >>>>>>>>> tasks > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> It makes sense to not refer to DAGs > > > >> > >>> in > > > >> > >>>>>> these > > > >> > >>>>>>>>>>>>>>>>>>> introductions > > > >> > >>>>>>>>>>>>>>>>>>>>> to > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Airflow, because *Airflow doesn't > > > >> > >>>>>> orchestrate > > > >> > >>>>>>>>> DAGs; > > > >> > >>>>>>>>>>> it > > > >> > >>>>>>>>>>>>>>>>>>>>>>>> orchestrates > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> workflows*. > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> The > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> DAG is the model that, for reasons > > > >> > >>>>>> irrelevant > > > >> > >>>>>>>> to > > > >> > >>>>>>>>>>> almost > > > >> > >>>>>>>>>>>>>>>>>>>>> every > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> user, workflows must adhere to. > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> So, I propose at least adding an > > > >> > >>> alias > > > >> > >>>>> for > > > >> > >>>>>>> the > > > >> > >>>>>>>>> term > > > >> > >>>>>>>>>>>> "DAG" > > > >> > >>>>>>>>>>>>>>>>>>>>> and > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> updating documentation to replace > > > >> > >>> "DAG" > > > >> > >>>>>> with > > > >> > >>>>>>>>>>>> "workflow". > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> For example, instead of... > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> @dag( > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> schedule="@daily", > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ... > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> dagrun_timeout=timedelta(hours=1) > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ) > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Users could do... > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> @workflow( > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> schedule="@daily", > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ... > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> run_timeout=timedelta(hours=1) > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ) > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> And with that... I will start > running > > > >> > >>>>> away. > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > >> > >>>>>>>>>>>>>>>>>>>>>>> > > > >> > >>>>>>>>>>>>>>> > > > >> > >>>>>>>>> > > > >> > >>> > > ------------------------------------------------------------------ > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> --- To unsubscribe, e-mail: > > > >> > >>>>>>>>>>>>>>>>>>>>> dev-unsubscr...@airflow.apache.org <mailto: > > > >> > >>>>>>>>>>> dev-unsubscr...@airflow.apache.org> > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> For additional commands, e-mail: > > > >> > >>>>>>>>>>>>>>>>>>>>> dev-h...@airflow.apache.org <mailto: > > > >> > >>>>>>>>>>> dev-h...@airflow.apache.org> > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > >> > >>>>>>>>>>>>>>>>>>>>> > > > >> > > > > > >> > > > > > >> > > > > > --------------------------------------------------------------------- > > > >> > > To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org > > > >> > > For additional commands, e-mail: dev-h...@airflow.apache.org > > > >> > > > > > >> > > > > >> > > > > >> > > > > >> > > > --------------------------------------------------------------------- > > > >> > To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org > > > >> > For additional commands, e-mail: dev-h...@airflow.apache.org > > > >> > > > > >> > > > > >> > > > > > > > > > >
