Cool! Thanks for sharing your thoughts here Kevin, user perspective is always important.
Thanks & Regards, Amogh Desai On Thu, Jul 10, 2025 at 6:59 PM Kevin Yang <kevinsjy...@gmail.com> wrote: > Hi, I joined airflow community around two months ago and I would also like > to share some thoughts, probably more from a user perspective. I agree that > LLM can streamline DAG creation, especially for users/teams that are new to > Airflow. However, in production system, static DAG files may be more > favorable by devops or platform teams that manage the Airflow service. > > As Jarek mentioned, there are security concerns for authoring DAGs through > UI. I also received feedback from people to avoid using dynamic DAG > generation. Basically, avoid delegating DAG creation to DAG parsing on the > server end. To deal with those concerns, we have to build a framework, > similar what Amogh mentioned, to render explicit DAG code in CI and deploy > those as static Python files. > > I would also not recommend this practice, and I think providing proper > prompts to the existing AI tools are also very helpful to write proper DAG. > > Thanks, > Kevin > > > Sent from Outlook for iOS<https://aka.ms/o0ukef> > ________________________________ > From: Amogh Desai <amoghdesai....@gmail.com> > Sent: Tuesday, July 8, 2025 2:04:33 AM > To: dev@airflow.apache.org <dev@airflow.apache.org> > Subject: Re: [DISCUSS] - LLM-Powered DAG Generation > > Strongly concur with everything Jarek had to tell. > > I would be in favour of a strong NO for this. The use case of having an > embedded > UI to type in natural language is as good as opening your IDE locally with > integrated > LLM models and asking it to generate the dag using natural language and > drop it in the > DAGs folder. If you have airflow hosted somewhere, you can consider using > the git sync model > to achieve similar results. > > There also have been a couple of drag and drop plugins built on top of > Airflow > to construct dags in a more naive manner, for example: > https://github.com/Joffreybvn/airflow-dag-builder > is one I remember from a couple of years ago. > > Soon, there also will be multi language support for Airflow tasks and that > is also something > you can consider using if Python has a learning curve :) > > Thanks & Regards, > Amogh Desai > > > On Tue, Jul 8, 2025 at 1:10 AM Jens Scheffler <j_scheff...@gmx.de.invalid> > wrote: > > > Hi, > > > > also sharing the opinion that no dedicated UI is needed for this - but > > would be very welcoming to share experience and maybe expereince and > > starting promt to get going. So if you want to post (somewhere, e.g. > > medium) an article about how this is possible, that migth be worthwile > > to share. > > > > CFP for the next Airflow summit is over but there might be also other > > oppuortunities to share and spread the use w/o needing to integrate into > > Airflow codebase. Maybe even on Monthly townhall hosted by Astronomer. > > (The taks from 2 years ago might be a bit outdated though they share the > > same ground idea - but a lot has evolved since then...) > > > > Jens > > > > On 07.07.25 12:53, Kaxil Naik wrote: > > > Yup agreed with Jarek. A strong no from my side. We don't want to allow > > > authoring DAGs from Airflow UI especially just to provide an LLM > > interface. > > > > > > > > > > > > On Mon, 7 Jul 2025 at 12:27, Jarek Potiuk <ja...@potiuk.com> wrote: > > > > > >> Also you might take a look at Airflow Summit videos > > >> > > https://www.youtube.com/playlist?list=PLGudixcDaxY2NIjMYT8t5zA9KJ47wTCkM > > >> -> > > >> and look back to 2023. There were at least several talks about using > > LLMs > > >> to generate Airflow Dags, and our users are doing it already - and I > > guess > > >> it's quite natural for people to generate the Dags with the help of > LLMs > > >> already. > > >> > > >> On Mon, Jul 7, 2025 at 8:52 AM Jarek Potiuk <ja...@potiuk.com> wrote: > > >> > > >>> FYI I added your email directly - because apparently you are not > > >>> subscribed to devlist - please do subscribe following the "community" > > tab > > >>> on our website. > > >>> > > >>> I don't want to cut down your wings and excitement, but this is a > > >>> deliberate choice that Airflow UI does not allow to author DAGs. This > > is > > >> a > > >>> security feature. And our security model > > >>> > > >> > > > https://airflow.apache.org/docs/apache-airflow/stable/security/security_model.html > > >>> is very clear that "UI users" do not have (and should not have) > > >>> capabilities of authoring DAGs (not as Python code - that allows > > >>> arbitrary code execution). Maybe (and that is something we might > > consider > > >>> in the future) if there is a declarative way of creating DAGs which > > does > > >>> not allow to provide arbitrary code, we could allow that, but we have > > not > > >>> even settled on the idea of having a single declarative way of > creating > > >>> Dags. > > >>> > > >>> Also Airflow DAGS are just Python Code placed in a folder. And there > is > > >>> absolutely nothing stopping you to open your IDE with Claude , > Cursor, > > >>> Copilot, use the prompt of your choice and ... generate DAGs with > LLM. > > >>> There is absolutely no need to have a UI for that.- all the IDEs out > > >> there > > >>> already have a fantastic LLM integration, with capability of adding > > >> prompt, > > >>> using MCP servers (there are even several MCP servers for Airflow > > created > > >>> by the community and we are discussing about creating our own MCP > > server > > >>> https://lists.apache.org/thread/xgd66v6s7zf0xkvy3c7ysqvn4csgmw0 - > > those > > >>> IDEs have code completion, syntax check, allow you to interact with > the > > >>> Agents and approve/reject proposals when you are using agents to > create > > >>> your DAGs. They even allow you to use your own models that can be > > >> RAG-ified > > >>> based - for example - on the private DAGs your company might have. > This > > >> all > > >>> works **today**. > > >>> > > >>> I don't think personally there is any benefit of creating a similar > > >>> feature in Airflow UI. I can't see any to be honest. Maybe others > have > > a > > >>> different opinion or maybe you can explain what benefits you see by > > >> adding > > >>> such a "UI feature" to Airflow itself (and also the problem about > > >> security > > >>> is extremely important and a huge blocker for the whole idea - until > > this > > >>> is somewhat addressed the whole idea is basically impossible to be > > >> accepted > > >>> by the community. > > >>> > > >>> J. > > >>> > > >>> > > >>> On Mon, Jul 7, 2025 at 8:37 AM Harikrishnan Girikumar < > > >>> harikrishnangiriku...@gmail.com> wrote: > > >>> > > >>>> Hello Team, > > >>>> > > >>>> My name is Harikrishnan(Hari), I have an idea/improvement proposal > for > > >>>> Airflow. > > >>>> LLM-powered feature within Apache Airflow to significantly enhance > > the > > >>>> DAG > > >>>> authoring experience. Users would be able to provide natural > language > > >>>> descriptions or queries and leverage Large Language Models (LLMs) to > > >>>> automatically generate and modify Airflow DAGs. This aims to > > democratize > > >>>> DAG creation, reduce the learning curve for new users, and > accelerate > > >> the > > >>>> development of complex workflows. For example: We can have a UI tab > > in > > >>>> Airflow where users can add their respective authentication > > credentials > > >>>> for > > >>>> the LLMs they want to use (OpenAI, Claude or their personal model > > >> serving > > >>>> link etc.) they can select their AI from drop down and a chat window > > to > > >>>> input queries like: Create a DAG to copy my data from S3 to Postgres > > and > > >>>> the code generated would be copied to DAG folder. We can restrict > the > > >>>> Prompts to be strictly for DAG generation for initial trial, further > > >> down > > >>>> the line a RAG feature could be added where a Vectorized version of > > >>>> Airflow > > >>>> documentation is used to improve the accuracy of DAG creation. > > >>>> > > >>>> I am really excited about this feature, this would reduce the > learning > > >>>> curve and improve the interaction for new users. Let me know your > > >>>> thoughts, > > >>>> looking forward to hearing from the team. > > >>>> > > >>>> Regards, > > >>>> Hari > > >>>> > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org > > For additional commands, e-mail: dev-h...@airflow.apache.org > > > > >
