Yup agreed with Jarek. A strong no from my side. We don't want to allow authoring DAGs from Airflow UI especially just to provide an LLM interface.
On Mon, 7 Jul 2025 at 12:27, Jarek Potiuk <ja...@potiuk.com> wrote: > Also you might take a look at Airflow Summit videos > https://www.youtube.com/playlist?list=PLGudixcDaxY2NIjMYT8t5zA9KJ47wTCkM > -> > and look back to 2023. There were at least several talks about using LLMs > to generate Airflow Dags, and our users are doing it already - and I guess > it's quite natural for people to generate the Dags with the help of LLMs > already. > > On Mon, Jul 7, 2025 at 8:52 AM Jarek Potiuk <ja...@potiuk.com> wrote: > > > FYI I added your email directly - because apparently you are not > > subscribed to devlist - please do subscribe following the "community" tab > > on our website. > > > > I don't want to cut down your wings and excitement, but this is a > > deliberate choice that Airflow UI does not allow to author DAGs. This is > a > > security feature. And our security model > > > https://airflow.apache.org/docs/apache-airflow/stable/security/security_model.html > > is very clear that "UI users" do not have (and should not have) > > capabilities of authoring DAGs (not as Python code - that allows > > arbitrary code execution). Maybe (and that is something we might consider > > in the future) if there is a declarative way of creating DAGs which does > > not allow to provide arbitrary code, we could allow that, but we have not > > even settled on the idea of having a single declarative way of creating > > Dags. > > > > Also Airflow DAGS are just Python Code placed in a folder. And there is > > absolutely nothing stopping you to open your IDE with Claude , Cursor, > > Copilot, use the prompt of your choice and ... generate DAGs with LLM. > > There is absolutely no need to have a UI for that.- all the IDEs out > there > > already have a fantastic LLM integration, with capability of adding > prompt, > > using MCP servers (there are even several MCP servers for Airflow created > > by the community and we are discussing about creating our own MCP server > > https://lists.apache.org/thread/xgd66v6s7zf0xkvy3c7ysqvn4csgmw0 - those > > IDEs have code completion, syntax check, allow you to interact with the > > Agents and approve/reject proposals when you are using agents to create > > your DAGs. They even allow you to use your own models that can be > RAG-ified > > based - for example - on the private DAGs your company might have. This > all > > works **today**. > > > > I don't think personally there is any benefit of creating a similar > > feature in Airflow UI. I can't see any to be honest. Maybe others have a > > different opinion or maybe you can explain what benefits you see by > adding > > such a "UI feature" to Airflow itself (and also the problem about > security > > is extremely important and a huge blocker for the whole idea - until this > > is somewhat addressed the whole idea is basically impossible to be > accepted > > by the community. > > > > J. > > > > > > On Mon, Jul 7, 2025 at 8:37 AM Harikrishnan Girikumar < > > harikrishnangiriku...@gmail.com> wrote: > > > >> Hello Team, > >> > >> My name is Harikrishnan(Hari), I have an idea/improvement proposal for > >> Airflow. > >> LLM-powered feature within Apache Airflow to significantly enhance the > >> DAG > >> authoring experience. Users would be able to provide natural language > >> descriptions or queries and leverage Large Language Models (LLMs) to > >> automatically generate and modify Airflow DAGs. This aims to democratize > >> DAG creation, reduce the learning curve for new users, and accelerate > the > >> development of complex workflows. For example: We can have a UI tab in > >> Airflow where users can add their respective authentication credentials > >> for > >> the LLMs they want to use (OpenAI, Claude or their personal model > serving > >> link etc.) they can select their AI from drop down and a chat window to > >> input queries like: Create a DAG to copy my data from S3 to Postgres and > >> the code generated would be copied to DAG folder. We can restrict the > >> Prompts to be strictly for DAG generation for initial trial, further > down > >> the line a RAG feature could be added where a Vectorized version of > >> Airflow > >> documentation is used to improve the accuracy of DAG creation. > >> > >> I am really excited about this feature, this would reduce the learning > >> curve and improve the interaction for new users. Let me know your > >> thoughts, > >> looking forward to hearing from the team. > >> > >> Regards, > >> Hari > >> > > >
